Without testing, how can you be sure?

There's a lot of questions about "Is testing really needed". By the way, the correct answer is Yes. It seems that during development most developers say they'll develop the tests for their own equipment. This will ensure the unit won't be tested well.

Developers don't necessarily design for testability. Quite often they design connector's that can't be mated to easily, circuit cards that are embedded into the system to where they can't easily be tested, or circuit cards that just hard to deal with.

Designers know what parts of their design is weak and typically test in a way that their circuit will pass. Not consciously, but more of a sub-conscious because they don't want to see their baby fail. Or testing to what is designed and not in a way to truly verify the design.

I say testers are needed and need to be brought in early if they are to be able to help. Also, since quite often management don't see proper testing as a necessity, the test budget and schedule is shorted or cut to the bone. This will hamper the the testing effort and typically cause shotty testing of a unit.

If shotty or only partial testing is desired before sending a unit out to a customer, bring testers in late, or not at all. If you want to ensure the customer is satisfied, bring in the testers early and let them make sure a good job is done.

Software in Testing

Even in testing, you have to pay attention to software. It's part of the test development knowledge base. The old paradigm of "Software's free" needs to be left in the dust. If it's ignored or the attitude of you design the hardware and plan the test...oh yeah, just do the software needs to be left behind and you need to move forward into the 90's (yes, the 1990's). If you ignore the software, you'll pay in the long run.

I say this because this is the attitude from a test manager. He said there were no software problems with the test sets at his site. Everyone had a good laugh but I think he was serious.

Even in Test Engineering people need to continuously improve your software skills. If you're a hardware guy who had 1 software class you need to try to improve. If you're a degree in Software Engineering and have been developing software for 15 years there's still room to learn new software techniques.

If the software is poorly written the tests will be hard to maintain or possibly won't work properly. Without the proper software discipline the overall testing process won't go smoothly. As you move through the test development process, things will be rocky, problems will occur, and it will take to long to fix the problems. Software is one of the corner stones of testing.

So don't ignore it or it will come back to bite you.

What is a "Test" Person

Lately, I've been thinking about what exactly is a "Test" person...or more to the point a good test person. Someone who is good at testing needs to have multiple skills, good at each but not necessarily an expert in them. The skills needed are Software Engineering, Electrical Engineering, and Systems Engineering. The good test person needs to be good at Software because software drives the test. With tools such as TestStand, NI is trying to make software easier, when in actuality they're trying to make people who don't do software think they can. But if you do software poorly, the tests developed don't work well.

Electrical Engineering skills are needed because that's what is tested and the interfaces need to be understood. Also, the Unit Under Test (UUT) has to be understood well enough to know it's being tested throughly.

Systems are needed to tie the tests together and to understand how the UUT fits into the rest of the system so it can be tested.

A good tester has qualities of each and most are very good in one area and adequate in the others.

I was pondering this as I was trying to understand a test, in TestStand, where the developer used a lot of nifty TestStand features that just made the test harder to follow and understand what was really going on. I think the person who wrote it understood Systems really well, knew hardware well enough to use it, and knew a lot of the details of software without being that good at developing software that will need to be maintained by others.

Testing is Tricky Business

Actually, testing can tricky if your customer doesn't know what they want. You can't base real requirements on "Test this!". You say OK, you look at all the inputs and outputs and test them, you test the circuits behind the interfaces as best as possible. You even tell them in advance what you're going to test, they say "that looks good" (you keep all the e-mail transactions) .

You go to the test set sell off and you get the conversation "That's not what we wanted". "Well...What did you want", "I don't know, but when I see it, I'll know it". That's not how it really goes because after the "That's not what we wanted" you start cussing (in your head) and discussing what they signed up for and what they say they signed up for.

Again, this is the need for the good requirements up front so that they understand what they will be getting and you understand what they want.

Recent experiences have shown that the above discussion happens more in testing than the good requirements.

Good luck on getting through the cussing and discussing part. It's never easy.

Flexability vs. Usability

I've had some family and work issues lately but I'm going to continue this blog, it makes me thinks about what's important in testing.

Lately I've been working with National Instruments PXI 6229 Multi-function I/O board and NI DAQmx drivers. Using DAQmx has made me think about flexibility versus Usability. First, DAQmx is a departure from the standard method of get a handle to a device and use it. DAQmx functions wants a DAQmx task to be created and used. The task is associated with a set of operations, whether they're Digital I/O, Analog I/O, counter operations, A/D operations, etc. The task is then started and runs however long it takes to complete or how long is specified in one of the many operations setups.

This allows the general DAQmx routines to operate on any type of board. The problem is usability. Not all boards support all operations. A Digital I/O does not support counter or clock operations. Not all operations are as clearly unfit as those. Also, it's not known whether the function will work on the board until the software is run and the function returns an error. The PXI 6229 that allows clocked operations but doesn't have a clock which disallows any "clock" DAQmx functions. (Clock operation require using the counter or an external clock on a PFI line) Things like this are not obvious. Another usability problem is that there are many clock functions and it's hard to tell which one is needed. When trying to operate the [nonexistent] clock on a 6229 it's not clear if the function itself is not usable in the situation or if the board does not support it.

The examples NI provides aren't for all boards. The examples won't run on all boards. Some of the Digital I/O examples won't run on boards with Digital I/O.

Overall, the flexibility the DAQmx functions allow inhibit the usability of the functions. NI needs to apply the KISS principle, Keep It Simple Stupid.

The test development environment

This is less about computer development environment and more about physical development environment. In the past I've been in the unfortunate situation of going off-site to develop. It went slow and I got phyically sick at least once a week. This situation is about to happen again.

I bring this up because a test set we are working on is being sent to the customer's site. In the earlier parts of the contract the tests will be developed here. Due to the lateness of the UUT's being developed, the test development process will be moved to their site, where the UUTs will be. They originally promised to send UUT's for us to verify our tests in April 2007 but changed the agreement, rather ignored the agreement and changed what we had to do. That meant the we have to send a test set up to there site and the use their UUT's. The same UUT's that their engineers will be using at the same time we would like use them. Or maybe we have the midnight to 6am shift to use the UUT's.

This winds up putting all the pressure on us. We're in a strange place working the test set. If we need technician help we have to find one of there technicians to help us. Same with tools, seating, and, most of all, UUT's. We will be at their mercy on when we can use the UUT's and guess who will be at the bottom of the priority list for use of the UUTs. (answer: us)

The only good thing about this situation is we will have at least some access to the UUT's to complete our test verification. Nothing else is good about it. We'll be out of our environment, staying in hotels, away from family, and eating out all the time. It gets old quick.

As for me, I'm only going up there occasionally, I can get my stuff done down here. Since I'm a single dad, it would be tough for me to go up there for 3 months like some of the others.

Have a great day

NI and Education

I'm currently involved with the Grand Prairie Independent School District (ISD) to help develop academic programs for a Technical High School currently being planned and built. When we started I found out, to my dismay, that GPISD only had a simple Intro to Engineering class. They do have a programming class under the area of business and an electronics class under science. But nothing to lead people into engineering. I was appalled at the lack of Engineering curriculum.

Now, I'm interested in helping them get some engineering programs in place for the new technical high school. I was looking at what academic information NI has. They have an NI Academic area on there website. It is mainly LabVIEW based academics...as if that were a surprise.

They also have NI ELVIS which a LabVIEW based electronic is an integrated design and prototyping platform for engineering and science laboratories to teach concepts in measurements, circuits, controls, microcontroller, and embedded design. It is a bit pricey for high schools and I think targeted to colleges.

Hopefully, the GPISD can afford to put in place a good engineering program. Whether they can afford NI or not is yet to be seen.

Project lead the way is a good, somewhat canned engineering curriculum. There are other school districts that use Project Lead the way and GPISD will probably follow suite.

Well, have a good day.