Future ot Test Development

Since I am a test developer I think a lot about how to make my own job easier. I believe National Instruments is on the right track in the future of test software with LabVIEW. Graphical is very much the way to go, especially for the Hardware guy forced to write software tests. It still has room for improvements especially with the amount of mousing that it requires. I have mild Carpel Tunnel Syndrome but when I use LabVIEW, it flares up and I have pain in my wrist. LabVIEW also easy to leave out a lot of the discipline really needed for software, things like clean code, readability, and documentation. But LabVIEW is still the way.

In the farther future, I believe that the development of hardware and software will be more coupled and much more automated. NI, again, is doing well on this. With NI-Scope, NI-DMM, and NI-Switch. When you get an NI-DMM card, any DMM card, and use NI-DMM to deal with it. All that really needs to be known is the basics of what DMM measurements need to be made. It's all drop in from there.

The more distance test development future should bring more automated and integrated development. I believe the Test Engineer will specify what signals, tolerances, and communications channels needed to be tested for each of the UUTs that are to be tested. A test development (TD) tool will select the best instruments for the job and layout the Interconnect panel (ICP) (The ICP is where all signals from all the instruments come out to the outside world) The Test Engineer would verify the instruments are right.

The TD tool would use the signal to be tested and instrument it chose to create an Integration Test Adapter (ITA) to self test the station. (ITA take the signal from the Instrument through the ICA and routes it to cables, switch cards, and other instruments). The wire lists for the ITA fabrication would come from the tool. From the ITA and wire lists the software for a test set self test would be generated.

Then the tool will take Save Nowthat ICA output and the UUT signal specs that we started with and would develop the ITA and wire list for each UUT test. Using the UUT wire lists and ITA a UUT self tests software and UUT test software would be generated.

The switch cards needed to route the signals for the Station Selftest and UUT tests would be added to the design.

Below is a simple flow I threw together.

This whole thing is conjecture on my part and very over simplified but, in general, seems doable. Overall, eventually, there will most likely be a lot more automation of layout and test.

Future of Test

One thing that a lot of test engineering companies are doing really well is gathering and analyzing failure data. But in the future the data could be gathered to a central repository and the fault data mined more efficiently, the way some companies data mine for business intelligence. Very likely using some AI techniques to analyze the data. Potentially some type of Neural Net, advanced statistical analysis, and Bayesian mathematics that accepts the failure information and gives the most likely fix. Well, that's what I hope since AI is my hobby. Both test set and Unit Under Tests (UUT) could be analyzed and information about future failures be determined in advanced from current failures.

When a test set or UUT failure occurs the failure information and analysis could be directed to the Test Engineer's pocket PC. He/She would be able to attach to the test set from where ever they are and trouble shoot most problems. Through the remote link some problems would be fixed and some would need to be coordinated with on-site technicians.

As for operating the tests, there could be a central operator planning, scheduling, and controlling all of the tests in a factory. The operator would have the various test screens on his/her screen to interact with the test software. There could be some people on the work floor to make sure everything gets hooked up correctly and the UUT's flow from one manufacture operation to the next.

I do believe wireless and the web will pay a much bigger role in future test. Remote operations will be more common and more advanced failure detection, reporting, and analysis will be the norm.

I also believe that AI will pay a bigger role in test...and not just because it's my hobby.

Future of Test

At work, I've been dealing with innovation, future of test software, and even doing a little bit of real engineering work in my spare time.

With all this I keep wondering, what is the Future of test set? Of test software? How much will change before I retire? When can I retire (different topic)? Also, I was doing an install of some software and had some time to contemplate the future.

I know in the future the Engineers creed of "Smaller, Better, Faster" will be in effect. Also the concept of Cheaper is being pushed really hard. So what will happen in the future.

Smaller is easy. We've seen it happen for years in standards like VME going to VXI and to PXI as well as chassis like the compactRIO. With these different architectures we've also seen faster speeds. So in the next 10 years I'm sure there will be a sub-compactRIO or RIO-micro.

I'm pretty sure distributed systems will be more common. A system where there is a central control and information repository with connections to small "brick" testers that can be put just about anywhere, temperature chambers, remote less access able test area's. There will be wireless connections where possible so that even wires aren't needed. Some type of Ethernet (hardwire or wireless) will connect to the brick and the brick will control and communication with the units under test (UUT). All test results and operator interface will back at the central computer. The central computer will control several different tests at one time.

There will be more embedded test programs in the units under test (UUT), there will be less discrete interfaces. Since UUTs being tested are getting smaller (better faster) too, there will be less area for interfaces. Software expertise will be needed to develop Operational Test Programs (OTPs) to load into units to test them.

With distributed test systems and smaller test sets there will be more "soft" instrument front panels on the display screens and less physical instrument front panels. The soft instrument front panels on the screens would give an image of what the real instrument front panel would look like if it were real. They would control the various instruments manually when it's needed and would minimize when the device isn't being used. Touch screens would be great for this function so that the operator would still "touch" on the interface.

Of course, all this are just some possibilities for the future.

From Hack to Engineer

I talked about people being the most important tool of Test Engineering. People are the most important factor in any Engineering discipline. People, Engineers, are the constant though out development. No matter how good or thorough the processes are people still have to implement them. And if the processes are not good the people still have to develop the product.

There are several levels of skill that go along with doing the job of Engineer. A hack (not be be confused with hacker) learns as he goes, acts and then thinks, and cleans up his messes only when he absolutely has to, if he can't pass it on to someone else.

Then theirs the craftsman Engineer. He studies, he plans, uses his best practices and tools and takes pride in his work. But a craftsman is not to the level of Engineer because while, he develops a well crafted product, it lacks predictability and certainty.

Then theirs Engineers. With Engineers it's all about knowing instead of guessing. An Engineer doesn't estimate, he/she calculates. An Engineer doesn't hope, he knows. Estimates can formed based on what they devise. Engineers basically have consistency.

Consistency is what Engineers are about. They still need the hack characteristic of learning as they go and the craftsman characteristic of using best practices and tools.

The best way to get a good product is to have good Engineers working on the product. We also need to encourage and train the hack Engineer to become a Craftsman. We also need to encourage and train the craftsman Engineer to become a real Engineer.

SQA - a necessary evil

Software quality is not a myth in Test Engineering…although it's not always accomplished. I know that around here it's not always acknowledged that Test Engineering has software so that the SQA people don't bug us to much. I believe that if everyone had a good software discipline, we wouldn't need SQA. The fact of the matter is, even most of the pure software engineers don't have that type of discipline.

If people are not monitored, at least occasionally, they tend to fall into the bad habits as schedules get tight or work loads increases. The code will degrade, maybe not into Chaos, but to at least crappy code. With a police presences (SQA) to keep the crowd of programmers from moving in the dangerous direction, it remains fun and good code is developed. Without over site, the results would be code chaos (or maybe just bad code). With the software police around, we move in the right direction.

Test Software, at least where I work, is not monitored by SQA like it should be. Typically the SQA people, or whomever sets up their budgets, don't budget enough for TE Software. The mission critical flight software gets the attention while test software is left alone.

Test Software needs to follow some standards, meet some criteria, needs to be verified to a certain level of "goodness", just like every other software. The standards are followed when we know someone is making sure we follow the standards.

The test software lead should demand the Software Quality group to do their job for test engineering, like for the other software groups. This is probably a departure from what most test engineer's want, but if the company wants solid software, then it needs to be done. If the SQA can't do the monitoring of the test software then the Test Engineering lead needs to do at least the basic monitoring. I know the lead doesn't have time but it needs to be done. The goal is good solid test software.

Basically, at the very least, the test lead needs to make the following is done:

• Code reviews are done and problems found early

• Make sure the requirements are met in the design and again in the code

• He needs to make sure at least minimal style is met for reuse ability sake and for debugging.

• He needs to make sure the code is checked and verified against the requirements.

The lead shouldn't just ask the engineer's “did you do it?”, it's the same as you letting the engineer's police themselves. Then need to be checks. This duty can be delegated as long as whomever it's delegated to knows what to look for.

Again, the goal is good code, the policing the engineers is how to get there.

Peer Reviews

"Peer reviews are a waste of time", "Peer reviews don't help", "We don't have time to peer review the", peer reviews are evil, blah, blah blah! I've heard this for years and all I can say to this is "Bologna!" A second or third pair of eyes on something is useful. Haven't you ever had a problem while debugging code and said "Hey Bob, can you take a look at this" and then either through your explanation or something Bob says you have an "a-ha" moment can get past the problem? A peer review is like trying to get to the "a-ha" moment before you realize you need to.

Statistics have shown that Peer Reviews do help if they're done right. (see the book “Best Kept Secrets of Peer Code Reviews” by Jason Cohen.

First, some good reasons to do peer reviews. The best reason is defects found early cost less to fix, simple economics. A second pair of eyes can help. Another reason is that less experienced programmers can learn from more experienced programmers at peer reviews. One note is that typically everybody can learn something from other peoples code.

Second, to shoot down some myths. People have a lot of reasons to not do Peer Reviews. Things like "my code is already good enough". To this I have to ask, have you ever had any problems with your code? If you answered No then you are better than any of the Rock Star programmers that I've heard about. Peer reviews are to get rid of problems before they get out.

How about “We don't have time for peer reviews.” Then how much time do you have to fix problems later? And how much will it cost?
Another myth is that peer reviews don't help. They don't help if you don't do them. They don't help if you don't really review the code. If you find just a few defects it can be worth doing it. My Mom and Dad used to say “Anything worth doing is worth doing right”.

There are alway excuses for not doing peer reviews, but no good reasons.
People have ego problems with peer reviews, too. There's the “Big Brother” effect where programmers feel like the peer review is to monitor their every move. This is not how it should be, it should be about removing defects and not about monitoring.

Earlier I said peer reviews help if they're done right. That means the code is really reviewed, the reviewers read over the code and look for potential problems, makes sure it meets the requirements, and is understandable. If this is done then defects are found and fixed. If the code is just glanced over, fewer defects are found, problems are found later and costs more to fix.

To close there are some truisms about Peer Reviews:


1) Hard code has more defects – the more complex the code, the more potential defects.
2) More time yields more defects – The more time spent reviewing the more defects found
3) It's all about the code – Review the code not the programmer
4) The more defects the better – Defects found early are cheaper to fix

The bottom line is a better product and that's what peer reviews should be all about. Happy reviewing!

Technical Debt

I was browsing around the web and found a interesting concept, Technical Debt. This was a concept brought up by Ward Cunningham in OOPSLA '92. It is being taken over by the Agile people under Agile Project Management . I think it's a good concept for test engineers to take into account, too. The Agile portion and blogs by Steve McConnell (Technical Debt and Technical Debt Decision making) discuss it from a software standpoint. But for test engineers, it can be more than just technical debt in the software. The articles don't go into much detail on exactly what technical deficit is.

Here is what I think. For software some items that would cause technical deficit would be issues:

- Hacked together code or code with a lot of short cuts
- Spaghetti code
- Code that is complex or hard to follow
- Incomplete or inadequate error checking.
- Code to be implemented later
- TBD's or incomplete requirements.
- Poor or Incomplete design
- Anything “owed” to the software that is deferred.

I was contemplating technical debt for the rest of test and hardware can most certainly have technical debt, too. Again, it would be anything owed to the hardware. Some of this would be:

- Incomplete schematics
- Unspecified connectors
- Incomplete grounding or shielding
- Incomplete wire lists
- Incomplete wire specifications where they were needed.
- Again, anything “owed” to the hardware

The mechanical portions could have technical debt as well. It would be things like:

- Incompatible of incomplete layout
- Unspecified mechanical connections,
- Other items that are needed but left unspecified.

(I not as much an expert on hardware or mechanical aspects of test but I'm sure a lot of the hardware people can add to these lists.)

The blogs by Steve McConnell on Technical Debt equate it to financial debt because they have a lot of the same issues. Technical debts, as well as financial debts, have to be taken care of at some point in time or they will bite you in the rear end. If they are not the hardware and mechanical technical debt can be disastrous at integration time or when test operators actually use the equipment. The Software technical debts can come back at anytime, like an unpaid bank loan. Some software debts may bite you at integration time, or worst validation time, or even worse, while the equipment is in use on a production line. To keep it in financial terms, this would be like a bankruptcy. You could get past the technical bankruptcy but your reputation could be ruined for a while.

Also, the hardware technical debts can cause the software have a technical deficit if a hardware debt has to be fixed in software. It's like the hardware defaulting on a technical loan co-signed by the software. This seems to happen quite often.

Overall, when technical debt is taken on, it needs to be taken into account for future releases. If there is to much, it can cause a burden on your future of your tests. If it's not taken care of it could bring everything crashing down, potentially during validation or production.

The scary thing is if you don't realize you're taking on technical debt you have bigger problems and your people need to be trained or replaced.