One thing I've noticed is that many test engineers, for coding and design, just hack something together so it can be thrown onto the test set. Sometimes it’s because of schedule pressure, sometimes it just more fun to touch hardware, sometimes it’s because it’s EE’s doing the coding with only one software class worth of training. In any case, this has got to stop!
Test Engineers (TE) doing software need to have some software discipline. They don’t need the same discipline as tactical or man-rated software development needs, but Test Engineers do need software discipline none the less. TE's need to do some unit testing, some stress testing, some fault path testing, Diagnostics levels, some issue like that.
Many TEs use the excuses of “our software is different” or “Our software is to hardware intensive” to avoid code testing. I say any 10 year old can give excuses, what about results? What is the quality of your software?
As I see it there are two basic types of software testing, internal testing and external testing testing.
First the external testing or unit testing. In the software world there is quite often a test group. The software developers develop software and throw it over the wall to testers. In the Test Engineering world, there's not always that luxury. That's the reason to have software self discipline. There's more to external testing but for now I'm limiting myself to unit testing.
Definition: Unit Test is the testing of a module or a logically grouped set of modules where inputs and outputs, limits, stress, path test occurs.
To test a unit(s) it can be as simple as a function calling the unit(s) being tested over and over with various inputs and then checking the output. If NI's TestStand is in use then a sequence is set up to call the module(s) under test and the output checked.
One tool that I use quite often is called CUnit. It's a unit testing framework and there is a whole range of _Unit type of frameworks. JUnit for Java, CPPUnit for C++, xUnit for MS Visual stuff (I.e. NI Measurement Studio) and some others I can't think of right now. A lot of this can be found at SourceForge.org, a great source of open source software.
Since test software is quite often hardware intensive, sometimes some hardware simulation is needed, and I use the word “simulation” in the weakest form. It's as simple as creating a dummy routine to replace the hardware instrument driver that returns the data you want to test with. I.e. good data, erroneous data, stressed data, etc.
To test, the driver routine would call the module being tested. The module being tested calls the driver simulation which returns a predetermined set of results which allows the module being tested to return a predetermined result. If the returned result is really what you expect, the test worked. Otherwise, the module needs to be fixed.
Overall, the point I'm making is not to hack something out and throw it on the hardware. There are options, whether you've thought of them or not. By The Way, this isn't a comprehensive list of ways to unit test. Use your imagination, I know you have one, you're an Engineer!
I just want people to step out of their little box and think “How can I make my code better?” and “What can I do to keep from giving incorrect results or give errors?”
Note: Some of these test methods are the part of Agile development called TDD, Test Driven Development (TDD). It just means your code should be driven by the tests. I didn't want to scare anyone with the word "Agile."
Sunday, April 27, 2008
Friday, April 18, 2008
Zephyr - test management tool
Yea, this blog is about testing, usually about NI stuff, but competition is good for us all. Saying that, someone e-mailed me about a test management system called Zephyr. I read about it and it sounds like it has a lot of potential.
Zephyr is a complete test management system. It has collaboration tools, customizable dashboards for instant status, Test Desktops, with built in metrics, and test execution capability with defect tracking.
You can read about it's list of product features here and see an overview video here. There's also a limit person trial version here. It has some very good looking interfaces.
I have only read about the product and have not tried it out but it sounded like it had some very useful features. Since I have not actually used it I can only give you my impressions. Some things I liked were:
- Central repository for documents or at least links to the documents.
- Test/QA/Management Collaboration
- Instant update of test status
- It sounds like it was for Agile development with all the communication and collaboration, but I'm sure it can be used for any type of development.
The ability to manage widely distributed test systems is impressive and massively useful.
A couple of big questions, at least for me.
One, how well it plays with TestStand and other NI products. NI is the big dog on the block for test software and electronic, at least at Lockheed and to get a foothold it would need to complement our NI abilities.
Two, We have a lot of legacy code and equipment from NI that we don't want to lose. Since our previous tests cases are in TestStand written in CVI or LabVIEW, how well you can re-use your old tests? How well will it interact with the legacy code?
I'm probably download and try out the trial version but it would be in my spare time. Unfortunately, that wouldn't be until the end of the year.
All-in-all Zephyr looks like it has potential and I would like to hear from anyone who tries it.
Zephyr is a complete test management system. It has collaboration tools, customizable dashboards for instant status, Test Desktops, with built in metrics, and test execution capability with defect tracking.
You can read about it's list of product features here and see an overview video here. There's also a limit person trial version here. It has some very good looking interfaces.
I have only read about the product and have not tried it out but it sounded like it had some very useful features. Since I have not actually used it I can only give you my impressions. Some things I liked were:
- Central repository for documents or at least links to the documents.
- Test/QA/Management Collaboration
- Instant update of test status
- It sounds like it was for Agile development with all the communication and collaboration, but I'm sure it can be used for any type of development.
The ability to manage widely distributed test systems is impressive and massively useful.
A couple of big questions, at least for me.
One, how well it plays with TestStand and other NI products. NI is the big dog on the block for test software and electronic, at least at Lockheed and to get a foothold it would need to complement our NI abilities.
Two, We have a lot of legacy code and equipment from NI that we don't want to lose. Since our previous tests cases are in TestStand written in CVI or LabVIEW, how well you can re-use your old tests? How well will it interact with the legacy code?
I'm probably download and try out the trial version but it would be in my spare time. Unfortunately, that wouldn't be until the end of the year.
All-in-all Zephyr looks like it has potential and I would like to hear from anyone who tries it.
Apologies
I want to apologize. I haven't been blogging much for a while due to my Mom being, first in the Hospital, and now in a nursing home.
Things are settling into more of a routine so I should be able to blog more, but she does come first.
Thanks
Things are settling into more of a routine so I should be able to blog more, but she does come first.
Thanks
Wednesday, March 26, 2008
Design Tools
From what I've seen, test engineers don't always do a good job of designing software. They just sort of code away until it's done. And if there is any complexity to the design, they code even harder until it's done. Not a lot of real design is done.
There are a lot of simple design methods that can be done for a better test set. Just a simple flow chart to layout the design is good at a minimum. Something like MS Visio or any drawing program can used.
Another design tool is to do what is called swim lane charts. This is where each separate item (i.e. UUT, Test Set, Other Computer) gets a lane and every time something is required of that item, communication to or from that item, some process/decision/operation block is put in that lane. It shows the interaction between each of the different items and the operation each item does.
UML is another design tool, especially when doing object orienting programming. I read about UML but not done design with it so I don't have a good grasp of it. I know it's a way of displaying objects and showing the interaction between them. The software engineers tend to use UML for their designs.
State diagrams are also really useful for showing transitions between states.
My main thought on all of this is don't just code away with little or no fore thought. Use your tools for a better test set software design, it will work better in the long run and be more maintainable.
There are a lot of simple design methods that can be done for a better test set. Just a simple flow chart to layout the design is good at a minimum. Something like MS Visio or any drawing program can used.
Another design tool is to do what is called swim lane charts. This is where each separate item (i.e. UUT, Test Set, Other Computer) gets a lane and every time something is required of that item, communication to or from that item, some process/decision/operation block is put in that lane. It shows the interaction between each of the different items and the operation each item does.
UML is another design tool, especially when doing object orienting programming. I read about UML but not done design with it so I don't have a good grasp of it. I know it's a way of displaying objects and showing the interaction between them. The software engineers tend to use UML for their designs.
State diagrams are also really useful for showing transitions between states.
My main thought on all of this is don't just code away with little or no fore thought. Use your tools for a better test set software design, it will work better in the long run and be more maintainable.
Monday, March 17, 2008
Test and a lack of understanding of software.
“Software is a necessary evil”, a statement I’ve heard several times made by a test engineering manager. Other of my favorite statements by managers are “Test development is the development of hardware…and then you do software” and “We have no software problems” as everyone developing software laughs or shakes their head.
It’s absurd to think of test development in one dimension, the hardware dimension. If you’re involved with CMMI, Processes, etc. you’ll know that it all started with problems with software, or the mis-understanding of software and it's development. Manager’s didn’t know how software worked, or what it takes to develop it. Software practitioners were just “winging it” and learning by tribal knowledge. Projects were brought down by software problems.
Test engineers developing software for ATE need to at least come into 90’s and develop software using at least some standard software engineering approaches. Simple things such as keeping the requirements. Test requirements very often involve hardware and software and not exclusively one or the other. That’s fine, you still need to understand the software portion of the requirement.
The requirements need to be understood and traced up to any other higher requirement or derived requirement of something in the UUT that is required to test. This can be done in a tool, such as requisite pro or DOORS, or in an Excel spreadsheet. But you need to know what you’re testing.
Other tools that can help is a tool called Requirements Gateway by NI. I haven't used it personally, but it seems like a useful tool.
More test tool stuff to come
It’s absurd to think of test development in one dimension, the hardware dimension. If you’re involved with CMMI, Processes, etc. you’ll know that it all started with problems with software, or the mis-understanding of software and it's development. Manager’s didn’t know how software worked, or what it takes to develop it. Software practitioners were just “winging it” and learning by tribal knowledge. Projects were brought down by software problems.
Test engineers developing software for ATE need to at least come into 90’s and develop software using at least some standard software engineering approaches. Simple things such as keeping the requirements. Test requirements very often involve hardware and software and not exclusively one or the other. That’s fine, you still need to understand the software portion of the requirement.
The requirements need to be understood and traced up to any other higher requirement or derived requirement of something in the UUT that is required to test. This can be done in a tool, such as requisite pro or DOORS, or in an Excel spreadsheet. But you need to know what you’re testing.
Other tools that can help is a tool called Requirements Gateway by NI. I haven't used it personally, but it seems like a useful tool.
More test tool stuff to come
Monday, March 10, 2008
Test Engneers
Most people dealing with Automated Test Equipment (ATE) call themselves “test engineer’s” (TE) or test people. However, since there is typically no degree’s in test engineering, they get degrees in Electrical engineering or software engineering or computer science and engineering and then are recruited in the “test group”. I’ve seen most TE’s come out of the group with EE degrees, a lot of SW Engineers don’t like to touch hardware. But then a lot of EE’s feel that doing software makes them dirty or is FM (Freaking Magic) and only took minimal, how to code classes in college.
Since most TE’s are out of EE the software portion of ATE is “hacked” out (the old-school meaning of hack, to bang out code until it works) the code is less than optimal. In other-words, it works for the normal case and little else.
Some people who have software degrees or have been doing software for a long time work toward having more software discipline in there ATE code, but it still tends to be lacking in most area’s. Software tools tend to go unused or limited use, some believe an IDE is all that is needed.
There’s more to developing software than just an IDE and some time. Tools are available to help develop code. I think software tools aren’t used in test due to a lack of understanding, lack of knowledge, and lack of patience.
Since most TE’s are out of EE the software portion of ATE is “hacked” out (the old-school meaning of hack, to bang out code until it works) the code is less than optimal. In other-words, it works for the normal case and little else.
Some people who have software degrees or have been doing software for a long time work toward having more software discipline in there ATE code, but it still tends to be lacking in most area’s. Software tools tend to go unused or limited use, some believe an IDE is all that is needed.
There’s more to developing software than just an IDE and some time. Tools are available to help develop code. I think software tools aren’t used in test due to a lack of understanding, lack of knowledge, and lack of patience.
Sunday, March 2, 2008
Time to come into the 90s
I apologize for not writing in a long time. I've had family illnesses, Mom going into nursing home and all the financial items that come along with that...things like that. It's not an excuse, it's part of life. I asked myself, what is important to me...there was no contest, it was family.
While real-life has gone on personally, work-life has continued, too, since I like to eat and live indoors. One of my biggest pet-peeves that has been affecting me is people are stuck in "the way we've always done it." It's really just fear, the new way of doing things may affect their job, may affect the bottom line, they may not really understand new ways of doing things.
Have you ever thought where we'd be without evolution? Whether you believe in Creation or Evolution (I personally believe in Creation but we Evolved from there) businesses, business practices, business methods evolve. That's why some business that don't change with the times die-off (ex. Braniff air-lines and Remington Rand Typewriter company) and others change, evolve and continue to thrive (IBM and Harley Davidson Motorcycle company). People in all businesses need to learn to change and evolve or die off. People within the business need to evolve to keep the businesses evolving.
The people who are stuck in the 70s and 80s need to come into the 90s or their businesses will stay in the 70s and 80s. They need to change but I wouldn't want to push them to fast by trying to get them to come into the 2000s.
They need to evolve, try to get up to date in their thinking,
New ideas need to be tried, at least ideas like Object Oriented Programing (OK, that's not new but it is to them), Agile programming, using ASICs and FPGAs. At least get past vacuum tubes and transistors. All this is not magic, it's just evolution, or coming into the 90s from whatever decade you're stuck in.
It is a disservice to the people who work for the less evolved, too. If you want to compete in this world you at least need to be involved in this world.
I don't know how to get these people up to date, or even close to up to date, but for the businesses sake and their sake, it needs to happen.
While real-life has gone on personally, work-life has continued, too, since I like to eat and live indoors. One of my biggest pet-peeves that has been affecting me is people are stuck in "the way we've always done it." It's really just fear, the new way of doing things may affect their job, may affect the bottom line, they may not really understand new ways of doing things.
Have you ever thought where we'd be without evolution? Whether you believe in Creation or Evolution (I personally believe in Creation but we Evolved from there) businesses, business practices, business methods evolve. That's why some business that don't change with the times die-off (ex. Braniff air-lines and Remington Rand Typewriter company) and others change, evolve and continue to thrive (IBM and Harley Davidson Motorcycle company). People in all businesses need to learn to change and evolve or die off. People within the business need to evolve to keep the businesses evolving.
The people who are stuck in the 70s and 80s need to come into the 90s or their businesses will stay in the 70s and 80s. They need to change but I wouldn't want to push them to fast by trying to get them to come into the 2000s.
They need to evolve, try to get up to date in their thinking,
New ideas need to be tried, at least ideas like Object Oriented Programing (OK, that's not new but it is to them), Agile programming, using ASICs and FPGAs. At least get past vacuum tubes and transistors. All this is not magic, it's just evolution, or coming into the 90s from whatever decade you're stuck in.
It is a disservice to the people who work for the less evolved, too. If you want to compete in this world you at least need to be involved in this world.
I don't know how to get these people up to date, or even close to up to date, but for the businesses sake and their sake, it needs to happen.
Subscribe to:
Posts (Atom)
