Software Architecture and Test Engineering

After a hiatus, I'm back. I broke up with my girl friend which gives me much more time to do the things I enjoy doing and a lot less stress. Robotics, Artificial Intelligence, and my blog. Good nerdy engineering enjoyment.

The Test Engineering group I'm in is made up of mainly hardware people. A lot of them write software, some of them think they can write software, a couple of them really can write software. A software architecture is something required by the process but foreign to them. The bosses, who are mainly hardware types, want to make sure we've thought about the software before we start. Mainly because the hardware guys typically start writing software get to a point where it's a big mess and then patch as well as they can.

In the software past, the software architecture was called the preliminary design. Somewhere along the line someone came up with a paradigm shifting idea to rename it to software architecture. No matter what it's called, it's still just a high level view of the software job at hand.

Almost all of our tests are sequential, simple stimulate and check the UUT. TestStand is our standard test sequencer and we use it very simply, sequential testing with some very minor looping.

It can be a simple functional flow or a simple high level flowchart. We do set of similar tests followed by another set of tests, etc. The architecture is simple, non-complex by real software standards, and straight forward. Almost any thing should be acceptable as an architecture.

Occasionally the architecture will be more complex. Once this happens, then more advanced techniques are needed. More advanced flowcharts or especially UML would be a good candidate. Something to show interactions between different processors and different threads (although, where I work, threads are magic and unknown)

Any of this can prove to the bosses that we have thought about it, the flowchart is what they can understand. The people who are suppose to provide this are agonizing over what to provide and how to do it, taking a lot of time. The good software people are saying to let us know what format you want and we'll give it to you.

This is why we need people who can do software and not just hardware guys who code C. We know how to plan the software.

TestStudio vs. TestStand

I've been trying to learn about the Teradyne test sequencing product TestStudio. I wanted to compare it with NI's TestStand.

I'm going to say now that there is no comparison, it's not ready. TestStudio locks up so often that it's hard to focus on anything except the problems.

The main problem we're having is when we try to connect DLLs compiled by LabWindows CVI. About every third node (Test Step) that was set up to attach to a DLL would hangup TestStudio. The easier hangups only required TestStudio to be kill (via task manager) and restarted. Occasionally the PC would have to be reboot to continue working in TestStudio.

The Teradyne eknowledge site (the on-line help) was almost non-existent. I'm told by Teradyne that issues submitted by a company can only be see by people from that company. That means people can't see any product problems, issues, fixes, or comments . However, when we entered a problem in there on-line case tracking system, I did very quickly get a call to talk about it. Unfortunately I typically got excuses and that fixes we going to be in up coming versions.

My overall impression on TestStudio is that it's not ready. It's as if its an immature product only tested in on very specific uses. Quality control on the product seems to be left up to the user. The problem is that it's been around since about 2001 and a lot of these kinks should have been worked out.

Bottom line is that if you have a choice, go with TestStand. In my opinon TestStudio is not ready.

Sadness

I just want to let everyone know my Mother passed away last week from Pancreatic Cancer at the age of 86. She will be missed.

I'll continue my blog next week.

TestStudio vs TestStand

Teradyne has a software product called TestStudio that is a competitor with National Instruments TestStand. Since I've had my training class inTestStudio so I'm ready to review it. I waited for the training so I would have a better understanding.

I've noted before that the installation was difficult, each individual piece had to be installed separately so it kept me busy. I also had to install an older version of NI-VISA so that TestStudio would install and then re-install the latest.

The training was not very good. It was developed to be used with hardware available, which we don't have yet. This casued the instructor to skip several of the exercises and parts of exercises. A lot of the training material was vague in what you were suppose to do and there were quite a few error's in the material.

The product seemed very immature considering it's been around since at least 2001, possibly longer. During the training TestStudio kept poping up errors. One person had to change computers due to the unknown TestStudio problems. My TestStudio crashed and corrupted my project to the point I had to start over from scratch.

TestStudio is browser-based, of which I am not a big fan. But that is more of a personal preference and was one of the smaller issues. Overall, TestStudio has a lot of small Quirks. Some panels use "OK" buttons to save data, some the "X" in the corner and a few you use the File >> Save. A utility called Express Services, which is big in TestStudio, has to be specified at project creation, however, if a project is created without the Express Services option and it is decided Express Services are needed, you have to start over with a new project. Like many IE based programs, dialogue boxes can get covered up. TestStudio can appear to be closed eventhough it's not really due to a hidden popup. Also, the file browser does not always keep the location of the last browse.

One of the biggest annoyances is that when you add a new node (Test Step) an "are you sure" type of window pops up...every time! No option to not do have it pop-up is available.

Also, when passing parameters to DLL's from nodes (test steps) it is passed as char *. You can't just pass integers, doubles, etc. This limits the direct re-use of DLLs.

TestStudio's big feature appears to be setting up switch matrices between instruments and the UUT. It seems to do well at this but, since we didn't have hardware, we haven't had a chance to check this out.

Bottom line is, TestStudio is a poor second rate competetor to NI's TestStand. It's been around for several years, at least since about 2000, but seems to operate as a buggy and immature product.

Next week, I'm going to review more of TestStudio features (?).

NI Week is here!

It's finally the NI Week time of year and I am excited! I drove down early this morning. I found I couldn't sleep in spite of my need to get up early and be awake for the 3 hour drive down.

I've come on an Expo pass this year so my learning opportunities will be more limited than previous years but that doesn't dampen my desires to learn, network, and generally be nerdy...one of my favorite pastimes.

I am waiting for the Keynotes to start and thinking...WHO came up with the easy-listening, AC/DC re-mix of T.N.T. along with all the other sad remixes of good music. Basically, this my first disappoint in NI-Week in the last 3 years. I have a feeling I will get over it as the day goes on.

The NI Demonstrations were good but I really enjoyed Jeff K. future state of NI speech. I'm impressed evey year by his talks.

The Expo floor, of course, is very informational, lots of good information and potential uses. The unfortunate thing is, most of the information I gather will either be to new and different to be useful or will fall on deaf ears.

I am going to document what I find out and pass it on.

Again, this year, there is only one CVI demo and they won't let it run because it's to dangerous. It's a desructive test of a CD. Basically the CD crumbles. I guess CVI programmers are braver than LabVIEW programmers. I'm proposing a CVI controlled icamera and potato gun to take out the LabVIEW demo's...I don't think they'll go for it.

Loading TestStudio

I've tried to head into the comparison of TestStand and TestStudio with an open mind, be objective, filter out my biases. However, TestStudio has made this hard to do. Before I do a side-by-side comparison, I plan on learning a little about TestStudio.

We received and I loaded the TestStudio software along with some driver software for the hardware we were going to use. Once I began to explore TestStudio I looked in the help file for a Getting Started section, which I found. It was found to be lacking. It had short sentences like “Create One Library tree” but it had no links to how to accomplish this and no descriptions on how to do this. I searched around and figured this step out eventually, I'm pretty sure. But then the next sentence had the same shortcomings, short, to the point, with no "how" to do anything.

I decided to do it the good old fashion way, trial and error. Choose actions from the menu or right-click and choose actions from the list. I found how to create a step for an annoying “beep”...always fun. However, since TestStudio is browser based, I received a script error. I could not find the options menu item in the TestStudio browser, so I went to my Internet Explorer (IE) and Internet options to turn on scripting. Our IT department, in a fit of control and paranoia, had set no scripting and made it where ordinary users could not change it. While I'm not an ordinary user, I'm not going to do anything to my work computer that would get me in trouble...like give myself admin privs.

Luckily the “beep” node went in and worked in spite of the script error. Accomplishment! I decided to try attaching a node to a DLL. I got the same script error, along with a “Teradyne TestStudio DLL addin has encountered a problem and needs to close. We are sorry for the inconvenience.” message and an error pop up message with only a large red X, no text.

I gave up for the moment and was going to try to talk to the Teradyne guys to get some set up help. So I went back to work, fired up TestStand and immediately got a “Could not start” error. It worked last week, I loaded TestStudio, and now it doesn't work.

I had flashbacks to the Microsoft browser wars of the 90's when IE would cripple other browsers. I'm going to continue to learn, continue to try things, and try to continue to have an open mind.

NI Week...Woohoo

NI Week is almost upon us, it's the time of year to look forward to the new NI products and to network with all the NI people. I'm looking forward to the NI Week. I would be looking forward to all the seminar's and learning opportunities except that this year, some people at the place I work sees no immediate, short term benefit to conferences like this and AutoTestCon. Even in the long term it is hard to show measurable benefit. Since it's mostly secondary benefits, long term learning and networking, that is hard to measure, some groups will be poorly represented.

I am, however, going to to NI Week on an EXPO pass. I am excited about going and I am planning on making the most of my learning opportunities that I will have on my EXPO pass. I want to find out what various vendor's believe is the future of testing. I want to learn about the various products available to help us do our testing, I want to learn all I can...no matter what others think of conferences. I know there are learning opportunities, I know I am going to learn, I know I'm going to learn all I can, and I know it will help me in the future.

Overall, I'm excited about NI Week and all of it's opportunities.

Is Software Engineering relavent to Test Engineering

I've been trying to get a hold of Teradyne's TestStudio in order to compare it to TestStand. It's been about a month and a half since I've been trying to get a copy. Good news...sort of. I'm told my constant requests may have finially paid off. I'm told, Teradyne finally relented and is going to send us a preliminary copy of TestStudio. It may be in the building, it may be in the mail, so I haven't actually seen it. So, hopefully soon, I can do a comparison.

On to other things...

I've been thinking a lot about where Software Engineering fits into Test Engineering. While I understand the fact that it is an integral part of testing, I've noticed a prevailing thought, at least where I work, that software is secondary and trivial piece of the overall testing practice, "You can get anyone off the street to do the software." (qoute from a manager at work)

This thought was brought about when it seemed like I was the only one trying to get a preliminary copy of TestStudio. The tech lead dealing with getting TestStudio seemed unconcerned on getting the software tools while spending endless hours on getting exactly the right hardware. TestStudio was even picked as a software tool with almost zero input from people who deal mainly with software.

Due to the shrinking of many UUTs, a lot of tests are moving to software. Either the UUT is loaded with an Operational Test Program (OTP) or the UUT only has communications port, test software is becoming more and more important. People need more software expertise rather than less software software expertise.

I'm just frustrated knowing that most of the software decisions are being made by the software inept and I may not have a path up the technical ladder because of my leanings toward software.

If you're reading this, thanks for reading my vent.

Peer Review Tool

Last year I wrote a blog about Peer Reviews where I noted all the good peer reviews can do...as long as you do them. I believe that, when done properly, they can catch a lot of problems due to a "second pair of eyes" looking at the code. It also gets others at least sort of familiar with your code. That way, if you get hit by a beer truck, it's easier for someone else to pick up the code.

A peer review tool is an essential part of a groups set of tools. While you can keep track of defects on a piece of paper or in a document file, a peer review tool tends to work much better. On that note, SmartBear software has a peer review tool called CodeReviewer.

CodeReviewer - a simpler CodeCollaborator

SmartBear has a code reviewer sale going on, 5 seats of CodeReviewer for $5, follow the link to read more about it.

Yeah, this is sort of an ad, but lines up with my thoughts and beliefs on code reviews, so I'm putting it out there.

May your defects be trivial and you code be solid!

Little about TestStudio

I had hoped to start comparing TestStudio to TestStand but with all our efforts to actually get a copy of TestStudio or much information about it have been thwarted by the people we're trying to buy it from. All the T's and C's haven't been finished up so we can't get an advanced copy. And, as I've said before, NI and their 30 day trial period pretty much set's the standard.

I could compare the Tech Support...but wait, I'm running into the same problem. We can't get into the e-knowledge until we actually purchaced there product. While I'm not saying the makers of TestStudio has anything to hide, what are they hiding? Again, NI sets the standard of customer support.

While I do understand their need for money, I just don't understand the business model that operates in a way contrary to helping the customer.

I'm just venting my frustration, after all the years of dealing with NI, their support, their mature products, it's frustrating to run into roadblocks like these TestStudio roadblocks.

Comparing TestStudio and TestStand..Not!

Comparing TestStand and TestStudio at this point is...impossible. At the momment, I'm still waiting on the official PO to go to Teradyne so that they will be so kind as to give us the software we're buying. We have sent them 10% of the PO for long lead items but that isn't enough insurance to actually get the software.

It's starting to wear thin, I just want to learn a little about this monstrosity called TestStudio. I am assuming it is a steaming pile of ... code. Until I actually get to see a copy of it, I'm not going to change my opinion.

Once I get to play with TestStudio I really am going to compare it to TestStand. Okay, I do have one bit of pre-knowledge of TestStudio that I can compare to TestStand. Any parameters passed out of TestStudio to a DLL or other code is only allowed to be passed in Character strings. TestStand does it right and passes the value in the form it needs (i.e. integers, doubles, pointers, etc).

Last week the score was TestStand 2, TestStudio -2. The score is now TestStand 3, TestStudio -3 for TestStands ability to pass parameters correctly and Teradynes unwillingness to allow people who are going to use it to actually use it.

TestStudio, Hmmm...

It's been a while, I've doing a lot of playing with my Genetic Algorithms program, writing a Neural Net program, and studying Chaos.

First, Congratulations to the Wendy, former Queen of CVI. She's moved on MIT, graduate school, and opportunities up there. Way to go!

Second, Welcome Adri K, the new Queen of CVI and all NI things Text. Everyone needs to go to NI Week and say hi to her or the NI CVI community.

Now to the meat of the subject. TestStudio, not TestStand but TestStudio.Teradyne's offering in the test sequencing arena. It seems to be cheap knock off of TestStand put out by Teradyne. This is the product we'll be using on our next test set. It was not my choice or the choice of anyone who will have to develop on it.

Since this choice has been made for use and I've been told to accept it, I want to write about comparisons between TestStudio and TestStand.

First comparison...none. Teradyne, who is apparently very big on keeping the purchaser of the product in the dark, will not allow us to have an evaluation copy or access the support site until they get their PO.

We are told it's great, or at least the only product that will run its proprietary switch manger. We're also told the switch manager is a .dll. .dll's will run under TestStand, so why is TestStudio the only way to run the switch manager?

I guess I'll have to wait until we actually get the software before I can compare it. Since TestStand allows a 30 day evaluation and TestStudio doesn't, plus we can't get to the support. Is Teradyne's product so bad they can't even allow people to evaluate it?

TestStand 2, TestStudio -2. Negative since I can't even get to any documentation except Propaganda sheet.

Post Robotic relaxation

Everything is calming down now after the FIRST Championships. I'm trying to capture some lessons learned and keep some thoughts and idea's for next time. If anyone has any fund raising idea's I'd like to hear about them. The money was probably the biggest hurdle.

We, team 704, does have pictures from Atlanta. One thing I'm doing now is buying a new and better camera with stabalization. A lot of my pictures came out fuzzy.

Below are some pictures on the team 704 odyssey that, mostly, I took. Also another link to the Google pic

FIRST Championships

This has been an exhausting weekend at the FIRST Robotics Championship in Atlanta Georgia. We arrived Wednesday night but it all started early Thursday morning. We got in un-crated the robot. We ran some practice rounds and started scouting teams as well as warming up the Robot.



More fun was on Friday morning...the real thing started...qualifying rounds. We started out strong, three wins in a row. Unfortunately the win streak didn't last. We had some poor luck of the draw, our alliance was with some weaker robots and we were against 3 robots that all wound up in the final rounds. Our alliance lost. We lost one more Friday when two robots were blocking us and our alliance robots weren't coming to help break us loose.

Saturday looked promising at first, at least until we lost our first round...then our second round. There were no good explanations except not much cooperation between alliance teams.

Alas, at the finals team selection, our team number was not called. The race for the Championship was over for us.

It was an exciting weekend. The high school kids had a lot of energy and some of it translated to me. I had a lot of energy...more or less. Over all it was fun and tiring, it's most certainly a labor of love.

Headed for the FIRST Championships!

Team 704, the SGP Robotics club, is headed from Atlanta GA for the FIRST Robotics Championships! This will be SGP's first trip to the big game!



We have the airplane tickets, another team will be taking some of our equipment to Atlanta in their trailer, and the excitement is mounting. I'm very excited! I contacted the Lockheed Martin internal newsletter about doing a story on the FIRST Championship. It's an event where there are potentially hundreds of future employees!

I was so excited I pulled out my NXT robot and started playing with it. I've also updated my personal web page with the robotics stuff from Dallas and Lone Star tournaments.

Building Robots to take over the world

I apologize for not writing lately. I've been busy lately working with the South Grand Prairie Robotics team. I tell everyone it's to build robots to take over the world but it's really to build a robot for the FIRST Robotics Challenge.



The FIRST Robotics Challenge is (FRC) is a great organization where high school kids are given a game, a “kit”, and 6 weeks to build a robot to play the game.

The game is called Lunacy. Basically, you try to put your balls in your opponantes trailer. Every robot has a trailer behind it and so while you're trying to dump in someone's trailer, they're trying to dump in someone elses trailer.

The kit is a set of parts to build a basic Robot with enough programming to have a joy stick control it. From there, each team needs to figure out how to dump balls in opponents trailer, what to do in autonomous mode (the first 15 seconds of the match)

FIRST is a national challenge sponsored by a lot of technical companies who will need engineers in the future and want to get the kids interested now. They are a very organized group who have regional robot tournaments in various parts of the country.

The team I'm working with is Team 704, associated with the South Grand Prairie High school Robotics team. Each team, once a number is assigned, keeps that number and the numbers aren't re-used. If a team disbands, the number is no longer used. Team 704 has been going since 2001 sponsored by physics teacher extraordinaire Phil Harris.



I'll write more later. I just want close by saying that high school kids who are even vagually interested in robots should go to usFirst.org and find a team in your area. If you're an adult (who is a kid at heart) you can help mentor a team. (Even though they're using LabVIEW)