SLTF Consulting
Technology with Business Sense

Adequate testing means more than just checking against the specification

Scott Rosenthal
July, 1995

Testing an embedded system involves many more issues than the testing of a PC program. Not only does the embedded system contain a program analogous to the PC, it must also contend with startup code, hardware-control issues and asynchronous power interruptions. Hence, embedded-system testing requires not only testing to a requirements document, but it also demands of the tester an innate system knowledge, skepticism-and a healthy dose of luck. No matter how detailed the document, my experience is that this process finds few system problems. If you think about it, the reason for this truth should be obvious. Consider that any designer worth his salt develops systems to address the design's requirements. Who ever set out purposely to design something that didn't meet the system's requirements?

Therefore, the most likely places defects show up are in a system's "undocumented requirements." For example, I once designed an analyzer for use in a grain elevator. One of these instruments came back for repair after having been in the field for a while. Inside the instrument was a soft rubber wheel for moving grain. Apparently, this soft rubber was also ideal for mice to use in whatever it is mice do with soft rubber. The wheel was chewed apart, and we had to switch to a hardier wheel. The instrument met the requirements testing, which apparently didn't take into account all environmental conditions.

Xilinx revisited

Proving that these problems don't ever go away, in a previous column (Ref 1) I detailed a problem with a Xilinx chip powering up into an illegal state from a brownout condition. Although none of us were smart enough to anticipate this type of problem, any good tester will tell you, once you've identified a weakness in the system, there's a good chance that additional problems are hiding down the same path. With that lead in, let me describe a really neat tool we used with Xilinx debugging, fixing and testing.

First, a quick review of the problem: In a brownout condition (caused by quickly switching the instrument Off then On), the Xilinx chip wouldn't reset properly unless one of two conditions was met: either Vcc has to drop below ~0.1V or a specific timing sequence had to occur. The tough problem was how to devise a test that would guarantee that the Xilinx problem was really fixed. That's how I came across a simple tool to verify my fix.

A recent article (Ref 2) describes this tool, called Poc-it from MicroTools Inc (Simsbury, CT (800) 651-6170). It switches AC power On and Off either directly or through a relay. In addition, to close the loop, the instrument can also count the occurrences of two TTL-level inputs representing good and bad results. What you end up with is a simple tool (aren't the best ones simple?) that inexpensively allows you to design a true test of the Xilinx problem.

I connected four UUTs to the instrument's switched AC outlet and set it to quickly cycle the power- 10 msec On and 25 msec Off-and started it on its torture testing. Within 100 cycles, all four instruments had their Xilinx chips fail to reset properly! Great, I now had a definitive test of the problem. I then installed a patch board with a PIC processor for controlling the Xilinx reset and reran the procedure. I stopped it after 20,000 cycles and had no failures! The Poc-it allowed me to run a controlled test, which not only proved that it could detect failures, but it proved the fix.

What's next?

After seeing the Poc-it's success, I thought of other ways to automate our testing. I put together an extensible automated tester that allowed me to design tests around system requirements while adding tests to challenge the system in unexpected ways. The tester also gives me traceability, which is important in today's regulatory environment.

The automated tester, like Poc-it, is simple in concept and implementation. When designing a test system for an embedded device, obviously the first thing to do is figure out how it can stimulate the UUT. One of the more-obvious places is through its keypad. For generating simulated user inputs, three basic options are available: implement solenoids to physically press the keys, use a serial port to simulate keypresses or wire relays in parallel with the keys on the keypad. I went with the relays because this option requires no changes to the UUT software, and wiring the relays is relatively easy on most embedded systems. With the relay in parallel with a switch, closing the relay stimulates the hardware in the same fashion as pressing a key.

The UUT for which I originally designed the test system sports a 16-position keypad. It also has four other digital inputs, including a Door Open switch and a Lamp On signal. To stimulate this device, I purchased a CIO-ERB24 relay card from ComputerBoards Inc (Mansfield, MA (508) 261-1123), which provides 24 Form C relays. I controlled the board with an old laptop computer (no hard drive, single floppy) using CBI's PPIO-DIO24H box connected to the parallel printer port. I then wired the relays in a matrix arrangement that matched the UUT's keypad and attached a passthrough connector on the end of the cable. With it I can remove the keypad cable in the UUT, plug the relay cable into the UUT and reattach the keypad cable to the relay cable. This arrangement gives me the option to press the keys with my fingers or simulate pressing the keys with a relay. I then connected the remaining relays to the other digital inputs to simulate their functions. Now, with just a bit of software, I had the beginnings of a powerful test system.

When putting together a home-brew system, basically you have two choices: First, assume you'll never need this setup again and not design an extensible system. We've all been guilty of this crime-justifying it with complaints that there's no time or budget to do anything else. Whatever the excuses, the result is a monolithic program that's fragile and difficult to change.

The alternative is to take a few hours and craft an extensible program that you can use for a long time. For my test application, I started with one of my monitor programs (see Refs 3 and 4) and added a new command to activate a script processor. To control the relays and specify test sequences, I designed a simple script language. It's not fancy, but it does allow an easy path to develop new test scripts. Also, because I wrote the code, I can add new elements to the "language" any time. The program design is small and written in C for easy future extensibility.

The language concept is simple. I created an object type (see Listing 1) with various properties, including the instance name, the script string code and the relay to use.

Listing 1-Testing object
typedef struct {
char *Name; /* name of this data */
char *Code; /* character code for test script */
int Default; /* the default value on program start */
int RelayPort; /* the relay port to use */
unsigned int Mask; /* the mask to use for the I/O port */
int (*Function)(void *ObjectPtr); /* the function to execute */
char *StatusMsg[2]; /* status message for 0 and 1 */
/* the program fills the following: */
int CurrentValue; /* the current value of this data */
long Value; /* the value of the object */
} TObject;
/* functions for relay and time delay control, and keypresses */
int RelayFunction(TObject *ObjectPtr);
int TimeFunction(TObject *ObjectPtr);
int KeyFunction(TObject *ObjectPtr);
/* always present, the delay object */
TObject Wait = {"Time Delay", "T", NO, EOF,
0x08, (void*)TimeFunction, "", ""};
/* F1 key */
TObject Key1 = {"Key 1", "k1", NO, PORTC,
0x20, (void*)KeyFunction, "", ""};

I then made instances of the object for each function I wanted to control, for example Door, Key1 and Wait. The script language itself is also simple. It consists of a command and a parameter. The command is just the code for an object instance. Again, it could be Door, Key1 or Wait. The parameter is some condition for the command. For example, with Door, 0 could mean open the door and 1 could mean close the door. For Wait, the parameter is the delay time in milliseconds. For Key1, the parameter indicates how long to pause after pressing the key. Another idea might be to indicate how long to hold down the key. Now, it's easy to string these object instances together to create a test script (see Listing 2).

Listing 2-Typical test script
; all test after a semi-colon are
; comments
k1,0 ; press key 1
k2,1000 ; press key 2 and wait 1
; second
lamp,1 ; turn on the lamp
t,10000 ; wait 10 seconds
k3,0 ; press key 3
; this sequence will now repeat
; indefinitely

The script processor works by reading the script file one line at a time and processing it. At the end of the list, it starts over. In the future, I plan to add features including digital inputs, feedback and script linking. Having written the code myself and keeping it object-oriented, I can easily add new objects as needed. Hence, you can use the same program fundamentals for testing different systems.

Other benefits

In addition to the obvious advantages of automated testing, this effort also offers other benefits. For example, it uncovered a number of subtle defects in my compiler, but the one that might strike many developers is a caution about using a heap with malloc(). The function malloc() allows a program to grab space on the heap. Conversely, the function free() returns this space to the heap. The problem I had was that after calling free(), the next malloc() started looking for available heap space at the last location freed with free() instead of at the beginning of the heap as I would've thought. Hence, depending upon the sequence of key presses, the program would at times severely fragment the heap, which in combination with the free() idiosyncrasy gradually stepped the pointer to the top of the heap higher in memory until it ran out of room. The result was that the UUT ran out of memory even though there was plenty of unused memory available. Once I figured out what was happening it became clear that the only way to fix the problem was to rework the operation of the malloc() function, a process that would be difficult at best-even if I could sweet talk my compiler vendor out of the source code for the built-in functions.

Finally I decided that since the only way I found this problem was through a torture test, the error was an acceptable situation. However, regardless of the outcome, I'm still ahead because I now know about a potentially dangerous "quirk" in my compiler, and I have a system to automate the testing of my products. PE&IN

References

1. Rosenthal, S, " It sometimes takes a microcontroller to boot up an FPGA," PE&IN, Nov 1994, pgs 75-78.

2. Japenga, B, "If it's not tested it doesn't work!" PE&IN, Apr 1995, pgs 53-56.

3. Rosenthal, S, " For digging out hardware bugs, monitor programs fill the bill," PE&IN, July 1994, pgs 56-58.

4. Rosenthal, S, " Monitors are valuable, easy to use and create," PE&IN, Sept 1994, pgs 73-78.

Adapted from an article that appeared in Personal Engineering & Instrumentation News.