Introduction §
Imagine you needed to run firmware unit tests with the actual device in the loop, but nothing quite does it the way you want. This is where you decide to make a 1-click automatic HIL unit testing software rig!.
I know for a fact there are existing solutions for problems similar to this, but none of them supported mock serial streams to emulate another device in the loop, nor did anything work on PlatformIO in an automatic way.
What I needed was a solution that:
- Worked with PlatformIO
- Compiled and deployed code to the board automatically
- Ran test code with support for mock serial streams to emulate another device (I had an LTE modem connected with AT commands)
- Allowed for serial capture from a serial analyser (Saleae in this case)
- Run a bunch of tests sequentially, one after another
- Displayed the results in a PlatformIO-compliant way
- Catch all the bugs!
This is all hosted on Github!
Implementation §
Turns out, PlatformIO has it's own test runner framework which makes the deployment and output capture easy. AND, Saleae has an API for it's logic analysers.
Perfect.
I used both of these to build a python runner to startup the logic analyser software, wait for the output to read in, then parse it and return it back to PlatformIO.
Simply:
def create_all_test_cases(self) -> List[TestCase]:
test_cases = []
# Get lines of serial output
timestamped_lines = self.get_timestamped_lines()
for index, line in enumerate(timestamped_lines):
if re_match := re.search(TestParser.TEST_REGEX, line["line"]):
test_cases.append(self._generate_test_case(
timestamped_lines, re_match, index))
return test_cases
Here is an example test where the characters "10" are injected into the fake serial stream (Like an LTE modem talking over serial) and then checked in the serial stream output.
void square(Stream &io)
{
io.print("value? ");
int x = io.parseInt();
io.print(x);
io.print("*");
io.print(x);
io.print("=");
io.println(x * x);
}
test(exampleTest2)
{
MockStream ms;
ms.input.print(10);
square(ms);
Serial.print("MOCK_OUTPUT: ");
Serial.println(ms.output);
assertEqual(ms.output, "value? 10*10=100\r\n");
}
These C++ tests are compiled and flashed to the board over JTAG, and then read by the logic analyser
I could not find test output for the mock stream. I am writing this 2 years after finishing the project
# Example: timestamped_lines
# {'duration': 0.858, 'line': 'TestRunner started on 2 test(s).'},
# {'duration': 0.002, 'line': 'Test exampleTest1 passed.'},
# {'duration': 0.004, 'line': 'AUnitPlatformIO.ino:12: Assertion failed: (3) != (3).'},
# {'duration': 0.002, 'line': 'Test exampleTest2 failed.'},
# {'duration': 0.003, 'line': 'TestRunner duration: 0.010 seconds.'},
# {'duration': 0.007,
# 'line': 'TestRunner summary: 1 passed, 1 failed, 0 skipped, 0 timed out, out '
# 'of 2 test(s).'}
# Example: timestamped_lines in text form only
# TestRunner started on 2 test(s).
# Test exampleTest1 passed.
# AUnitPlatformIO.ino:12: Assertion failed: (3) != (3).
# Test exampleTest2 failed.
# TestRunner duration: 0.010 seconds.
# TestRunner summary: 1 passed, 1 failed, 0 skipped, 0 timed out, out
Then all of this data is presented nicely in PlatformIO's test output window
pretend this is the output of this program - once again I do not have enough screenshots

source PlatformIO
- Because it's flashed over JTAG, the ESP32's main serial line over USB is free. That could be used as the main test output stream if ever needed.
- Another bonus of this, is that I routed my GPIO outputs (lights, relays, etc) to the serial analyser, so I can add tests for my GPIO pins to be at certain levels and capture that as well.
- Because I opted for a CSV intermediary from the logic analyser, the entire test report is stored and timestamped byte for byte if I needed to analyse it later.
Conclusion §
I wish I had more screenshots.
But nevertheless, I was so pleased to be able to make a testing rig that I could test the entire hardware stack of my project with nothing more than the DUT, programmer board and a logic analyser. Small enough that I could do it on the bus if I needed to.
I needed to test that my device could talk to an LTE modem over serial with AT commands, and this system completely automated it. It was a big step up for me in terms of repeatability and reliability for the testing.