The avrgirl project: manual testing

green baseball cap with the letters 'AVR' written on it

Publishing your code for others to use is a scary, but often rewarding accomplishment. It can be really exciting to witness peers and strangers incorporating a library of yours into their own projects.

Being an open source maintainer brings a lot of responsibilities to the table. Managing issues, pull requests, and emergency bug fixes can be a lot for one person to deal with. If your project cannot rely solely on automated testing, this can be a real burden. This is particularly true in the hardware library scene. I have experienced all sorts of weird bugs pop up when the same code worked the day before. Things like operating system changes and runtime upgrades can expose the fragility of how some hardware libraries work, more readily then something, say, a web based library.

I have run into this recently when publishing and maintaining libraries falling under the AVRGirl project, which I started working on in early June last year. While I have learned a lot during the process, there is still a lot more for me to know. These gaps in knowledge often manifest into slightly different experiences across users' operating systems and versions of NodeJS, which is what all AVRGirl packages are written in. When I say 'experiences', what I really mean is bugs, of course ;)

When I write unit tests for AVRGirl, the most notorious parts of working with hardware get stubbed out. This is stuff like serial interfaces, and responses from hardware are faked. This tends to be where a lot of things go wrong, and where compatibility issues rear their ugly head. In other words, my code needs to know about all of the edge cases and fragilities, and provide for them. Simulating this kind of stuff in unit tests is not at all practical and introduces some tech debt and readability issues for potential contributors. It also doesn't help me stay on top of OS and NodeJS version changes, and their effect on my libraries/code.

The problem

So let's break down exactly what all this means. We'll focus on one library in particular, which is AVRGirl Arduino. AVRGirl Arduino flashes precompiled code to a myriad of supported Arduino boards. What does this look like, in testing terms?

  1. 16 supported boards
  2. OSX, Windows, Linux (and all modern releases/versions within)
  3. NodeJS versions 0.10, 0.12, 4.x, and 5.x (currently alpha testing 5.x so we won't count it)

This works out to be approximately 144 integration test cases, if we're just counting operating systems as giant buckets, and not splitting them out to things like OSX Yosemite, El Capitan etc etc.

What I'm getting at here, is that I cannot sit down and manually test everything alone for both releases and environment changes/additions. Well, that and hold down a full time job and occasionally not code from time to time :P We all need time away.

AVRGirl Arduino was picking up a bit of interest a couple of months after releasing the first version, so I decided to reach out to potential testers by opening the following issue.

screencap of issue #4 under avrgirl-arduino github repository

It was a quick explanation of what I wanted and why. I added the 'help wanted' label and thought that would be it. After mostly radio silence, I started thinking long and hard about why I've had no shortage of developers opening pull requests with new features and bug fixes, but not a lot of feedback around if the library actually works as expected in these 144 different combinations. No one was chomping at the bit to sit down and plug in a bunch of boards and run some quick set up code.

Wait, but that was just it! I had not provided an easy way to manually test AVRGirl Arduino. Sure, I'd written some vague instructions in a Github issue, but that's not a very good experience for people wanting to help. I identified a few things that would need to be better if I was going to ask others to manually test for me:

  1. Manual testing is boring for most people
  2. You don't get a 'little green square' contribution reward for manually testing
  3. People who are new to hardware probably had no idea where to start even if they wanted to help. My issue made a lot of assumptions about others' knowledge, which is not inclusive.
  4. There were too many steps involved and too many bridges to cross in order for someone to invest in helping. People's time is valuable and I was not respecting that.

A proposed solution

I decided to write a small app to run within AVRGirl Arduino, its only job being to allow an easy way to manually test AVRGirl Arduino. Embedding it within the library itself allows for great portability, and no additional installation of anything else needed should someone like to help test. The downside is that it introduces more dependencies to AVRGirl, which carries more risk for issues. I tried to keep it relatively lightweight to mitigate this.

I called it AVRGirl Test Pilot, and connected it to the CLI tool within AVRGirl Arduino. The 2 steps required to start testing are:

  1. With NodeJS installed, run npm install -g avrgirl-arduino
  2. Run avrgirl-arduino test-pilot

That's it! A local server will spin up and a browser window will open with a prompt to log in via their Github account:

screenshot of login screen

Once logged in, the user will see the following steps to follow:

screenshot of 3 steps

The test process simply uploads some code to their Arduino and verifies it worked. The resulting report is automatically generated for them to submit:

screenshot of report screen

Submitting that form will send the report details to a remote database that I host. The screenshot above shows what details I'm after when the test is complete:

  1. The Arduino board they were testing with
  2. NodeJS version they're using
  3. Operating system and version
  4. Version of AVRGirl Arduino
  5. Version of Test Pilot
  6. Who the tester is (optional)
  7. Additional notes (optional)
  8. Any errors that might have been returned

This info is enormously useful for me. It gives me a place to start, and doesn't require the tester to go in and open an issue on the Github repository (where I would then begin to ask them a million questions about what software and hardware they were using etc).

In time when I finish setting up the Github organisation for AVRGirl, if the testers opted in, they'll be invited to be part of the awesome testers team within the organisation. This gives them a little 'badge' for their Github profile, which I hope will be seen as a nice little gesture to say 'thank you'.

So far, I've had two bugs come in from submitted reports, which had already made the efforts behind Test Pilot worth it. I'm especially curious to see if Test Pilot will help hardware beginners to feel less intimidated helping out with a project such as AVRGirl.

If you'd like to be a test pilot for AVRGirl Arduino, check out the 2 steps mentioned earlier in this blog post, or find help in the README of the AVRGirl Arduino repository on Github. And thank you so much in advance! <3

Suz Hinton
Hi! I'm a web developer and tech enthusiast living in Brooklyn, NY. I like to work on weird stuff. noopkat.com
comments powered by Disqus