Earlier this year, Equinox’s president, Brad LaJeunesse, put out a call to the wider Evergreen community for literal and figurative buy-in to a project for improving the QA processes and tools used with Evergreen development:
Several members of the Evergreen community pitched in and this Equinox-managed project is now funded. The deliverables are outlined below.
- Identify and assess the efficacy of existing resources and processes.
- Deploy examples of unit tests for each language and domain within Evergreen.
- Deploy examples of API tests for Evergreen’s internal and published APIs.
- Document, with narrative, an example usability test and analysis, and the resolutions to problems discovered.
- Identify or create best examples of developer documentation for each language and domain within Evergreen.
- Demonstrate through initial implementation how continuous integration can provide regression testing.
- Analyses of available UI testing automation tools, and, if possible, design of a UI Testing framework.
The project has been running for a little over a week and here are the results so far:
- Reviewed a historical perspective on unit testing within the Perl community and a modern perspective from the TDD (Test Driven Development) community, and some other resources on testing. This was to give us some grounding and a mental framework with which to work with, and help with the jargon used within the testing community:
- Identified the Perl and C unit testing frameworks in use with Evergreen and OpenSRF, along with their respective documentation, and how they are invoked through Autoconf. We’ll need to surface this as developer documentation within the community, along with our examples on how to write such tests. It seems very likely that we will be able to use some of the same frameworks for API testing in addition to unit testing.
- Surveyed competing and complementing tools. Every problem a person solves extends what they’re capable of accomplishing, which introduces yet more problems to solve, and so on. Here we’re hoping to leverage some of that collected wisdom within the wider development community. For example, we use Test::More which can analyze return values from code, but our code can generate output along multiple vectors, and a module like Test::Warn can help us tap into one of those.
- Improved an existing unit test and implemented a new one in Evergreen leveraging a complementing tool, Test::Warn, and shared it with the community:
- Started investigating how we could make better use of AutoTools to manage the test suites. AutoTools is a framework we use for building Evergreen and OpenSRF, and one of the most immediate tie-in’s is that it calls the test suites we use, and is itself called by Buildbot when testing an Evergreen build. Autoconf is also built around tests, and we’re wanting to see if there is functionality here that we can leverage.
From here, the next steps are as follows:
- Continue gaining experience with the Perl and C testing tools, and clean up and augment the existing tests.
- Introduce C tests to Evergreen, modelled after their use in OpenSRF. This may be easier than “porting”.
- Continue investigating AutoTools, and move into learning the in’s and out’s of Buildbot. Survey competing and complementing tools for them both.
- Start learning the testing frameworks of other languages, and in particular, pgTAP for PostgreSQL.