Quality Assurance

Earlier this year, Equinox’s president, Brad LaJeunesse, put out a call to the wider Evergreen community for literal and figurative buy-in to a project for improving the QA processes and tools used with Evergreen development:

Several members of the Evergreen community pitched in and this Equinox-managed project is now funded.  The deliverables are outlined below.

  • Identify and assess the efficacy of existing resources and processes.
  • Deploy examples of unit tests for each language and domain within Evergreen.
  • Deploy examples of API tests for Evergreen’s internal and published APIs.
  • Document, with narrative, an example usability test and analysis, and the resolutions to problems discovered.
  • Identify or create best examples of developer documentation for each language and domain within Evergreen.
  • Demonstrate through initial implementation how continuous integration can provide regression testing.
  • Analyses of available UI testing automation tools, and, if possible, design of a UI Testing framework.

The project has been running for a little over a week and here are the results so far:

From here, the next steps are as follows:

  • Continue gaining experience with the Perl and C testing tools, and clean up and augment the existing tests.
  • Introduce C tests to Evergreen, modelled after their use in OpenSRF. This may be easier than “porting”.
  • Continue investigating AutoTools, and move into learning the in’s and out’s of Buildbot. Survey competing and complementing tools for them both.
  • Start learning the testing frameworks of other languages, and in particular, pgTAP for PostgreSQL.