The eCos Test Farm
Software Reliability
eCosCentric operates a permanent 24x7 automated testing facility for eCos-based hardware. The facility runs all the public and eCosCentric custom testcases, permuted over a large number of different eCos configurations, on a diverse range of embedded target hardware. The tests and configurations are designed to push eCos through as wide range of states and coverage of code paths as possible, rigorously testing the stability of the hardware, eCos RTOS, RedBoot firmware and associated toolchains.
Before eCosPro Developer's Kits for general development boards or customer-specific boards are shipped, they must successfully pass through the testfarm. All development boards, and where possible customer-specific hardware, then become permanently resident in the test farm. This provides assurance that as the eCosPro codebase evolves, it continues to remain compatible with supported development boards and customer hardware alike, streamlining any future updates. It also provides direct access for our engineers to reproduce and debug any issues reported by customers, and the capability to rapidly check that any fixes do not cause regressions elsewhere.
The test farm consists of number of build hosts and a large array of target platforms, along with a plethora of associated relay and power controllers, port concentrators, field buses, network switches, USB hubs and devices, and other miscellaneous test hardware. Depending on it's functionality, each target platform has a defined set of configuration permutations and testcases, which a build host takes and turns into a stream of test binaries. These tests are subsequently downloaded and executed on the target hardware. Each test result, along with its output, execution time, memory usage and other information are logged in the testfarm database. Any failing tests are stored for later reproduction and analysis. Database triggers alert developers of new or intermittent failures. The testfarm database contents can be browsed, queried and filtered to look at specific results, historical subsets and trends within the recorded results. A web based UI provides status, control and management of all aspects of the testfarm as well the database query interface.
The images below show an earlier incarnation of the current eCos test farm.
When operating at peak throughput the farm executes well over a million tests per week. Per target, the typical farm test run consists of between 22,000 and 44,000 individual test executions - depending on the specific architecture and feature set supported. Test runs typically take between one and two weeks depending on various factors. The fastest targets complete their runs in a handful of days, whilst the slowest can take several weeks.