Test the Testing
One of many benefits of test-driven development (TDD) and automated unit and regression testing is that it provides additional defect-detection safety nets. Development teams catch more of their own defects before delivery to test. In my own recent experience (and that of others I've talked with), this is very valuable and very timely.
In so many shops, recent cost cutting and offshoring has diminished the quality of component and system testing. Often this decline is staff related: reduced tester availability and increased turnover, even within offshore test groups. Defect escapes are costly and embarrassing. So what's a developer to do?
We can borrow one groovy solution from the Three Dog Night days. Those dusty pages of the 70s Harlan Mills papers describe defect seeding: deliberately adding defects to monitor defect discovery rates and coverage. Also known as bebugging (with a "b"), defect seeding objectively measures the quality of the testing. In short, it tests the testing.
Deliberately adding bugs seems to go against the grain: it's a destructive activity, as is test itself. Yet, properly measured and managed, it can be an important part of overall software quality engineering. If a test team misses known, well-placed bugs, this is identified and corrective action can be taken. That's much better than shooting in the dark. It can shed much-needed light when the test team is in a different location and direct oversight isn't possible.
Much of that 70s literature on bebugging was wrapped up with somewhat extreme, academic concepts - like limited (statistical) testing and the idea that tools could automatically inject and remove defects or faults. Much simpler approaches are possible; such as:
- Developers deliberately code important but non-fatal defects in new code they write, while at the same time writing xUnit test cases that will fail because of the injected defects. The tools themselves provide the metrics, and this avoids the risk of forgetting about some seeded bugs when it's time to remove them after the test iteration.
- After releasing a sprint or build to test, the development team continues to actively test the product themselves. Any defects found by the developers are privately recorded and tracked to determine if and when the test team finds them.
Simple bebugging practices such as these and others can, over time, improve test quality and coverage. And you'll have real metrics to see how well Test passes The Test.