…is surprisingly hard to do. Yesterday I got fed up enough at trying to figure out whether our stats-gathering code was working to actually write a script to generate an artificial sample of perfect data. It took me about 20 minutes. When I’d finished I showed it to a colleague and she said “Why did no-one think of this until now?” I had no answer. If anyone should have thought of it, it should have been me. It seemed so obvious a thing to do, once I actually got irritated enough to sit down and do it. Up until then we’d been doing our testing using real data. More life-like? Yes, but impossible to verify automatically. If the system claims 15,768 occurrences of ‘x’ in a total sample size of 28 million, how do you know it is right? Check by hand?

Automated repeatable tests. Such a small amount of effort for such a huge gain.