Nobody enjoys software testing. It is boring, repetitive and only there to annoy people: developers get to fix things they considered done; users have to wait another few weeks for a release. However, everybody sees it as important, so agile software development and modern DevOps procedures baked it into the process. The “Definition of Done” – a key term used in agile software development –  explicitly includes testing.

Well implemented, the process was clear and successful. Scenarios were written before the first line of code, executed at every release or change to the underlying system, and when the last test passed, someone signed a document and the system went live. That signature meant “verified” and “verified” translated to: “Unless something changes, what holds today will also hold next week.”

The underlying foundation of this contract rested on pillars that were so clear that nobody bothered to write them down. Reproducible behavior stems from the same input into the same system generating the same result. WHAT, IF, ELSE…. That result could be written down in advance. Risk arose when something changed, and that event triggered retesting until all gates passed, adding up to the “verified” verdict. For traditional development, testing was a phase with an end.

AI-based systems have withdrawn from the contract.

Every day is Groundhog Day

Deep-learning AI, and particularly generative AI systems, break these assumptions. Any large language model answering with any degree of creativity (with a temperature above zero) varies its output every single time it answers. Today’s output will differ from tomorrow’s; Today’s might be on point, tomorrow’s useless. When we conduct GenAI audits, we experience that on a regular basis.

That is not a defect; it is what makes the technology useful for open-ended work. But it means a single test run proves almost nothing. A compounding element with generative AI is that we are testing prose, where a wrong answer still sounds right, is delivered with confidence, but may contain statements that are not covered by the sources.

Another break happens behind the scenes. Large AI providers continuously ship updates to the engine that processes your data, and the weights behind your application continuously change. The next request is routed to a different model variant, or the context assembled around a question grows or shrinks, and the answer moves with all these changes. 

Simultaneously, inputs change. The knowledge base that feeds a retrieval system gets updated, new documents are added, others are removed. Questions users ask drift from the ones the system was trained and tuned on. In each case behavior changes without anyone releasing anything. What worked yesterday does not necessarily work tomorrow.

Acceptance testing becomes an endless loop

Good development practice has always put the definition of acceptance criteria before any single line of code was written. For Deep Learning based systems, this is even more important, because with varying output, criteria have a tendency to drift towards acceptance of the best case of observed system behavior.  

Then, there comes a paradox. While AI systems need very rigid acceptance gates to stop them from being waved through based on their best behavior, passing a gate doesn’t mean much in the long run. They need the same testing over and over, because any GenAI or ML solution that passes with flying colors on one day might fail miserably the next.

With AI, testing never ends.

What is success in AI?

In traditional testing, most test cases are focused on “right” or “wrong.”, and the acceptance gate is clear. A certain process has to work, has to correctly process an input and deliver an output. For AI, this is different. Does success mean that we expect 100% correct answers on all test cases? It is unlikely that any AI system will ever pass such a gate. So is it 95%? And what is acceptable for the remaining 5 percent? Do we tolerate one confidently wrong answer in 20 prompts? Or do we expect an honest “I don’t know”? 

Then comes the question of reliability. How many repeats must a system perform reliably before we call it stable? Is it three? Or fifty? Do these repeats have to happen at different times of the day, factoring in variability in AI data center load that can influence the answer? 

Not defining these criteria properly is why so many generative AI systems are released into production that should never see the light of day. They get released because they perform well in a good moment, and because that one-time answer was exactly on point, while the failed ones were downplayed. Or they are released because the system finally performs well against a specific test set that was heavily used during development, with specific fine-tuning that made it perform well exactly in those cases. But this is brittle by definition: a small change in the user’s prompt, a typo or an unexpected term would have thrown it off and destroyed its performance. But it was never tested against those variations. In fact, such a system was never ready.

What this looks like in practice

Testing has to be rethought from the ground up.

Acceptance criteria with test cases have to be part of the use case: When defining use cases and their business value, success and pass criteria have to be an integral part of those. Defining what an acceptable result is, what success rates are expected, and what shape of failure is tolerable needs to happen before any development starts.

Define the risks: The obvious ones come first – everything that could interact with any regulation needs to go into the risk catalog. Data protection issues, elements that fall under the EU AI Act’s high-risk categories. But that’s not where it ends. Business risks warrant equal attention, and – particularly with applications that make visible or impactful decisions – an evaluation of possible impacts. A machine learning algorithm that stops an industrial process and unnecessarily triggers costly downtime, or a customer-facing chatbot that quietly disenfranchises prospects and customers – both are real risks of AI applications worth considering.

Focus on the actual “intelligence” first: This is where deep-learning-based systems fail most. Providing access rights, preventing leaks, ensuring a nice user interface and fast responses – these are the things that are traditional development work. You will manage them if needed, and once a release is near, they to need to be completed, tested, and released. But having a nice and speedy interface with underlying quality issues from GenAI workflows is a result of doing what’s easy first. In our Chatbot Audit reviews, it’s the common failure mode. Nicely designed, fast bots delivering poor answers for real customers and thus very limited business value.

Don’t take one gold answer as confirmation for a rollout: Many stage gates of AI system introductions get passed on the lucky shot. Seeing that the solution can provide solid output occasionally often leads to the assumption that – with more fine-tuning and training, this can be turned into a stable and reliable solution. Instead, it is more helpful to ensure that a small test set produces reliably solid results across many turns. This is the litmus test for an AI system: stability. But beware: don’t optimize against this test set by hardcoding rules that literally produce the required reaction. That would be hardcoding AI.

Build stress tests: On top of your initial test cases driven from the key objectives of a system, ensure that you have test cases that are not hail-mary passes to the system. Give it ambiguous data, include typos, non-existent entities and check what happens. If the system performs well on a sunny day, that doesn’t mean it can survive reality.

Keep one test set hidden: Once a prototype is tuned against a set of questions, that set stops being evidence. This is familiar where models are trained specifically to pass a gate, for example by adjusting a prompt that fails to know that exact question and its answer, or tuning retrieval until the score improves. This is all hand-operated overfitting that never holds up in real life. The best practice: a development set is looked at constantly; that is its job of developers. A validation set is held back, run at release decisions, never used for debugging, and never the source of the examples written into a prompt. 

Track and re-test continuously. This is the bottom line. testing never stops, and every substantial change in its outcome needs a protocol. 

Automation is not optional

Monitoring AI has to become a routinely automated process. Just like we routinely monitor servers, routers, websites and other end points for availability and performance, we have to build test suites that continuously evaluate if AI solutions keep their release day performance.

Automation is not optional here; no organization runs dozens of evaluations by hand for years without tiring. Either a machine does it, or nobody will do it. And since it will most likely an AI system that monitors that AI system, human spot checks are still required to ensure the judge stays honest.

Quicksand needs probing at every step

Done was always a convention, a line drawn where the risk was judged low enough to stop looking. For systems that don’t shift internally, it worked. Now, AI systems can be like quicksand. What was stable yesterday is suddenly giving today.

That’s why the sign-off has to move – away from being an event. Release day is every day, every hour, and the only way to honestly keep up with it is by a mechanism that makes it run as reliably as the health check on key server parameters, without anyone remembering.