Nobody enjoys software testing. It is boring, repetitive and only there to annoy people: developers get to fix things they considered done; users have to wait another few weeks for a release. However, everybody sees it as important, so agile software development and modern DevOps procedures baked it into the process. The “Definition of Done” in agile software development – the agreed quality criteria an Increment must meet before it is considered complete – makes those expectations explicit.

Well implemented, the process was clear and successful. Scenarios were written before the first line of code, executed at every release or change to the underlying system, and when the last test passed, someone signed a document and the system went live. That signature meant “verified” and “verified” translated to: “Unless something changes, what holds today will also hold next week.”

The underlying foundation of this contract rested on pillars that were so clear that nobody bothered to write them down. Reproducible behavior stems from the same input into the same system generating the same result. IF, THEN, ELSE… In functional tests, the expected result could therefore be written down in advance. Risk arose when something changed, and that event triggered retesting until all gates passed, adding up to the “verified” verdict. For traditional development, release-oriented testing had a defined endpoint.

AI-based systems have withdrawn from the contract

Testing a moving target

Deep-learning AI, and particularly generative AI systems, break these assumptions. Any large language model answering with any degree of allowed creativity varies its output every single time it answers. Today’s output will differ from tomorrow’s; today’s might be on point, tomorrow’s materially worse. When we conduct GenAI audits, we experience that on a regular basis.

That is not a defect; it is what makes the technology useful for open-ended work. But it means a single test run proves almost nothing. A compounding element with generative AI is that we are testing prose, where a wrong answer still sounds right, is delivered with confidence, but may contain statements that are not covered by the sources.

Another break may happen behind the curtain. Managed AI services may introduce model updates, deprecations or routing changes outside your application’s controlled workflow. Where routing is dynamic, requests may also be processed by different model variants.

Simultaneously, inputs change. The knowledge base that feeds a retrieval system gets updated, new documents are added, others are removed. Questions users ask drift from the ones the system was trained and tuned on. In each case, behavior can change without any application-code release. What worked yesterday does not necessarily work tomorrow.

Acceptance testing is now an endless loop

Good development practice has always put the definition of acceptance criteria before any single line of code was written. For deep-learning-based systems, this is even more important, because with varying output, criteria have a tendency to drift towards accepting the best case of observed system behavior.  

Then, there comes a paradox. While AI systems need very rigid acceptance gates to stop them from being waved through based on their best behavior, passing a gate doesn’t mean much in the long run. They need the same testing over and over, because any GenAI or ML solution that passes with flying colors on one day might fail miserably the next.

With AI, the release decision remains, but it becomes conditional: it applies to a defined intended use and a known behavior-relevant system configuration. Assurance then continues in production, and material changes must trigger re-evaluation. In other words: with AI, testing never ends. 

What is success with AI?

In traditional testing, most test cases are focused on “right” or “wrong”, and the acceptance gate is clear. A certain process has to work, has to correctly process an input and deliver an output. For AI, this is different. Does success mean that we expect 100% correct answers on all test cases? It is unlikely that any AI system will ever pass such a gate. So is it 95%? And what is acceptable for the remaining 5%? Do we tolerate one confidently wrong answer in 20 prompts? Or do we expect an honest “I don’t know”? Some failure types, however – such as access-control bypass, confidential-data leakage or unauthorized actions – should not be averaged into an acceptable error rate at all. They are hard gates.

Then comes the question of reliability. How many repeated runs must a system perform reliably before we call it stable? Is it three? Or fifty? There is no universal number: sample size should follow the risk, expected failure prevalence and confidence required. Repeated runs should use the production configuration; load and latency belong to a separate performance test.

Not defining these criteria properly is why so many generative AI systems are released into production that should never see the light of day. They get released because they perform well in a good moment, and because that one-time answer was exactly on point, while the failed ones were downplayed. Or they are released because the system finally performs well against a specific test set that was heavily used during development, with specific fine-tuning that made it perform well exactly in those cases. But this overfitting is brittle by definition: a small change in the user’s prompt, a typo or an unexpected term would have thrown it off and destroyed its performance. But it was never tested against those variations. In short: such a system was never ready.

What this looks like in practice

Testing itself does not have to be reinvented from the ground up: the established discipline remains or has to even be tightened. What has to change are the testing workflow and the evidence model. Here are the key cornerstones of LLM testing:

Acceptance criteria with test cases have to be part of the use case: When defining use cases and their business value, success and pass criteria have to be an integral part of those. Defining what an acceptable result is, what success rates are expected, and what shape of failure is tolerable needs to happen before any development starts.

Define the risks: The obvious ones come first – everything that could interact with any regulation needs to go into the risk catalog. Data protection issues, elements that fall under the EU AI Act’s high-risk categories. But that’s not where it ends. Business risks warrant equal attention, and – particularly with applications that make visible or impactful decisions – an evaluation of possible negative outcomes. A machine learning algorithm that stops an industrial process and unnecessarily triggers costly downtime, or a customer-facing chatbot that quietly disenfranchises prospects and customers – both are real risks of AI applications worth considering.

Focus on AI-specific quality early: This is where deep-learning-based systems fail most. Access rights, leak prevention, user-interface quality and performance remain essential and must be tested. They are, however, comparatively well-understood engineering disciplines. The new blind spot is the quality of the AI behavior itself. In our GenAI Audits, inconsistent or irrelevant answers are the common failure mode. Nicely designed, fast bots delivering poor answers for real customers that create very limited business value and sometimes even damage a company’s reputation.

Don’t take one gold answer as confirmation for a rollout: Many stage gates of AI system introductions get passed on the lucky shot. Seeing that the solution can provide solid output occasionally often leads to the assumption that – with more fine-tuning and training – this can be turned into a stable and reliable solution. Unfortunately, this hope is rarely warranted. Instead, it is more helpful to ensure that a small test set produces reliably solid results across many turns. This is an important litmus test for an AI system: stably correct output. But stability alone is not enough: a system can be consistently wrong. But beware: don’t optimize against this test set by hard-coding rules that literally produce the required reaction. That would be hard-coding around the evaluation rather than improving generalization. 

Build robustness tests: On top of your initial test cases derived from the key objectives of a system, ensure that you have test cases that are not easy passes for the system. Give it ambiguous data, include typos, non-existent entities and check what happens. If the system performs well on a sunny day, that doesn’t mean it can survive reality.

Keep one test set hidden: Once a prototype is tuned against a set of questions, that set stops being independent general evidence. It may still be valuable as a regression set, but it should no longer be the only basis for a release decision. Too often, models are trained specifically to pass a gate, for example by repeatedly adjusting a prompt until it produces the desired answer to that specific test question, or by tuning retrieval until the score improves. This is all hand-operated overfitting that never holds up in real life. The best practice: while developers constantly work with a development set, a validation set is held back, run at release decisions, never used for debugging, and never the source of the examples written into a prompt. 

Track and re-test continuously: This is the bottom line. Testing never stops, and every substantial change in its outcome needs a protocol. 

Automation is not optional

The idea of “humans in the loop” as the primary mechanism for continuous monitoring is simply unrealistic. It doesn’t scale with the speed of LLMs.

No organization runs large numbers of evaluations by hand for years without tiring. Either a machine does it, or nobody will do it. And since it will most likely be an AI system that monitors that AI system, human spot checks are still required to ensure the judge stays honest. Deterministic checks should be automated wherever possible.

Ultimately, monitoring AI has to become a routinely automated process. Just like we routinely monitor servers, routers, websites and other endpoints for availability and performance, we have to build test suites that continuously evaluate if AI solutions keep their release day performance.

Quicksand needs probing at every step

Done was always a convention, a line drawn where the risk was judged low enough to stop looking. For systems that don’t shift internally, it worked. Now, AI systems can be like quicksand. What was stable yesterday is suddenly failing today.

That’s why the sign-off has to move – away from a one-off event to a conditional release decision backed by continuous evidence. Release day is no longer the only quality checkpoint. Automated re-evaluation has to keep running as reliably as a health check on key server parameters, without depending on someone remembering to trigger it.

The original ideas, structure, and much of the language are human-created, but AI was used to develop, enrich, or rework portions of the content – for example, researching sources, rewriting sections for clarity, or expanding on arguments.