Ordinary software gives the same answer twice. AI does not, so the tests your team already has cannot tell you whether it is right. We build the scored test set, run it on every change, try to break the system on purpose, and give you a number you can show a board or an auditor.
This is for you if
Why it is different
This is not ordinary testing with a new label. Each of these breaks an assumption that normal automated testing depends on, which is why AI systems tend to reach production untested.
A test that checks the output matches an expected string fails on a correct answer worded differently. Scoring has to judge meaning, not characters, which means the test itself needs building rather than writing.
What we do insteadScore against meaning, and check the facts and format separately from the wording.For a summary or a draft reply, five different answers can all be acceptable and a sixth clearly not. Correct has to be defined as a range, agreed with the people who actually do the work.
What we do insteadAgree what good looks like with your experts, then score consistently against that.Overall accuracy barely moves while one supplier, one document type or one language collapses. Averages hide exactly the failure that reaches a customer.
What we do insteadScore by slice, so a drop in one group is visible even when the total is steady.A hosted model is updated by its provider, your documents change, your customers start asking differently. Ordinary software does not decay on its own. This does.
What we do insteadRun the same set on a schedule, not only at release, and alert on the drop.Instructions hidden in a document, questions phrased to get around a rule, requests designed to make it reveal its configuration. No conventional test suite looks for any of this.
What we do insteadA deliberate attempt to break it, repeated as a suite rather than done once.Services
Sitting with the person who does the job today and writing down what a good answer is, what an unacceptable one is, and which mistakes actually matter. Every number later depends on this and it cannot be done from a specification.
Drawn from your own history, deliberately including the awkward ones, the rare ones and the ones that caused an argument. A test set of easy cases produces a high score and tells you nothing.
Per document type, per supplier, per language, per customer. This is what catches the failure that an overall figure hides, and it is the difference between a report and a dashboard nobody acts on.
For anything that answers from your own material: does every statement come from the source it cites, does it refuse when the answer is not there, and does it retrieve the right passage in the first place. Retrieval and answering are scored separately because they fail separately.
Instructions hidden inside documents it reads, attempts to get around its rules, requests for information it should not give, and prompts designed to run up cost. Structured, repeatable, and rerun rather than done once at launch.
Where a system affects people, whether outcomes shift with a name, a gender or a location. Uncomfortable to run and much more uncomfortable to be shown by somebody else.
The model is slow, the provider is down, a document is corrupt, the answer comes back malformed. Ordinary QA work, routinely skipped on AI features because everyone is looking at the clever part.
Time to first word, time to finish, cost per request, and what all three look like at your busiest hour rather than with one user.
The suite runs on every change, in your own system, and blocks a release that drops below the threshold you set. A test set run by hand once a quarter stops being run by month four.
The same scoring against a sample of live traffic, so you find out that accuracy has slipped from your own dashboard rather than from a customer.
What was tested, how, on what data, what the results were, what is still weak and what is being done about it. Suitable for a board paper, a customer's security questionnaire or an EU AI Act file.
An independent read on a supplier's AI, or on something your own team built before anyone was scoring it. You get the same evidence, and we have no stake in the answer being flattering.
If something of yours is live and unmeasured, we can usually tell you how it is doing within two weeks.
Our stack
Some of this comes from twelve years of ordinary software testing and some is specific to AI. Both halves are needed, and most teams have only one of them.
Example
A company asked us to check an assistant a supplier had built for them. The supplier reported 94% accuracy and the number was not dishonest. It was just measured as one figure across everything.
We rebuilt the test set from their own history and scored it by slice. Overall it came out at 93%, close enough to confirm the supplier's figure. Underneath, two things were wrong. Questions about one product line, about 6% of the volume, scored 61%, because that line's documentation had been written by a different team in a different format. And every question asked in the second language they support scored below 70%, which nobody had tested at all.
Neither was visible in the headline number, and both were reaching customers daily. The fix was not a better model. It was reformatting one set of documents and adding the second language to the test set so the gap could not reappear unnoticed.
The suite now runs on every change in their own pipeline. The score they quote to their board is by slice, because a single average is what let this sit for five months.
An example, with invented figures, to show what slice-level scoring finds. The pattern is real and common.
FAQ
Yes, and a good part of this work is exactly that, including systems built by other suppliers. We have no interest in the result being flattering, and if the honest finding is that it works well, that is what the report says.
Two to four weeks for a first one. Most of that is building the test set with your experts, which is the part that has to be done properly. After that, running it again is automatic.
It depends entirely on what happens when it is wrong and whether a person checks. A suggestion a user edits can live at 85%. Something posting to your ledger unattended cannot. We set the threshold with you, per task, before measuring.
That is the intended end state. The suite lives in your repository, runs on every change and blocks a release below the threshold. If it only runs when we run it, it stops running.
Yes. For systems in the higher risk tiers the Act expects evidence of testing, human oversight and record keeping. We produce the testing evidence and can tell you where the rest of the obligation sits.
It builds on it, and that is the point. Half of this is ordinary test engineering, which most AI specialists have never done, and half is specific to models, which most QA teams have not met yet. Very few places have both, and it is why we sell this at all.
Next step
What it does, who relies on it, and what the current accuracy claim is based on. If the answer to the last one is a demonstration, that is the most useful thing you can tell us.
Ahmedabad, India. We work with teams in the US, UK, Europe, Singapore and the Gulf, and we are used to the time difference.