AI, usefully · 5 min read · Sep 21, 2026

The AI returned valid JSON. The answer was still wrong.

An AI system reviews an account and returns:

{
  "account_id": "A17",
  "risk_level": "low",
  "recommended_action": "approve",
  "confidence": 0.92
}

The application accepts it.

Every required field exists. The values have the expected types. The JSON parses without an error. The workflow moves to the next step.

There is only one problem: the account should not have been approved.

Its revenue had declined for four consecutive months, two recent payments had failed and the model had interpreted missing transaction data as zero failed transactions.

The response was structurally perfect and meaningfully wrong.

As AI becomes part of analytics and operational workflows, this distinction matters. We spend considerable effort making model output consistent enough for software to consume. But a system can produce exactly the requested format while misunderstanding the evidence inside it.

A schema can verify the shape of an answer. It cannot verify its truth.

The validation stack

“Valid” has more than one meaning.

1SyntaxCan the response be parsed?

2SchemaAre required fields and allowed values present?

3Business rulesDoes the output obey the product’s deterministic constraints?

4EvidenceCan each important claim be reproduced from source data?

5ActionIs the system permitted to perform the proposed action?

Passing a lower layer does not imply that the layers above it will pass.

There are several kinds of “valid”

Syntax validation: Can the response be parsed? Is it valid JSON, or did the model add commentary, omit a comma or return malformed data?

Schema validation: Does it contain the required fields and permitted values? For example, risk_level may have to be one of low, medium or high, while confidence must be a number between zero and one.

JSON Schema supports rules for required fields, value types and whether unrecognized properties should be allowed. Its documentation also makes an easily missed point: defining a property does not make it required unless it is explicitly listed as required. See the JSON Schema reference.

Business-rule validation: Does the output obey the rules of the product?

An approval may be invalid if identity verification is incomplete. A campaign recommendation may be invalid if it exceeds the available budget. A chart may be invalid if its denominator contains ineligible users.

These checks should be deterministic code, not another polite request for the model to “double-check.”

Evidence validation: Can every important claim be traced to the supplied data?

If the model says revenue is stable, which values support that statement? If it identifies an anomaly, what baseline did it use? If it recommends contacting five customers, do those five IDs exist in the eligible population?

Action validation: Even if the answer is reasonable, is the system permitted to act on it?

A suggestion shown to an analyst is different from an instruction sent automatically to a payment, messaging or account-management system.

The higher the consequence, the stronger the boundary between recommendation and execution should be.

Confidence is not proof

A confidence value such as 0.92 looks reassuring, but what does it mean?

It might be a score generated by the model because the schema requested one. Unless the system has tested whether scores like 0.92 are actually correct about 92% of the time, the number is not calibrated evidence.

The same problem appears when we ask an AI to grade its own answer. A model may repeat the assumptions that produced the mistake rather than independently challenge them.

Instead of asking, “Are you confident?”, ask for things the system can verify:

the IDs of the records used;
the metric definition applied;
the time window selected;
the calculation or query result;
the rules that passed or failed;
and the uncertainties that remain unresolved.

A vague confidence score sounds scientific. A traceable claim is more useful.

Separate generation from checking

Request → candidate → structure → calculation → rules → evidence → action

Suppose an AI assistant is asked which customer accounts experienced the largest decline.

The model can help interpret the request, select an appropriate governed metric and propose a query. But the warehouse—not the language model—should calculate the decline.

Then a validator can confirm that every returned account exists, the current and comparison periods have equal definitions, the decline was calculated at the correct grain, missing data was not treated as real zeroes, the accounts satisfy the requested eligibility rules and the final statement agrees with the query result.

If any check fails, the workflow should stop, retry with a clear error or send the case for review. It should not quietly convert uncertainty into a complete-looking response.

NIST’s AI Risk Management Framework treats validity and reliability as properties that must be measured and managed throughout an AI system’s lifecycle—not assumed from a convincing output. Explore the NIST AI Risk Management Framework.

Give AI a contract that includes failure

Many prompts describe the ideal response but never explain what the model should do when the evidence is insufficient.

Return the requested result using only the supplied evidence.

For each conclusion, include the supporting record IDs; the metric and time window used; whether the conclusion was calculated, retrieved or inferred; any missing input that could change the result; and a validation status.

If a required input is missing, return status: needs_review and identify the missing input. Do not replace missing values with zero, guess an identifier or recommend an action unsupported by the evidence.

This improves the model’s behaviour, but it is not the validator.

The application must still enforce the schema, recalculate critical values and block invalid actions.

Test the failure path, not only the happy path

Teams often test AI workflows with clean examples where every field is present and the expected answer is obvious.

Reliability is revealed by messier cases: a required value is null; two sources disagree; the requested entity does not exist; a metric has changed definition; the evidence supports several interpretations; an identifier belongs to an ineligible record; or the correct answer is simply “not enough information.”

A useful test set should include these cases and define the expected failure behaviour.

The goal is not only to see whether the model answers correctly. It is to confirm that the surrounding system refuses to proceed when the answer cannot be trusted.

Try this

Take one structured AI workflow and write five checks beside it:

Can the response be parsed?
Does it match the required schema?
Does it obey the product’s business rules?
Can its important claims be reproduced from source data?
Is it safe and authorized to perform the proposed action?

Then create one test case that passes the first two checks but fails the third or fourth.

If the workflow still proceeds, the system is validating formatting—not decisions.

Valid JSON means the answer fits the container.

It does not mean the container holds the truth.