One deployment's evaluation process runs an LLM-as-judge audit on top of the scoring system itself. During one of those audits, it found that the hourly scoring job was inventing prices and dates roughly half the time instead of returning what was actually in the conversation.

The root cause was the model returning free text that the code then tried to parse, exactly the failure mode most bot-scoring setups have and never check for. The fix was moving that job to forced tool-use: the model has to return schema-shaped data, and a deterministic validator on top rejects anything that doesn't add up.