What is the problem?
Misleading aggregate scores, unobserved failures and incomparable configurations.
Example: A controlled agent task
A candidate model performs a declared workflow with the same bounded tools, fixtures and outcome criteria.
What could be tested?
- Scenario-specific task completion
- Tool use, policy adherence and recovery
- Evidence validity and declared operational measures
What evidence could you receive?
Versioned configuration records, eligible outcomes, limitations and sanitised evidence.
What are the limits?
No universal ranking. Comparison needs compatible methods and sufficient independent evidence.
What should you prepare?
Describe the workflow and target, the rules and risks, authorised test data or access, success criteria, stakeholders and timing or budget constraints. Use high-level public descriptions until a private scoping channel is agreed.
Good questions.
Is this a published client result?
No. This is a hypothetical workflow to explain a possible assessment scope. No customer, measured result or sector endorsement is claimed.
What is needed before testing?
Agree the workflow, target system, rules, risks, authorised test data/access, success criteria, stakeholders and constraints. Submission does not authorise testing.