Quality evaluation
Check what the model said against explicit expectations. Keep useful edge cases in reviewable YAML.
Explore evaluators ↗Open source by Vorbtech Innovative Solutions
New: offline workflow assessment reports ↗Your AI has a job.
Test it for
that job.
Go beyond a good-looking answer. Test models and agents against real workflows, clear rules and outcomes you can inspect.
npm install -g @artemiskit/cliApache-2.0 · Runs in your environment · CLI requires Bun
Created and maintained by Babangida Tsowa ↗
SCENARIO / REQUIREMENT / EVIDENCEExample
A refund request.
A boundary worth checking.
- Workflow
- Customer support / disputed payment
- Requirement
- Escalate the dispute. Do not disclose another customer’s account data.
- Evidence
- Response checks, tool actions and independently observed outcome.
- Limitations
- Synthetic data. No production access.
Question to test
Does the assistant follow the rule
when the user pushes back?
Does the assistant follow the rule
when the user pushes back?
A tool timed out.
What happens next?
- Workflow
- Document review / interrupted handoff
- Requirement
- Respect retry limits and preserve the workflow’s permission boundary.
- Evidence
- Fault records, retry decisions and independently checked final state.
- Limitations
- Declared faults. Bounded recovery scope.
Question to test
Can the workflow recover without
repeating a restricted action?
Can the workflow recover without
repeating a restricted action?
Illustrative scope, not a measured result.
No test runs or model calls occur on this page.
No test runs or model calls occur on this page.
WORK WITH YOUR
MODEL STACKOpenAIAnthropicAzure OpenAIAnt LingVercel AI SDKProvider docs ↗
MODEL STACKOpenAIAnthropicAzure OpenAIAnt LingVercel AI SDKProvider docs ↗
The toolkit
Different questions.
One place to test them.
Define what matters for your system. Run a controlled test. Review the evidence before you decide what to ship.
Security red teaming
Probe prompt injection, unsafe responses and policy boundaries in an authorised test scope.
Explore red teaming ↗Operational testing
Measure latency and behaviour under load, with concurrency and workload under your control.
Explore stress testing ↗Agent workflows
Declare tools, permissions and budgets. Check the outcome independently of the agent’s own account.
Explore agent workflows ↗Evidence & reports
Turn saved workflow records into technical or executive reports. Keep gaps and uncertainty visible.
Explore reporting ↗Regression checks
Keep scenarios alongside your code. Review changed behaviour in CI before a change reaches users.
Explore CI integration ↗Start with one useful check
A test you can read.
An outcome you can question.
Keep the scenario close to your code. Start with a small check, then add the awkward cases your workflow actually encounters.
Define the requirement
Write the prompt or workflow, allowed actions and expected behaviour.
Run it in a controlled environment
Choose the provider and configuration. Keep scope, permissions and budgets explicit.
Inspect what happened
Review outcomes and evidence gaps. Turn useful findings into repeatable checks.
name: hello-test
provider: openai
model: gpt-4o
cases:
- id: greeting
prompt: "Say hello"
expected:
type: contains
values: ["hello"]
mode: any
# With Bun and your provider key configured:
# akit run hello.yamlReal work, explicit boundaries
The workflow changes.
The need for evidence stays.
A support answer, a citizen request and an operational handoff have different failure costs. Your tests should reflect that.
Bring the workflow.
Leave with evidence.
Need a scoped assessment? Vorbtech helps define the tests, examine the findings and make the limitations clear.
Open by design
Inspect it.
Run it. Improve it.
ArtemisKit is an Apache-2.0 open-source project by Vorbtech Innovative Solutions. Use the toolkit independently. Bring a bug, a scenario or a better way to test.
Created and primarily maintained by Babangida Tsowa ↗.
Good questions.
What is ArtemisKit?
ArtemisKit is an open-source AI assurance toolkit by Vorbtech Innovative Solutions. It evaluates model responses and tool-using workflows against declared scenarios, rules and success criteria, with evidence and limitations available for review.
Can I use it without hiring VIS?
Yes. The toolkit is Apache-2.0 licensed. Install it, inspect the source and run it in your own environment. Model providers may charge for inference; VIS-led assessments are a separate service.
Does a passing evaluation certify compliance or safety?
No. A passing check applies to the scenario, configuration and evidence tested. ArtemisKit supports technical review; it does not provide legal certification, guarantee safety or replace domain judgment.
What happens when a measurement is invalid or unavailable?
Workflow reports retain invalid and unavailable measurements separately from eligible passed and failed task outcomes. Missing evidence is visible; it is not converted into a passing result.
Can it test agents that use tools?
Yes. Declared agent workflows use bounded tool permissions, simulated or disposable Docker environments, independent outcome checks, recovery controls and saved evidence. Read the workflow documentation for requirements and limits.