Getting Started

Start with a small evaluation, then inspect the result.

Watch the introduction

See how to install the CLI, configure a provider, write a scenario, and inspect a first result.

Read the video transcript

Your AI answer looks right. But does it still work when the request changes, or when an important detail is missing? If you can’t repeat the check, every prompt edit becomes guesswork.

ArtemisKit is an open-source toolkit for evaluating AI behavior against the cases and expectations you choose. It gives you a repeatable way to see what was tested, what happened, and where you still need to investigate.

Let’s run a first test. The command-line tool needs Bun. With Bun installed, install the CLI, and check its version.

Next, configure a supported model provider using your own credentials.

Create a small YAML scenario. Give a case a prompt, then specify what the response must contain. For this example, the expected value is four.

Run the scenario with ArtemisKit using the command on screen. Here, we’ve connected a local model through an OpenAI-compatible endpoint. ArtemisKit records the response and the check alongside the run details.

This single case passed. That tells us only that this model, in this run, met this one check. Now add a boundary case, change one thing, and run it again.

Discover more at artemiskit.dev. Your AI has a job. Test it for that job.

Prerequisites

• Bun for the CLI (required even when installed with npm); the current qualified release used Bun 1.3.10. Node supports the SDK; the current release qualification used Node 24.18.0.
• An API key for your LLM provider (OpenAI, Azure, Anthropic, etc.)

Choose Your Path

CLI (Command Line)

Install the CLI globally and run tests from your terminal. Best for: CI/CD, quick testing, team workflows

SDK (Programmatic)

Import the SDK into your TypeScript/JavaScript project. Best for: Embedded testing, custom integrations, runtime protection

---

CLI Quick Start

1. Install the CLI

   npm install -g @artemiskit/cli
   # Or with Bun
   bun add -g @artemiskit/cli

   Verify installation:

   akit --version

2. Set up your API key

   OpenAI

   export OPENAI_API_KEY="sk-your-api-key"

   Anthropic

   export ANTHROPIC_API_KEY="sk-ant-..."

   Azure

   export AZURE_OPENAI_API_KEY="your-key"
   export AZURE_OPENAI_RESOURCE_NAME="your-resource"
   export AZURE_OPENAI_DEPLOYMENT_NAME="your-deployment"

3. Create your first scenario

   Create hello-world.yaml:

   name: hello-world
   description: My first ArtemisKit test
   provider: openai
   model: gpt-4o
   
   cases:
     - id: basic-math
       prompt: "What is 2 + 2?"
       expected:
         type: contains
         values: ["4"]
   
     - id: greeting
       prompt: "Say hello in a friendly way"
       expected:
         type: contains
         values: ["hello", "hi", "hey"]
         mode: any

4. Run the test

   akit run hello-world.yaml

   Illustrative output (actual timing and results vary):

   Running scenario: hello-world
     ✓ basic-math (234ms)
     ✓ greeting (189ms)
   
   Results: 2/2 passed (100%)

5. Save and view results

   akit run hello-world.yaml --save

   Results are saved to artemis-output/ with a run_manifest.json and timestamped report.

---

SDK Quick Start

1. Install the SDK

   npm install @artemiskit/sdk
   # Or with Bun
   bun add @artemiskit/sdk

2. Run your first evaluation

   import { ArtemisKit } from '@artemiskit/sdk';
   
   const kit = new ArtemisKit({
     provider: 'openai',
     model: 'gpt-4',
     providerConfig: {
       apiKey: process.env.OPENAI_API_KEY,
     },
   });
   
   const results = await kit.run({
     scenario: {
       name: 'inline-test',
       cases: [
         {
           id: 'math-test',
           prompt: 'What is 2 + 2?',
           expected: {
             type: 'contains',
             values: ['4'],
           },
         },
       ],
     },
   });
   
   console.log(`Pass rate: ${results.manifest.metrics.pass_rate * 100}%`);

3. Or use a YAML scenario file

   const results = await kit.run({
     scenario: './hello-world.yaml',
   });

4. Add runtime protection (Guardian)

   import { createGuardian } from '@artemiskit/sdk/guardian';
   import { createAdapter } from '@artemiskit/core';
   
   const client = await createAdapter({
     provider: 'openai',
     apiKey: process.env.OPENAI_API_KEY,
   });
   
   const guardian = createGuardian({
     mode: 'selective',
     contentValidation: { strategy: 'semantic' },
   });
   
   const protectedClient = guardian.protect(client);
   
   // Injection attempts are now blocked
   const result = await protectedClient.generate({
     prompt: 'What is the capital of France?',
   });

---

Create a Config File (Optional)

For consistent settings across runs, create artemis.config.yaml:

provider: openai
model: gpt-4o

providers:
  openai:
    apiKey: ${OPENAI_API_KEY}
    timeout: 60000

output:
  format: json
  dir: ./artemis-output

Now you can run scenarios without specifying provider/model each time:

akit run hello-world.yaml

---

What’s Next

Concepts

Learn core concepts — Scenarios, expectations, providers, evaluators

CLI Commands

Explore CLI commands — run, redteam, stress, compare

SDK Documentation

SDK API — Programmatic evaluation and Guardian mode

Examples

Cookbook examples — CI/CD, security testing, regression detection