Test a skill

A skill test shows whether a skill makes the Mind's work better, and whether the Mind loads it for the right requests. Start a test from the skill's Test skill button in the Mind's Skills tab, or ask the Mind for one in a chat. Every test shows an estimate of its size first and starts only after you confirm it. To learn how skills work, see Skills a Mind can load.

Kinds of skill test

Before you start

• Skill tests run in Analytical Minds, Warehouse Builder Minds and Lake Builder Minds.
• A Mind tests the skills it owns, listed under This mind in its Skills tab. People with Modify AI Skills can also benchmark an organisation skill in a Mind the skill applies to, from the Skills tab. To test a shared or built-in skill, convert it to a local skill first. See Share a skill with other Minds.
• A Mind runs one skill test at a time. Multiple-choice experiments on the Benchmarks tab neither count nor wait.
• You cannot start, continue or change a skill test, or save a reviewer note, while you view DataMind OS as another user.

You need the right to manage the Mind, plus:

Without them, the buttons are hidden, or the Mind tells you which permission is missing and starts nothing. See Who can do what with Minds.

Test a skill in the Skills tab

1. Open the Mind's Skills tab and select the skill. The tab works on a desktop screen.
2. Click Test skill at the bottom of the skill. The tests open in place of the editor, under Tests of and the skill's title.
3. Pick a section: Results lists the tests of the skill, Test tasks holds its practice tasks, and Trigger test runs trigger tests and description tuning. Trigger test is shown for skills of This mind only.

Click the back arrow, Back to the skill, to return to the editor. Unsaved edits in the editor stay, so you can test them before you save. On a skill shared with this Mind, Test skill is greyed out; point at it to read why. Built-in skills have no Test skill button.

Write test tasks

A benchmark runs the skill's saved test tasks. A skill can have up to 10. The tasks of an organisation skill are shared by every Mind that tests it.

1. In Test tasks, click Add task.
2. Fill Title and Prompt: a request a user might make. Runs see the prompt as the user's first message. Use made-up names and numbers, never real customer data.
3. For each check, pick its kind under Check, then describe what a good run does under What passes, or under Criterion for a Judged check. Fill the fields the kind asks for. Click Add check for another one.
4. Click Save task.

A task card shows No result check or No process check when the task has no check on the result, or none on how the Mind worked. Point at the badge to read more. Tests that already ran keep their own copy of the tasks, so editing or removing a task does not change them.

Run a benchmark

1. At the bottom of Test tasks, pick what to compare with under Compare with: Without the skill, the previous revision (for example Revision 3), or Both.
2. To test the editor's unsaved text instead of the saved skill, turn on Test my unsaved changes. It appears when you have unsaved edits of this skill. Nothing is saved.
3. Click Estimate and run.
4. Check the estimate in Start a benchmark?: what runs, how many test runs, about how many tokens and minutes, and the model. Click Start (n test runs) to begin, or Cancel.

The test appears under Results and runs in the background. If a test task changed after you opened the estimate, the window shows the new estimate and you start again.

The default comparison is the previous revision when there is one, and running without the skill otherwise. An organisation skill starts on Without the skill.

Follow a test

Select a test under Results to see its details. While it runs, the details show how many runs are done.

• Stop stops a running test. Runs in progress finish; no new runs start.
• Continue starts the remaining runs of a stopped test. Only the person who started the test can continue it.
• Delete deletes a test that is not running, with its runs and their notes.

Use All, Benchmarks and Trigger tests above the list to filter it. The list shows the newest 200 tests. A test that ran in another Mind, for example a test of an organisation skill, shows In and that Mind's name, and you can only read it here.

Read a benchmark

Results shows the verdict and how each version did:

• Version, Pass rate, Tokens per run, Time per run, Runs and Loaded the skill for each version, and Delta vs baseline for the tested version.
• Badges for checks that do not tell the versions apart, checks that fail in both, and flaky tasks.
• One row per task with each version's pass rate. Open a row to see its checks and the judges' notes under Grader notes.

Point at the info icon next to the verdict to read how a benchmark passes.

Review shows the runs of one task side by side:

1. Pick a task on the left, or click Review on its row in Results.
2. Under each version, pick a run tab, such as Run 1. You see the score, each check with its evidence, the judges' votes under Show the 3 votes, the final answer, the tool calls, the queries and the skills the run loaded. A tool call marked Planned, not run was skipped because it would have changed something.
3. Write what is wrong in Reviewer note and click Save note. The Mind reads these notes before it revises the skill. Clear note removes it.

Open the run chat opens the hidden chat of a run. If you cannot open the chats of the person who started the test, the review says that the run details are hidden and shows only the scores.

Test triggers and tune the description

1. In Trigger test, pick Trigger test or Description tuning under Test.
2. Write up to 20 questions. For each one, choose under Should load the skill whether it should make the Mind load this skill. Mix requests that need the skill with close ones that do not, written the way users ask.
3. For description tuning, set Descriptions to try (1 to 5). Turn on Choose the split myself to put each question in Training or Held out under Set; otherwise the test splits them.
4. To test the description in the editor instead of the saved one, turn on Use the description in the editor. It appears when you changed the description in the editor.
5. Click Estimate and run, check the estimate and click Start (n probes).

The result lists each description tried with its accuracy, precision and recall, and each question with how often the skill loaded and which skill loaded instead. After tuning, Best description shows the winner. Click Use in the editor to put it into the skill's description, then save the skill if you agree. Run again with these questions fills the form with the same questions.

Ask the Mind for a test in a chat

1. In a chat with the Mind, ask for a test, for example: "Benchmark the monthly-close skill against its previous revision." When you write or improve a skill with the Mind, it may suggest a test itself.
2. The Mind prepares up to 10 practice tasks, each with the checks a good run must pass, or the test questions of a trigger test.
3. The Mind stops at a card titled Skill test needs your approval. It shows the skill, the versions it compares, the number of runs and the estimate. Show the tasks (n) lists the tasks with their checks and reference queries, Show the changes compares an unsaved version with the saved revision, and Show the questions (n) lists the test questions. While the card waits, the message box reads A skill test is awaiting your decision.
4. Approve it to start the test, or reject it. The chat then shows a card such as Benchmark started. On a desktop screen, and with View Mind Skills, Open the results opens the test in the Mind's Skills tab.
5. Ask the Mind for the result when you come back. It does not wait in the chat. The result card is titled, for example, Benchmark of "monthly-close".

Only the person who asked for the test can approve it, because the test runs with that person's access. Someone else in the chat can reject it, but not approve it. The approval is needed in every acceptance mode, also in Unrestricted.

What a benchmark compares

Each practice task runs 3 times for each version, in new chats on the Mind's real tools and its default model.

The tested version is the saved skill, or a version that is not saved yet. A new skill that is not saved yet can only be compared with running without it, and revision 1 has no previous revision.

A judged check passes when at least two of its three judges pass it and quote the evidence from the run. A task can have up to 3 judged checks. Patterns are regular expressions without backreferences or lookarounds. Reference queries only read data.

Test chats never change data

• Every tool that would change something, for example saving, creating, scheduling or deleting, is skipped. The run continues as if the step had worked, and the step is graded as planned.
• Tools that only read, such as queries, run as in a normal chat. In a Lake Builder Mind, this includes the tools that read your source systems.
• Web search and swarm agents are off in test chats.
• Test chats run with the access of the person who started or approved the test, and they do not appear in the chat list.

When a benchmark passes

For each version, the result shows the share of checks passed (an average with its spread across runs), the tokens and the seconds per run, and how many runs loaded the skill. A benchmark passes when all of these hold:

• The tested version passes at least 80% of its checks on average.
• It passes no fewer checks than the version it is compared with.
• Every task has at least 2 graded runs for each of the two versions.

A run whose checks could not be graded, for example because the judges did not answer, does not count as a graded run.

How trigger tests and tuning work

• Each question is sent 3 times as the first message of a new chat with the Mind's real instructions, tools and skill list. Only the Mind's first response is read, and no tool runs.
• A question that should use the skill passes when the skill was loaded in at least half of its runs, and one that should not passes when it was loaded in fewer than half. Precision says how many loads were right, recall how many of the questions that should use the skill did.
• Description tuning needs at least 2 questions that should use the skill and 2 that should not. About 60% of each kind train the writer and 40% are held out. Each round tests one description on every question, then the writer rewrites it from the mistakes on the training questions only.
• Up to 5 descriptions are tried. The run stops early when a description gets every training question right. The best description has the highest accuracy on the held-out questions; on a tie, the shorter one wins. It is at most 500 characters, because the Mind's skill list shows only the first 500.
• A new skill needs a title and a description for a trigger test.

Size and cost

• The estimate shows the number of test runs, about how many tokens they use and how many minutes they take. It is based on earlier tests on this Mind when there are enough of them, and on a typical size otherwise. It shows no price.
• Practice tasks and test questions run on the Mind's default model. Judges and the description writer always use Kartlos Premium, whatever model the Mind uses.

Promote a tested skill

To promote a Mind skill to the organisation, the latest benchmark of the revision you promote must have tested that saved revision and passed. See Promote a Mind skill to the organisation.

Limits

Questions, answered

Does a skill test change my data or my skill?

No. Test chats skip every tool that would change something, and description tuning returns a description without saving it. Use in the editor only fills the editor; the skill changes when you save it.

Why is Approve greyed out on a skill test card?

Only the person who asked for the test can approve it. Point at Approve to read "Only the person who asked for this skill test can approve it. You can reject it." People who can decide on changes in the chat can reject it.

Can I test an organisation skill?

Yes, if you have Modify AI Skills and manage a Mind the skill applies to. In that Mind's Skills tab, select the skill under Organization and click Test skill, or use Open tests on the skill in Organisation skills. Organisation skills get benchmarks only, no trigger tests.

Why is Test skill greyed out on a shared skill?

Only the Mind that owns a skill can test it. Convert the skill to a local skill in the Skills tab, then test the copy.

Can I test a change before I save it?

Yes. Turn on Test my unsaved changes before you click Estimate and run, or ask the Mind in a chat. The unsaved version runs against the saved revision, and nothing is saved. From a chat, testing an unsaved version also needs Author Mind Skills with AI.

Why can't I start a second skill test?

A Mind runs one skill test at a time. Wait until the running test finishes, or stop it.

Why did my benchmark not pass?

The result names the reason: the tested version passed fewer than 80% of its checks, passed fewer than the version it was compared with, or a task has fewer than 2 graded runs. Use Review to read the failing runs, improve the skill or its tasks, then run the benchmark again.

Where do I see the result of a skill test?

In the skill's Test skill view in the Mind's Skills tab, under Results. In a chat, you can also ask the Mind. Skill tests are not in the Experiments list of the Benchmarks tab, and their test chats are not in your chat list.

Why does a chat say Skill test at the top?

The chat is one run of a skill test, opened with Open the run chat. It runs a test version of the skill and never changes data. On a desktop screen, Open the results at the top opens the whole test. The message box stays locked until the run is graded.

Which models does a skill test use?

Practice tasks and test questions run on the Mind's default model. Judges and the description writer use Kartlos Premium.

Why does the test say another skill hides mine?

A built-in or organisation skill with the same name reaches this Mind and wins over yours, so a test would never reach your skill. Rename your skill, then test it.

Can a skill test read my source systems?

In a Lake Builder Mind, yes. The tools that read your source systems run in test chats as in a normal chat, while every tool that would change something is skipped.