A benchmark checks how well an Analytical Mind answers questions whose correct answers you already know. Use it before you roll a Mind out, or after you change its model, thinking effort or response mode, to compare the results.
Open the Mind and select Benchmarks; for finding and opening a Mind, see Find and open a Mind.
The Question Bank holds the questions that every experiment runs. Each question has a text, four options and one correct answer.
While an experiment is Running on the Mind, the bank shows "Questions are locked while an experiment is running on this mind", and Add question, Edit question and Delete question are disabled. A paused experiment does not lock the bank.
Deleting a question asks Delete question? and explains that the question will be removed from the bank, while question snapshots inside already-launched experiments keep their copy. Confirm with Delete.
A longer bank shows in pages of 10, with a line like "Showing 1-10 of 34" and Previous and Next.
An experiment runs every question against each variant, as many times as you set. A variant is one combination of model, thinking effort and response mode.
| Setting | What it does | Default |
|---|---|---|
| Name | Names the experiment in the list. Required. | Empty |
| Variants | Each variant is one model, thinking effort and response mode. Every question runs against every variant. | One row, with the Mind's default model, the Default thinking effort and Analyzer |
| Instances per question | How many times each question runs per variant. Range 1 to 50. | 1 |
| Max concurrent | How many conversations run in parallel. Range 1 to 50. | 10 |
| Show options upfront | Off: the options reach the model only when it is asked to answer. On: the options are included in the first message. | Off |
| Gliner PII scan | Scans the question and option texts once at launch and rejects the launch if personal data is found. Shown only while global PII detection is on. | Off |
Show options upfront changes what the score means. Off costs more tokens but measures real analysis. On is faster and cheaper, but a model can score well by eliminating options.
The Model list offers the models the Mind may use. If your role has Modify AI Configuration, it offers every model in the catalogue. See Choose models and AI behaviour.
Every conversation uses real model tokens. A large experiment can use a significant part of your usage.
Open an experiment by selecting its row in the Experiments list; its details replace the list until you go back with Back to experiments.
One card per variant, with the variant's own label, its score out of 100 to one decimal place, Avg time to answer, and chips counting its correct, wrong, failed and pending items. A variant with nothing scored yet shows N/A.
A table with one row per question, numbered #1, #2 and so on, and one column per variant. Each cell shows the correct answers out of the answered items, a warning marker when some items failed, and a pending count.
The score is the number of correct answers divided by the number of correct and wrong answers, multiplied by 100, shown to one decimal place. Failed items are left out of both figures. A variant with no scored answers shows N/A.
Every item has one of these states, and the diagram uses them as its legend:
| State | Meaning |
|---|---|
| Queued | Waiting to start. |
| Running | In progress. |
| Correct | Answered with the correct option. |
| Wrong | Answered with another option. |
| Failed | Could not be scored. It is not counted as wrong. Hover a failed item in the diagram to read the reason. |
Diagram opens the whole experiment full screen: one column per variant, with every question and each of its instances as a node. Select an instance to open its conversation.
Each item runs as its own chat on the Mind. The question message in that chat carries a From Benchmark badge, and the answer card shows the chosen option, such as Answer B - Correct or Answer B - Incorrect; see Message actions and editing. While the item is queued or running, the message box is locked and shows Benchmark in progress, and it unlocks on its own when the item finishes; see Chat states and recovery. Benchmark chats are not listed in your chat lists, so open them from the diagram. Inside such a chat, a strip above the message box reads Benchmark · this chat belongs to an experiment item and offers View diagram; opened from there, the diagram centres on that item with a You are here mark.
The buttons sit on the experiment's row in the Experiments list and in the header of the opened experiment.
A completed experiment keeps its results and can only be deleted.
You need Edit Mind; without it, the tab is not in the tab bar at all. The tab is also desktop only, and it is for Analytical Minds. On a Warehouse Builder or Lake Builder Mind it is greyed out, and its tooltip reads "Benchmarks are for analytical Minds." See Who can do what with Minds.
One for each question, instance and variant combination. Twenty questions, two instances and three variants create 120 chats; the Launch experiment? window shows the exact number.
An experiment on the Mind is Running, which locks the bank. Stop it with Stop experiment, and the questions unlock once no experiment is running.
Leave it off to measure real analysis. Turn it on only when speed and cost matter more, and accept that a model can score by elimination.
Yes. It removes the experiment's results and every conversation it created. The questions in the Question Bank stay.
A model, a thinking effort and a response mode. Every question runs against every variant, so the scoreboard compares them side by side.
Correct answers divided by correct and wrong answers, multiplied by 100, to one decimal place. Failed items are not counted. A variant with no scored answers shows N/A.
Yes. Stop experiment pauses it and keeps the results. Continue experiment resumes it from where it stopped.
The chat belongs to an experiment item that is still queued or running. The Benchmark in progress badge stays until the item finishes, then the box unlocks on its own. See Chat states and recovery.