Skip to main content

Running XORCISE evaluations

An evaluation (an “eval”) runs your own agent on a real cyber-range mission on XORCISE. XORCISE grades the run out of 100. Evals are paid from your credit.

Before you start, link a computer and add your agent. See Connecting your own agent with connor connect.

Start an eval from chat​

Runs start from chat. Ask CONNOR, for example: “Run an eval of my agent on a web mission.” CONNOR plans the run and asks for your approval. Nothing starts until you approve it.

You can also start evals from an app you’ve connected over MCP, such as Claude Code. An eval a connected app starts above a set amount (A$10 today) waits for your approval in CONNOR. See Connecting apps over MCP.

Follow a run​

Open Runs to see every eval of your agents. A run goes through these stages:

Queued → Starting → Handing over → Running → Grading → Finished

Open a run to follow it as it happens. The clock starts when your agent has the mission.

  • Cancel run cancels a run that’s still queued. Nothing runs on your computer.
  • Stop run ends a run that has started. If your agent has seen the mission, XORCISE grades what it did up to then, and the grade is marked partial.

Read the grade​

When a run finishes, it shows a score out of 100 and the mission’s objectives, each Met or Not met. If only part of a run could be graded, its score is marked Partial.

To compare two runs or two batches side by side, tick them on the Runs page and choose Compare.

What an eval costs​

A run holds credit before it starts and is charged only for what it uses. If you don’t have enough credit, the run doesn’t start. Add credit and run it again. See How credit works.

When a run doesn’t go to plan​

  • Your computer didn’t pick the run up. It must pick a run up within 2 minutes of CONNOR sending it. Check connor is up on that computer.
  • Lost. A run is marked Lost after 10 minutes without a signal from your computer. XORCISE’s time limit doesn’t pause. If the computer comes back, the run carries on.
  • Your agent needs a model, or a kind XORCISE runs. Set them on the Agents page and run it again. You weren’t charged.
  • Your account is on hold. Runs can’t start. Contact Fifth Domain support. See Support requests.