Testing

A test set is a list of texts with the answers you expect. Run it on a version of a prism and the Test tab shows where that version agrees with your labels and where it misses. Run it on two versions and it shows which one is better, question by question, and whether the difference is bigger than chance.

Only admins have the Test tab.

Test sets

Each case in a test set holds a text's fields, a reference, an optional context and group, and the expected answer for some or all of the questions. A question with no expected answer is not checked for that case.

From a labelled file

Run a file with expected-answer columns on Try, then click Save as test set. Save as a test set covers the dialog. New test set from a file in the Test rail opens Try at the file run.

From a result

Any result that kept its text can become a case. Open it in Rows and click Add to test set, or select several rows and add them one by one. The expected answers start from what the result reads as, and you correct the wrong ones. See Add to a test set.

Revisions

Saving into an existing set makes its next revision. A revision never changes. A run keeps pointing at the revision it answered, so older runs stay comparable.

A case keeps its id across revisions when its reference stays the same, so the app can follow one case from revision to revision. You can also fix a label from a case's panel when you compare two runs in Summary. That saves the set's next revision at once.

When you view runs of an older revision, a case whose text did not change is scored with its newest label. The bar then says Scored with the labels of revision N, and a fixed label counts without asking the model again.

The rail shows Delete older revisions and Delete test set for the set you chose. Deleting older revisions also deletes their runs. Neither can be undone.

Limits

A test set holds at most 1,000 cases and 5 MB of them.
Each case needs its own reference within the set.
A test set keeps the text of its cases until you delete it, whatever the prism's input storage. If the prism keeps only a fingerprint or nothing, the app asks you to confirm first.

Run a version

Choose a test set in the rail. Under New run, pick the version in Version to run and click Run. Any saved version can run, published or not. The box says how many calls to the model the run makes: one per case.

The app sends eight cases at a time and shows its progress. The figures show once every case has an answer.

Keep the tab open. Leaving the Test tab stops the run after the cases already sent.
Run the N remaining cases, in the bar, finishes a run that stopped. Failed cases are asked again. A case that has an answer is never asked twice.
One run answers at a time.
Test cases count against a per-account limit each minute. When the account reaches it, the app waits and goes on.

The rail

The left rail lists Test sets, each with its revision and case count. Under the chosen set, Runs lists its runs, newest first. Each run shows its version, marked live or saved, its revision, its date and how many cases it answered. A run shown as B is the one in view. A run shown as A is the one it is compared with.

Click a run to show it. The bar above the view has against, which picks the run to compare with, or Nothing to see the run alone. Swap swaps the two.

Summary

Summary changes with what is in view.

One run

With one run alone, Summary shows its Evaluation: one block per checked question, as in a file run's evaluation report. Download evaluation and Download misses save it as CSV, with each case named by its case id. A set with a holdout shows a separate Holdout report with figures only.

Two runs

With two runs of one revision, Summary opens with a one-line verdict, such as "Version 5 is better on Routing team." or "Keep version 4 live.". A short paragraph says which questions got better or worse, which only lean one way, and which changed within noise.

The Per question table has, for each question:

Agreement in each run,
Change, with its 95% range,
Flipped: how many answers were fixed and how many broke,
Real or noise?: Better, Worse, Leaning better, Leaning worse or Can't tell,
Held out, the agreement on the holdout.

Click a question to see its cases, in Newly wrong, Fixed, Near the edge and Any change. Each case shows its text, your label and both runs' answers. Click one to open it beside the list. j and k move through the list. There you can change a label.

A question one version does not ask says Not asked in version N.

How a verdict is reached

Only answers that flipped between the two runs count: wrong in one, right in the other.

Better or Worse: more answers were fixed than broke, or the reverse, by more than chance explains 1 time in 20.
Leaning: the split is likely but not settled. Leaning edge, in the bar, sets how likely. Changing it there applies to the view, and Save for this prism keeps it. The account default is on Defaults.
Can't tell: the flips are close to even, or too few to read.

A flip where both answers sat within 5 points of the edge that decided them is left out, because asking the same text again can land either way. Those flips count only when a question's near-edge flips lean one way by more than chance explains.

Under the verdict, a line can say how many more labelled cases could settle a leaning question, and how many flips a set of this size needs before any change reads as real.

Comparing two runs of the same version shows how much the model's answers move when asked again.

Report

Report needs two runs. It has:

Change per question, a chart of each question's change with its 95% range. A bar is red or green only when the change is real. Copy as markdown copies the comparison as a table.
Precision and recall per option, for each Choice question both versions ask. A rate from fewer than 10 cases is grey.
Yes / No at a cutoff, which places every labelled case by its % yes and shows each run's precision, recall and Unsure. Try a Yes cutoff moves the cutoff for the view without changing the Lens.

How cases are scored

Each run is scored against your labels the way the Lens reads its answers, so the figures agree with what callers receive.

Every block counts its cases before it gives a rate:

scored: the case has a readable expected answer and an answer.
without an expected answer: not checked for this question.
not run: the case was never answered, for example because the run stopped.
failed: the model gave no answer.
no valid answer: the model named an option the question does not have. It counts as a miss.
abstained: the answer was Other, Not enough information or Unsure. Other and Not enough information count in every figure. An Unsure Yes / No answer is left out of agreement, recall and precision, and its share is shown as Unsure.

Not run and failed cases are left out of every rate.

Every rate shows its sample beside it, such as 62% (31 of 50), so a figure from three cases never reads like one from 300.

The holdout

A set of 50 cases or more keeps about one case in five as a holdout. The app chooses these cases by their id, so a case stays in or out of the holdout across revisions.

Holdout cases are scored but never listed. They show only as counts and rates, in the Holdout report, the Held out column and a line under the verdict. When you fix labels or rewrite questions to win back the cases you can see, a prism can learn those cases and still miss new texts. The holdout is the check on that: if a change helps the listed cases but not the holdout, it may not help real texts.

The rail says how many cases are held out.

Test verdicts elsewhere

Publish dialog. When you publish a version, the dialog looks for the newest pair of finished runs, on one revision of one set, of the live version and the one you are publishing. If it finds them, it says whether the new version is worse on any question, such as 'On "Support sample", version 5 against version 4: Routing team is worse (6 broke, 1 fixed).', with See it in Test. When the new version is worse, the dialog focuses Keep version N instead of Publish. Without such a pair, it says so and links to Open Test.

Overview. The Live version box on a prism's Overview shows the verdict of the newest test run.

Home. Home raises a run that stopped before every case had an answer, and a saved version that waits to be tested or compared.

Test answers are not results

A test run's answers are kept with the run, not as results.

They don't show in Results, in exports or in the API.
Results retention does not delete them. They stay until you delete the test set, or its older revisions with their runs.