Marking queue

/ui/marking-queue is a cross-pipeline review queue of steps with no human accuracy feedback yet, grouped by pipeline then step, oldest first, with pipeline/team/stage (default testing) additive filters and stat cards (pipelines/runs/steps affected, marked coverage %). Each chip links out to /ui/runs/{id} to review and mark one step; operator and admin can also mark steps in bulk from here.

Why this page exists

Every knob on the accuracy and calibration tiers ultimately needs marked evidence to resolve against, and accuracy.min_human_marked specifically needs a human to have graded the step directly — not an inherited run-level rating, and not a deterministic check’s automatic label. The Marking queue is a single cross-pipeline view of everything still needing a human mark, rather than opening each pipeline’s Accuracy feedback page one at a time.

What it lists

It lists every step with no StepFeedback (i.e. human_marked would not count it), grouped by pipeline then by step (fan-out/parallel branches collapse to their group name, same as everywhere else in the readiness system), oldest first. A step that already has an automatic label — a failed deterministic check, or an inherited run-level rating — is still listed, but tagged with where that label came from, since neither one satisfies min_human_marked. Stat cards at the top give the shape of the backlog (pipelines affected, runs affected, steps unmarked, and marked-coverage %); filters for pipeline, team, and stage are additive.

Stage defaults to testing — the pre-promotion review case — but is a real, selectable filter, not a hard scope: a stage: production pipeline using calibration on its own merits (independently of any stage: testing → production gate) needs its backlog visible here too, so production and “all stages” are both one click away.

Every chip links to the run on /ui/runs/{id}, where the existing accuracy feedback widget (run-level and per-step) lets you review and mark one step at a time.

Bulk marking

When you have already reviewed a batch of steps and they all deserve the same grade, mark them together instead of opening each run. As an operator or admin:

1. Tick the checkbox on individual step chips, and/or click select all N next to a step name to select every unmarked item of that step, including those beyond the five chips shown.
2. A bar appears — “N step(s) selected” — with an optional notes field and Correct / Partial / Incorrect.
3. Choosing an outcome opens a confirmation stating the count; nothing is written until you confirm.
4. The page reloads with a summary (“Marked 12 step(s) correct (12 new, 0 updated)”), and the marked steps leave the queue.

Things to know:

• Fan-out branches are marked exactly. A selected triage/0 marks that branch execution, not the whole group, even though the queue displays branches collapsed under triage.
• It counts as human evidence. A bulk mark satisfies accuracy.min_human_marked and feeds calibration and promotion readiness exactly like a single mark, so use it for steps you have actually looked at, not to clear the backlog quickly. The select-all button deliberately selects unseen items too; the confirmation exists to make you pause on that.
• Overwrites. A step that already has a human mark would be updated, but unmarked-queue items by definition have none, so this mainly matters when the same step is also graded elsewhere meanwhile.
• Limit. One action carries at most feedback.bulk_max steps (default 500); the buttons disable above that and you do it in batches.
• Audit. One feedback.bulk_submit row per action (target: steps).
• Viewers see the queue as before, with no checkboxes.

The same operation is available as POST /feedback/steps/bulk and the bulk_submit_step_feedback MCP tool.

A step produced by a replay batch shows up here like any other unmarked stage: testing step, tagged REPLAY instead of “failed check” or “run feedback” — that tag takes priority over the others since it’s the fact a marker needs first: this step’s output came from a candidate configuration under evaluation, not a normal pipeline run.

Relationship to the confidence system

The marking queue is one of several places confidence and calibration make themselves visible in the UI — for the readiness-criteria and calibration angle on this same underlying data (why marked evidence matters, how min_human_marked and the calibration bins use it), see How confidence and calibration work — Where to actually see all of this.