Analytics API

These endpoints were added so the VectorStep Service MCP can author and inspect pipelines without importing the VectorStep codebase. The analytics endpoints share their aggregation queries with the /ui/insights/* pages, so the numbers they return can never drift from what the UI shows for the same data.

GET /pipelines/{name}

Full resolved pipeline config (.model_dump()) plus the raw YAML text from disk.

→ {"config": {...}, "yaml": "name: ...\n..."} — 404 if unknown.

GET /steps

Library step summaries: name, description, tags, executor, agent.

→ {"steps": [{"name": "...", "description": "...", "tags": [...], "executor": "...", "agent": "..."}]}

GET /steps/{name}

Full step config plus raw YAML from the step library directory.

→ {"config": {...}, "yaml": "..."} — 404 if unknown.

GET /agents

Agents merged across OpenClaw and VectorStep Gateway backends — read-only. A down backend contributes an empty slice plus an errors entry, never a 500.

→ {"agents": [{"name": "...", "executor": "openclaw"|"gateway", "model": "..."}], "errors": {}}

GET /pipelines/{name}/stats

Operational and judged-accuracy stats for one pipeline.

Query params: time_range (24h|7d|30d|all, default 7d), stage (production|testing|all, default production, matching every other rollup in the app — see Pipeline stages).

GET /pipelines/{name}/stats?time_range=7d&stage=production

→ {"pipeline_name", "runs_total", "status_counts": {...}, "success_rate", "tokens": {"input", "output", "total"}, "cost": {"total", "unpriced_steps", "currency", "by_source"}, "duration_seconds": {"avg", "p95"}, "accuracy": {"correct", "partial", "incorrect", "total", "correct_pct"}, "teams": [...]}

Note

success_rate = completed ÷ terminal runs (excludes still-running); accuracy is the separate judged rollup from RunFeedback — null/zeroed when nothing’s been graded, which is not the same as “0% accurate”. cost.total is a SUM that skips NULL (unpriced) steps; cost.unpriced_steps is a separate count of how many were skipped, so this can never be mistaken for a complete total. cost.by_source is {"manual", "live_exact", "live_cross_provider"}, each a SUM over that one persisted pricing tier — the three always sum to total. total/unpriced_steps/currency are unchanged; by_source is additive — see Cost accounting for what each tier means. 404 if the pipeline name is unknown.

GET /stats/pipelines

The same per-pipeline payload as above, for every pipeline with a run in range — the JSON behind the /ui/insights/pipelines table. Rank client-side to answer “which pipeline fails most / costs most / is least accurate”.

GET /stats/pipelines?time_range=7d&stage=production

→ {"pipelines": [{...}, ...]}

GET /pipelines/{name}/promotion-readiness

Owner-defined promotion-readiness readout for one pipeline (see Promotion readiness) — advisory only, does not gate or block stage: testing -> production. Evaluates every step against its effective readiness: config (four independent tiers: operational, confidence, accuracy, calibration) against evidence from the pipeline’s own current stage. Works for a pipeline of either stage; 404 if the pipeline name is unknown.

Note

bin_width/n_min are the INFORMATIONAL defaults used only for the observed-calibration snapshot on steps with no calibration tier configured — a configured step’s own values always win, and neither is read from config.yaml’s calibration: block (which only feeds the live runtime gate).

GET /pipelines/{name}/promotion-readiness?bin_width=0.1&n_min=20

→ {"pipeline_name", "pipeline_stage", "evidence_stage", "criteria_source": "configured"|"none", "verdict": "ready"|"not_ready"|"building"|"no_data"|"not_configured", "gathered_at", "summary": {"ready","not_ready","building","no_data","not_configured","total"}, "steps": [{"step_name", "kind", "executor", "when", "verdict", "confidence_threshold", "criteria": {tier: {...knobs, "source"} | null, ...}, "current_config": {"prompt_hash", "agent_version", "agent_version_source", "prompt_hash_source", "prompt_hash_matches_history"}, "evidence": {"runs_total", "rows_total", "marked_total", "first_seen_at", "last_seen_at"}, "tiers": {"operational", "confidence", "accuracy", "calibration": {"verdict", ...}}, "observed_combos": [...], "narrative": ["...", ...], "notes": ["...", ...] }, ...]}

POST /pipelines/{name}/promotion-readiness/preview

Preview a candidate readiness: block against the same evidence, without writing anything — the same response shape as the GET above, plus applied_to, scope ("pipeline"|"step"), yaml_snippet/ yaml_snippet_indented (server-generated, validated through the real ReadinessConfig — this endpoint cannot be used to produce YAML the loader would reject), yaml_target_hint, evidence_gathered_at, evidence_cache_hit (a 30s in-process cache, used only by this endpoint — the GET above always re-gathers fresh).

• body: {"readiness": {"operational": {"min_runs": 20}, ...}, "apply_to": ["step-name"] | null}
• 422 with Pydantic’s own detail on an invalid readiness: body; 400 if apply_to names a step that doesn’t exist.

GET /steps/{name}/stats

Same stats shape as /pipelines/{name}/stats, aggregated per step name across every pipeline that uses it (avg_input_tokens/avg_output_tokens instead of teams). Works for any step name with real execution history, not just steps registered in the step library via use: — a step defined inline in a pipeline’s own steps: list is included too. 404 only for a name nothing recognises: not in the step library, and no recorded execution anywhere.

GET /steps/{name}/stats?time_range=7d&stage=production

GET /steps/{name}/models

Per (agent, model, provider) breakdown for one step — un-blends the single aggregate above so “which model performs best for this step” can actually be answered (success rate, avg tokens, avg duration, judged accuracy per model, instead of averaged across every model the step has ever run under). Same rows as the existing /ui/insights/steps drilldown’s per-agent/model table. Same step-name resolution as /steps/{name}/stats above — works for inline steps with history, 404s only for a name with no history anywhere.

GET /steps/{name}/models?time_range=7d&stage=production

→ {"step_name": "...", "breakdown": [{"agent", "provider", "model", "runs_total", "status_counts", "success_rate", "avg_input_tokens", "avg_output_tokens", "avg_duration_seconds", "accuracy": {...}}, ...]} — sorted by runs_total desc.

GET /steps/{name}/calibration

Calibration bins for one step, per (agent, model, provider, prompt_hash, agent_version) — the same data behind the /ui/insights/steps calibration bins, exposed as its own stat.

Note

NOT time_range/stage scoped — always the step’s full production history, since calibration is a track-record measurement (windowing it would defeat the point). Works for any step with real history, including one defined inline in a pipeline rather than via use: — 404s only for a name with no history anywhere, the same resolution /steps/{name}/stats uses.

prompt_hash/agent_version are part of the bucket key so editing a prompt or a Gateway agent’s config starts a fresh bucket instead of silently pooling with the old one’s history. See Editing a prompt or an agent resets the measurement — deliberately.

GET /steps/{name}/calibration?bin_width=0.1&n_min=20

→ {"step_name": "...", "buckets": [{"agent", "provider", "model", "prompt_hash", "agent_version", "total_n", "bins": [{"lo", "hi", "n", "mean_label", "validated"}, ...] (always 1/bin_width bins), "recommendation": "runs scoring ~90% here are only 75% correct..." or null }, ...]} — sorted by total_n desc.

GET /steps/{name}/versions

Every prompt template version VectorStep has recorded for this step, newest first — “did that prompt edit actually help?” made answerable with data. Each version carries its diff against the version before it (null for the oldest) and its own calibration data scoped to just that version. NOT time_range/stage scoped, same reasoning as /calibration above. Same step-name resolution too — works for an inline step with history, 404s only for a name with no history anywhere.

GET /steps/{name}/versions

→ {"step_name": "...", "versions": [{"prompt_hash", "template", "first_seen_at", "last_seen_at", "runs_total", "labelled_n", "diff_from_previous", "calibration": [...] }, ...]} — sorted by first_seen_at desc.

Caution

May include one synthetic entry with "prompt_hash": null, for any run with no recorded hash (PipelineStep.prompt_hash IS NULL), which has no hash to register under. Its "template" is also null (no text was ever captured), but "runs_total"/"labelled_n" are real — don’t mistake this for the step having no history. "diff_from_previous" is null both for this entry and for whichever real version comes right after it (nothing to diff against).

GET /agents/{name}/versions

Every Gateway agent_version VectorStep has a snapshot for, newest first — the only place recoverable text for an agent_version survives once the Gateway agent moves on to a newer config. name is the bare agent name (no gateway: prefix). 404 if VectorStep has no snapshot rows for this agent at all.

GET /agents/{name}/versions

→ {"agent": "gateway:...", "versions": [{"agent_version", "soul_md", "agent_yaml", "note", "first_seen_at", "last_seen_at", "diff_from_previous", "used_by_steps": [...] }, ...]} — sorted by first_seen_at desc.

Note

note (instead of soul_md/agent_yaml text) means VectorStep couldn’t confirm that snapshot, not that the agent had no config. Same synthetic-entry behaviour as /steps/{name}/versions above, for rows where PipelineStep.agent_version IS NULL — "agent_version": null, "soul_md"/"agent_yaml": null, "note" explains why in plain language, "used_by_steps" is still real.

GET /stats/skills

Per-skill load rate, usage totals, and the loaded-vs-not-loaded comparison, for every skill with usage in range — the same aggregation the Skills list page, skill detail’s Usage tab, and Insights → Skills all share (skills_stats.py, computed once).

GET /stats/skills?time_range=7d&stage=production

→ {"skills": [{"name", "granted_steps", "loaded_steps", "not_loaded_steps", "load_rate", "loads", "file_reads", "script_runs", "script_errors", "errors", "loaded_failure_rate", "loaded_avg_effective_confidence", "not_loaded_failure_rate", "not_loaded_avg_effective_confidence", "pipelines": [...], "by_version": {...}}, ...]} — sorted by name.

load_rate is loaded_steps ÷ granted_steps as a percentage: of the steps whose agent was granted this skill, what fraction actually loaded it (load_skill called with loads > 0) — the number to watch when iterating on a skill’s description. by_version splits loaded_steps/loads/ first-seen/last-seen per skill version observed in range. The loaded_*/not_loaded_* pairs compare outcomes between steps that loaded the skill and steps that were granted it but didn’t — correlation, not causation, since an agent may reach for a skill specifically on harder inputs.

GET /skills/{name}/versions

Every skill version VectorStep has a snapshot for, newest first — the only place recoverable text for a skill version survives once its SKILL.md, bundled files, or scripts change on the Gateway. 404 if VectorStep has no snapshot rows for this skill at all.

GET /skills/{name}/versions

→ {"skill": "...", "versions": [{"version", "description", "skill_md", "files": {...}, "note", "first_seen_at", "last_seen_at", "diff_from_previous", "steps"}, ...]} — sorted by first_seen_at desc.

Note

note (instead of skill_md/files text) means VectorStep couldn’t confirm that snapshot, not that the skill had no content — same honesty rule as /agents/{name}/versions above. A bundled file the Gateway reported as binary, or one that would push the snapshot over its 512 KiB cap, is stored as null within files even when the version’s own text was confirmed; note explains that case too. steps counts distinct recorded steps whose PipelineStep.skills JSON shows this exact (name, version) pair as loaded.