Tutorial: gate a pipeline on budget

Everything gated so far has been about whether to trust an answer. This tutorial gates on something completely different: whether the pipeline is getting too expensive to keep running — using either budget.max_usd or plain token count, independent of confidence, deliberately.

Models used for testing

This series was validated internally against openrouter/anthropic/claude-sonnet-4.5 for first-responder and openrouter/anthropic/claude-haiku-4.5 for every other agent. Other models and providers should work too, but exact wording and confidence numbers can vary by model.

Prerequisites

Route escalations to a real channel completed. ~/.vectorstep/service/pipelines/alert-triage.yaml is identify-upstreams → check-upstreams (fan-out, with a verifier) → consolidate → write-up. first-responder isn’t part of this pipeline any more — it was retired back in the fan-out tutorial — so every step here runs through either generic-pipeline-step or upstream-checker, and per their agent.yaml files, both use the same model: anthropic/claude-haiku-4-5-20251001.

Cost in VectorStep is never one number from one source — every priced component resolves to one of three tiers, and all three are genuinely persisted (never recomputed later, never an ephemeral display-time guess). This tutorial works through all three, then gates on the result two ways.

1. Add a manual pricing table

One pricing.models entry covers this entire pipeline, since every step resolves to the same model. Add it to ~/.vectorstep/service/config.yaml:

pricing:
  currency: USD
  models:
    - match: {provider: anthropic, model: "claude-haiku-4-5-20251001"}
      input_per_mtok: 1.00        # verify against your provider's current pricing
      output_per_mtok: 5.00

Before this, every step’s cost was NULL, never 0 — NULL means “unpriced, unknown,” 0 means “priced, and this genuinely costs nothing” (e.g. a local model an operator explicitly rates at zero). If any component that ran (primary, verifier, grounding-judge) has no tier match, the whole step’s cost stays NULL rather than a silently partial sum built from only the components that happened to price.

Unlike the artifact-store and Telegram config changes in the last two tutorials, this one is picked up by /reload — pricing is one of the handful of config.yaml keys re-read on every reload/SIGHUP, not just at startup:

curl -X POST http://localhost:8000/reload

2. See the real cost

curl -X POST "http://localhost:8000/webhook?source=alertmanager&allow_testing=true" \
  -H "Content-Type: application/json" \
  -d @tests/fixtures/alertmanager_critical.json

What you should see

Open the run and look at each step’s cost badge — it should now show a real figure in blue (manual pricing’s colour), not “not priced.” A genuinely sub-cent cost (likely, for a haiku-class model on short prompts) is shown to 4 decimal places instead of rounding to a misleading $0.00 — e.g. $0.0031 rather than $0.00. Expand check-upstreams’s branches and hover a cost badge — since each branch has both a primary call and a verifier call, the tooltip itemises both: “Primary: $0.0021 — your pricing.models configuration” / “Verifier: $0.0018 — your pricing.models configuration”.

3. Live OpenRouter pricing — the other two tiers

Manual pricing is exact but has to be typed in and kept current by hand. The other two tiers come from OpenRouter’s public catalog instead, and are priced and persisted the same way manual pricing is — once, at step-save time — so a cost badge never changes after the fact just because OpenRouter’s prices moved.

Comment out or delete the models: entry from step 1 first. Manual pricing always wins over any live tier when both could match the same step — leave it in place and every badge below will keep showing blue manual pricing no matter what you set next, which looks exactly like live pricing “not working” when it’s actually working correctly. In ~/.vectorstep/service/config.yaml:

pricing:
  # models:                          # commented out — manual would otherwise keep winning
  #   - match: {provider: anthropic, model: "claude-haiku-4-5-20251001"}
  #     input_per_mtok: 1.00
  #     output_per_mtok: 5.00
  live_pricing:
    enabled: true
    refresh_interval_seconds: 3600
    allow_cross_provider: false   # leave off for now — see below

With just enabled: true, this only ever fills a gap for a component whose provider genuinely is openrouter — a match against the exact model that was actually called. This series’ own agents are configured with plain anthropic/... model strings (per the note at the top, real testing routed them through openrouter/anthropic/claude-haiku-4.5 instead) — if that’s your setup too, this is the tier you’d see.

This needs a full restart, not POST /reload. The OpenRouter catalog is only fetched eagerly once, at service startup — turning live_pricing on via /reload alone schedules the next fetch up to refresh_interval_seconds away (an hour, at the default), so nothing would price until then. Stop the service (Ctrl-C) and start it again:

cd ~/.vectorstep/service
source .venv/bin/activate
uvicorn src.main:app --reload --port 8000

Check the startup logs for OpenRouter catalog refreshed: N model(s), then re-trigger. Badges should show green instead of blue, with a tooltip naming the exact OpenRouter catalog id and per-Mtok rates matched.

Still seeing blue after this?

Manual pricing always wins when it matches — this is the single most common reason live pricing looks like it isn’t working. Double-check the models: entry above is actually commented out or removed, not just left in place alongside live_pricing:, and that you restarted (not just reloaded) after enabling it for the first time.

The third tier: cross-provider approximation

Suppose a step ran directly against Anthropic (not OpenRouter) and has no manual rate — live_pricing.enabled alone won’t price it, because its provider isn’t openrouter. Turning on allow_cross_provider lets a similar-sounding OpenRouter catalog listing price it anyway. In ~/.vectorstep/service/config.yaml:

pricing:
  live_pricing:
    enabled: true
    refresh_interval_seconds: 3600
    allow_cross_provider: true

This is a genuinely weaker claim than the other two tiers — a different vendor’s rate, possibly under different contract terms, matched by a fuzzy name comparison — which is why it needs this second, explicit opt-in on top of live_pricing.enabled rather than happening automatically. A step priced this way shows amber, with a hover explaining it’s a cross-provider estimate, not the rate for the API that was actually called.

allow_cross_provider is read the same way enabled is — a plain /reload picks up the flag itself, but if the catalog hasn’t been fetched yet in this process (see the restart note above), there’s still nothing to match against, so restart rather than reload if you haven’t already turned live_pricing on this session.

Keep the models: entry commented out from the previous step — with it still in place, this tier faces the exact same problem live-exact did: manual wins, nothing changes. Reload (this one’s just a config value, no new catalog fetch needed since live_pricing is already on) and re-trigger — this pipeline’s steps run directly against anthropic per their agent.yaml model strings, so you should now see amber badges where step 2 showed blue ones.

Mixed-tier steps

A step’s primary, verifier, and grounding-judge calls are priced independently, so they can land on different tiers on the same step — e.g. a manually-priced primary next to a verifier that fell through to a live estimate. Both amounts are kept on separate columns, nothing is blended, and the badge shows the weakest tier present with a ✻ marker so it never looks more certain than its shakiest component. Not something this pipeline happens to trigger, but worth knowing before you see it on a real deployment — see Cost accounting for the full model.

Put the manual pricing.models entry from step 1 back before continuing — the rest of this tutorial uses it for a predictable, exact number.

4. Force a budget abort

Add a budget: block at the pipeline level — a sibling of notifications:/ context_template:, not nested under steps::

budget:
  max_usd: 0.0001   # deliberately unreachable — see below

Reload and re-trigger:

curl -X POST http://localhost:8000/reload
curl -X POST "http://localhost:8000/webhook?source=alertmanager&allow_testing=true" \
  -H "Content-Type: application/json" \
  -d @tests/fixtures/alertmanager_critical.json

What you should see

The run should come back status=aborted — regardless of which step it was partway through, since the accumulator checks the running total after every completed step, not just at the end. The Run log should show a budget_exceeded event naming both figures: “Cost budget exceeded: 0.00 USD used (limit: 0.00 USD)” (both round to $0.00 at 2dp here — the point is the accumulated cost exceeded the ceiling at all, not the display precision).

One connection worth noticing: budget-exceeded fires the pipeline’s notifications.notify action — not escalate, which is what check-upstreams’s own low-confidence path fires. If you only added an escalate: entry in the previous tutorial, nothing will have gone to your log/Telegram channels for this abort; add a notify: entry alongside it if you want to see a budget abort actually reach a channel too.

5. Choose which tiers count: count_pricing

All three tiers are always persisted in full, regardless of what you’re gating on — count_pricing only decides which of that already-real money is allowed to abort a run. It’s a trust threshold, not a checklist: one value naming the least-trusted tier still allowed to contribute, in the same order the tiers already have.

budget:
  max_usd: 5.00
  count_pricing: manual   # only rates you typed in yourself can trip this run

count_pricing	Counts toward max_usd
manual	Manually-priced cost only
live_exact	Manual + live-exact
live_cross_provider (default)	Everything, including cross-provider estimates

Excluded spend is still real spend

The default, live_cross_provider, counts everything — including amber-tier cross-provider estimates, if §3’s allow_cross_provider is on. If you’d rather only a rate you configured yourself be able to abort a run, set count_pricing: manual.

Excluding a tier doesn’t mean that money wasn’t spent — it means the guardrail can’t see it. The first time a run excludes any spend, VectorStep logs a budget_cost_excluded run-log event naming the step and the amount, so this is never silent. Try it: set count_pricing: manual while allow_cross_provider: true is still active from §3, reload, and re-trigger — you should see this event in the run log even on a run that completes normally, and if the run does trip max_usd, the abort message reports both figures: “…(limit: 5.00 USD) — additionally, 0.02 USD was spent but not counted toward this limit (excluded by count_pricing).”

count_pricing can also be set on an individual StepConfig step, None inheriting the pipeline’s value — but not on a parallel/fan_out group, only a plain step:

  - name: write-up
    count_pricing: live_exact   # this one step trusts live-exact but not cross-provider guesses
    ...

Since check-upstreams in this pipeline is a fan-out, it always uses the pipeline-level budget.count_pricing — there’s no per-branch override to reach for there.

Once you’ve finished experimenting, set allow_cross_provider: false and count_pricing: live_cross_provider (or remove count_pricing entirely — that’s already the default) before moving on.

6. Raise the budget back to something realistic

budget:
  max_usd: 5.00

Reload and re-trigger once more — normal operation resumes, and the run completes exactly as it did in step 2. The point of budget.max_usd is a ceiling, not a target: it should sit comfortably above what a healthy run actually costs, only tripping when something is genuinely running away (a loop that isn’t converging, a fan-out with far more branches than expected).

7. Gate on tokens instead of price

Sometimes you don’t want to trust any price — a brand-new model with no manual rate yet, or a deployment that’s deliberately left live_pricing off. budget.max_tokens gates on raw token count, entirely independent of pricing:

budget:
  max_tokens: 5000       # abort run if accumulated tokens across all steps exceeds this
  max_usd: 5.00           # can be set together with max_tokens — whichever trips first wins

Reload and re-trigger — with a limit this tight, the run should abort with a Token budget exceeded: N tokens used (limit: 5,000) run-log event, independent of anything in this tutorial’s pricing setup. max_tokens and max_usd can be set together (at least one is required if budget: is present at all); whichever trips first names itself in the abort message. Raise max_tokens back to something realistic (or remove it) once you’ve seen it fire.

8. Where the same numbers roll up

The persisted cost you’ve been pricing feeds several other surfaces without any extra configuration, each now broken down by tier:

• Insights — the pipelines/teams/models/providers pages all carry a cost column/card alongside tokens, with a per-source breakdown. This pipeline has been stage: testing all series, and every Insights page defaults to production-only — switch the Stage selector at the top of any Insights page to Testing (or All stages) to actually see this pipeline’s numbers before you promote it. See Insights.
• Prometheus — vectorstep_pipeline_cost_total{pipeline, team, model, provider, source} — one counter, source one of manual/live_exact/live_cross_provider. sum without(source) for the total, or filter on source for just the tiers you trust.
• Team budgets — if you add a pricing.team_budgets entry (e.g. team_budgets: { platform: 1000 }, a per-team currency-units-per- calendar-month figure), /ui/insights/teams shows a spend-vs-budget bar and the vectorstep_team_budget_ratio{team} gauge. This is advisory only — going over it never blocks a run the way budget.max_usd does; enforcement stays per-pipeline.

Where next

Go to Promote your pipeline to production next — the capstone of the series.

Once you’re comfortable with the mechanics:

• Cost accounting — the full three-tier pricing reference, mixed-tier steps, count_pricing in full, and everything explicitly out of scope (FX conversion, price history, run-blocking team quotas).
• Team attribution — where cost rolls up by team, and the openclaw executor’s token-reporting gap mentioned in Using OpenClaw.
• Observability — the cost metrics used in this tutorial’s Prometheus counter, alongside the rest of VectorStep’s metrics and OpenTelemetry tracing.

Then the capstone — promoting your pipeline to production.