Tutorial: fan out over multiple services

The previous tutorials had first-responder check exactly one upstream dependency — GitHub’s status API — hardcoded into its prompt. Real services usually depend on several upstreams at once, and you don’t always know how many until the alert tells you. This tutorial turns that one hardcoded check into a dynamic fan_out: a step returns a list of upstreams to check, the runner spawns one branch per item concurrently, and their results join back into a single effective confidence before continuing.

Models used for testing

This series was validated internally against openrouter/anthropic/claude-sonnet-4.5 for first-responder and openrouter/anthropic/claude-haiku-4.5 for every other agent, including upstream-checker below. Other models and providers should work too, but exact wording and confidence numbers can vary by model.

Prerequisites

Turn on grounding completed. The triage step this tutorial replaces should have a confidence_threshold and a verifier from Turn on the trust knobs.

Grounding doesn’t carry into the fan-out

Grounding isn’t wired into fan_out/parallel branches — sequential steps only. If the previous tutorial left grounding: configured, it simply won’t run on any branch inside the fan-out below — no error, just not there. identify-upstreams and consolidate, both regular sequential steps, could carry a grounding: block if you wanted one; this tutorial leaves both without one to keep the fan-out mechanics the focus.

1. Write a more general upstream-checking agent

first-responder’s soul.md is deliberately narrow — it’s identity says its job is checking GitHub specifically and reading known-issues.md. Reusing it to check an arbitrary, runtime-determined upstream would create exactly the same soul.md-vs-prompt conflict Turn on grounding ran into: an identity that says “you check GitHub” fighting a prompt that says “actually check npm this time.” Rather than fight that again, write a second agent whose identity is already general enough for this job — the real-world design the previous tutorial’s closing note pointed at, not just a hypothetical.

mkdir -p ~/.vectorstep-gateway/agents/upstream-checker

~/.vectorstep-gateway/agents/upstream-checker/agent.yaml:

name: upstream-checker
model: anthropic/claude-haiku-4-5-20251001
max_tokens: 4096
tools:
  - fetch

~/.vectorstep-gateway/agents/upstream-checker/soul.md:

# Upstream Checker

You check whether a single named upstream dependency is currently having an
incident, given its public status-page URL. That is your entire job. Which
upstream that is changes every call — nothing about a specific upstream's
name or URL is part of who you are, only what you're told this run.

## What you do

1. Use the `fetch` tool on the exact URL you're given.
2. Determine whether that upstream is reporting an active incident.

## Confidence

Confidence measures how completely you checked the upstream — did the tool
call succeed and give you a clear status? A tool failure, timeout, or
unclear response should be reported honestly with low confidence, not
papered over with a guess.

## Output format

Respond with ONLY the JSON object your prompt asks for. No preamble, no
markdown fences, no commentary outside the JSON.

Reload the Gateway to pick it up:

curl -X POST http://localhost:18780/reload

2. Add a step that lists the upstreams to check

In ~/.vectorstep/service/pipelines/alert-triage.yaml, add a step ahead of triage that returns a fixed list for now — a real pipeline would likely infer this from the alert, but a hardcoded list keeps the focus on the fan-out mechanics rather than list-generation. See Designing dynamic fan-out for the real, inferring version of this step:

  - name: identify-upstreams
    executor: gateway
    executor_config:
      agent: generic-pipeline-step
      session_key: "agent:generic-pipeline-step:{{pipeline_run_id}}:identify-upstreams"
    confidence_threshold: 0.70
    on_low_confidence: escalate
    prompt_template: |
      A {{severity}} alert fired for {{labels.service}} in {{labels.environment}}.

      For this exercise, the upstream dependencies to check are fixed. Return
      ONLY this JSON, no other text:
      {
        "confidence": 1.0,
        "summary": "Checking 2 upstream dependencies: github, npm",
        "next_step_context": "Fan out to check each upstream's status page",
        "upstreams": [
          {"name": "github", "url": "https://www.githubstatus.com/api/v2/summary.json"},
          {"name": "npm", "url": "https://status.npmjs.org/api/v2/summary.json"}
        ],
        "reasoning": {
          "supports": "Fixed list for this tutorial",
          "contradicts": "",
          "assumptions": "A real pipeline would infer this dynamically"
        }
      }

This reuses generic-pipeline-step from the quick start — no tools needed for a step that’s just returning fixed data. upstreams is a list of objects, not bare strings; over can resolve to either, and passing a {name, url} pair per branch means the fan-out prompt below doesn’t need to invent a URL-naming scheme to go from “github” to a real endpoint.

3. Replace triage with a fan-out

In the same ~/.vectorstep/service/pipelines/alert-triage.yaml, replace the single triage step (confidence threshold, verifier, and whatever grounding config it had) with:

  - fan_out:
      name: check-upstreams
      over: "{{ steps.identify_upstreams.upstreams }}"
      as: upstream
      executor: gateway
      executor_config:
        agent: upstream-checker
        session_key: "agent:upstream-checker:{{pipeline_run_id}}:check-{{upstream.name}}"
      join: all_must_pass
      confidence_threshold: 0.70
      on_low_confidence: escalate
      on_abort: notify
      max_items: 10
      on_empty: skip
      timeout_seconds: 90
      prompt_template: |
        Check "{{upstream.name}}" (branch {{fan_out_index + 1}} of
        {{fan_out_total}}) for a {{severity}} alert on {{labels.service}} in
        {{labels.environment}}.

        Use the fetch tool on this exact URL: {{upstream.url}}

        Return ONLY this JSON, no other text:
        {
          "confidence": 0.0,
          "summary": "One sentence: {{upstream.name}}'s status and whether it's implicated",
          "next_step_context": "",
          "upstream_incident": true,
          "reasoning": {
            "supports": "Evidence from the status check",
            "contradicts": "Evidence against an upstream cause",
            "assumptions": "What you're assuming in the absence of data"
          }
        }
      verifier:
        executor: gateway
        executor_config:
          agent: upstream-checker
        combination_strategy: minimum
        trigger:
          always: true

join: all_must_pass (effective = min(all branch confidences)) is the natural choice here: don’t proceed confidently if any upstream check came back weak, since a single unchecked dependency could be the actual cause. any_must_pass (effective = max(...)) would be the right call instead if finding one strong signal were enough to proceed — useful when branches are alternative explanations rather than independent things that all need to check out.

{{ upstream.name }} and {{ upstream.url }} come from the {name, url} object each branch receives via as: upstream; {{ fan_out_index }} and {{ fan_out_total }} are injected automatically. The verifier: block works exactly as it did on the sequential triage step — it’s applied per branch here, not once for the whole group.

4. Add a step that consolidates the branches

Fan-out branch outputs are registered as "{fan_out_name}/{index}" — check-upstreams/0, check-upstreams/1 — referenced with bracket notation since dot-notation breaks on the /. With a fixed two-item list, referencing both by index directly is simplest:

  - name: consolidate
    executor: gateway
    executor_config:
      agent: generic-pipeline-step
      session_key: "agent:generic-pipeline-step:{{pipeline_run_id}}:consolidate"
    confidence_threshold: 0.70
    on_low_confidence: escalate
    prompt_template: |
      A {{severity}} alert fired for {{labels.service}} in {{labels.environment}}.
      Two upstream dependencies were checked in parallel:

      - {{steps['check-upstreams/0'].summary}}
      - {{steps['check-upstreams/1'].summary}}

      Summarise the overall picture across both upstreams.

      Return ONLY this JSON, no other text:
      {
        "confidence": 0.0,
        "summary": "One sentence: overall picture across both upstreams",
        "next_step_context": "",
        "reasoning": {
          "supports": "Evidence that makes this assessment credible",
          "contradicts": "Anything that complicates the picture",
          "assumptions": "What you're assuming in the absence of data"
        }
      }

For a genuinely dynamic list (not this tutorial’s fixed two), a real pipeline can’t reference branches by index like this since the count isn’t known at authoring time — that’s what identify-upstreams’s own next_step_context is for, carrying forward a brief the consolidating agent can use instead of enumerating branches it doesn’t know exist yet.

The full pipeline is now three steps: identify-upstreams → check-upstreams (fan-out) → consolidate. Reload and re-trigger:

curl -X POST http://localhost:8000/reload
curl -X POST "http://localhost:8000/webhook?source=alertmanager&allow_testing=true" \
  -H "Content-Type: application/json" \
  -d @tests/fixtures/alertmanager_critical.json

What you should see

Open the run in http://localhost:8000/ui/. Instead of one triage step, you should see three: identify-upstreams, then two rows under check-upstreams (check-upstreams/0 for github, check-upstreams/1 for npm) each with their own full Trust panel — self-report, verifier, prompt, agent trace — exactly like any sequential step gets, and then consolidate. The group’s effective confidence should be the minimum of the two branch confidences (that’s all_must_pass), and the run overall should behave the same way any threshold-gated step does: completed if both branches were confident, escalated if either one wasn’t.

5. What the guardrails are for

max_items: 10 and on_empty: skip aren’t decoration. identify-upstreams is a step like any other — it’s an LLM call, and LLM calls occasionally return something you didn’t ask for. Without max_items, a step that returned 200 “upstreams” instead of 2 would spawn 200 concurrent branches, each a real LLM call; the cap turns that into a clean step failure instead of a cost and rate-limit surprise. Without on_empty: skip, a step that returned [] — nothing to check — would hit on_empty’s default (complete, meaning effective_confidence = 1.0) and silently claim full confidence over zero actual checks; skip treats an empty list as “nothing to do here” instead of “everything’s fine,” which is the more honest reading of finding no upstreams to check in the first place.

Where next

Go to Store a full investigation as an artifact next — the next tutorial in the series.

Once you’re comfortable with the mechanics:

• Designing dynamic fan-out — this tutorial’s identify-upstreams is deliberately hardcoded to keep the focus on fan-out mechanics; this guide covers inferring a real, runtime-determined list instead, the way an actual pipeline would.
• Parallel groups & fan-out — the full reference, including static parallel: groups for a fixed branch set (this tutorial only covers the dynamic case) and weighted_average as a third join strategy.
• samples/pipelines/fan-out-multi-service-triage.yaml — a complete worked example with a larger, genuinely dynamic branch count and a next_step_context-based consolidation step, rather than this tutorial’s fixed two-branch, reference-by-index shortcut.