Skip to content
VectorStep
Blog

Tutorial: route escalations to a real channel

Every escalation so far has only meant a status change visible if you happen to check the UI. This tutorial wires a real notifications: block so an escalation actually goes somewhere — first the zero-config log channel, then a real Telegram bot.

Prerequisites

Section titled “Prerequisites”

Store a full investigation as an artifact completed — ~/.vectorstep/service/pipelines/alert-triage.yaml is identify-upstreams → check-upstreams (fan-out) → consolidate → write-up, and check-upstreams already escalates below its confidence_threshold: 0.70.

1. Add a notifications block

Section titled “1. Add a notifications block”

notifications: is a pipeline-level key, a sibling of name:/trigger:/ context_template: — not nested under steps:. Add it to ~/.vectorstep/service/pipelines/alert-triage.yaml:

notifications:
  escalate:
    - channel: log
      template: |
        ESCALATED: {{pipeline_name}} — {{step_summary}}
        Confidence {{confidence}} fell below {{confidence_threshold}} ({{escalation_reason}})
      config:
        level: error

escalate matches the on_low_confidence: escalate transition already configured on check-upstreams — this fires any time any step or group in the pipeline escalates, not just that one. {{pipeline_name}}, {{step_summary}}, {{confidence}}, {{confidence_threshold}}, and {{escalation_reason}} are populated automatically the moment a step escalates or aborts; unlike {{severity}}/{{summary}} earlier in this series, they need no context_template.include entry.

log needs no config.yaml entry — it’s always registered — so a plain /reload picks this up:

Terminal window
curl -X POST http://localhost:8000/reload

2. Force the escalation

Section titled “2. Force the escalation”

Temporarily raise check-upstreams’s confidence_threshold to something no real check is likely to clear:

  - fan_out:
      name: check-upstreams
      over: "{{ steps.identify_upstreams.upstreams }}"
      as: upstream
      executor: gateway
      executor_config:
        agent: upstream-checker
        session_key: "agent:upstream-checker:{{pipeline_run_id}}:check-{{upstream.name}}"
      join: all_must_pass
      confidence_threshold: 0.99   # deliberately unreachable — see below
      on_low_confidence: escalate
      on_abort: notify
      max_items: 10
      on_empty: skip
      timeout_seconds: 90
      prompt_template: |
        ...                        # unchanged from the fan-out tutorial
      verifier:
        executor: gateway
        executor_config:
          agent: upstream-checker
        combination_strategy: minimum
        trigger:
          always: true

Reload and re-trigger:

Terminal window
curl -X POST http://localhost:8000/reload
curl -X POST "http://localhost:8000/webhook?source=alertmanager&allow_testing=true" \
  -H "Content-Type: application/json" \
  -d @tests/fixtures/alertmanager_critical.json

What you should see

Section titled “What you should see”

Both branches should complete normally — upstream-checker genuinely finds GitHub and npm healthy — but 0.99 is higher than either branch, or their verifier-combined result, is realistically going to report. check-upstreams itself should come back escalated, and the run should stop there: consolidate and write-up never run — the pipeline halts on escalation, it doesn’t skip forward past it.

Check the service’s own logs (stdout, or wherever your log aggregation points) for a line like:

2026-01-01T12:00:03 ERROR vectorstep.notifications ESCALATED: alert-triage — Parallel group 'check-upstreams': 2/2 branches completed
Confidence 0.85 fell below 0.99 (low_confidence)

On the run detail page’s Run log, look for a notification_suppressed_testing event, not notification_sent — [testing] Notification routed to log: escalate → would have been log. That “would have been log” phrasing looks a little redundant when the configured channel already is log, but it’s the same message for every channel: this pipeline has been stage: testing (the default) this whole series, and testing forces every notification through log regardless of what’s configured — see what testing mutes. The log line above still fires for real either way; testing only affects routing, not whether the notifier actually runs.

Leave check-upstreams’s confidence_threshold at 0.99 for now — step 3 reuses this same forced escalation to test Telegram, so reverting it here would just mean bumping it back up again in a moment. It gets reverted once, at the end of step 3.

3. Add a real Telegram bot

Section titled “3. Add a real Telegram bot”

  1. In Telegram, message @BotFather and send /newbot. Follow the prompts (a display name, then a unique username ending in bot). BotFather replies with a token that looks like 123456789:ABCdefGhIJKlmNoPQRsTUVwxyZ — that’s your bot_token.

  2. Search for your new bot’s username and send it any message (e.g. “hi”) — Telegram only allows a bot to message you after you’ve messaged it first.

  3. Fetch your chat_id:

    Terminal window
    curl "https://api.telegram.org/bot<your-bot-token>/getUpdates"

    Look for "chat":{"id": ...} in the response — that number is your chat_id.

Export both as environment variables in the shell that will run the service, then add the credentials to ~/.vectorstep/service/config.yaml:

Terminal window
export TELEGRAM_BOT_TOKEN="123456789:ABCdefGhIJKlmNoPQRsTUVwxyZ"
export TELEGRAM_CHAT_ID="987654321"
notifications:
  telegram:
    bot_token: ${TELEGRAM_BOT_TOKEN}
    chat_id: ${TELEGRAM_CHAT_ID}

This needs a real restart, not POST /reload — same gotcha as artifact storage in the previous tutorial. Telegram’s notifier is built once from config.yaml when the service process starts; /reload re-reads pipeline YAML, not this block. Stop the service (Ctrl-C) and start it again from the same shell you exported the variables in:

Terminal window
cd ~/.vectorstep/service
source .venv/bin/activate
uvicorn src.main:app --reload --port 8000

Check the startup logs for Telegram notifier configured — if it’s missing, bot_token/chat_id didn’t resolve (check the env vars are set in this shell, and that ${TELEGRAM_BOT_TOKEN} matches the export exactly).

Now add telegram alongside log in the pipeline’s notifications: block — a single action can fan out to a list of channels:

notifications:
  escalate:
    - channel: log
      template: |
        ESCALATED: {{pipeline_name}} — {{step_summary}}
        Confidence {{confidence}} fell below {{confidence_threshold}} ({{escalation_reason}})
      config:
        level: error
    - channel: telegram
      template: "🚨 <b>{{pipeline_name}}</b> escalated: {{step_summary}} (confidence {{confidence}})"

Telegram messages render with HTML parse mode, so <b>/<code>/<a href> tags work in the template.

Reload (pipeline YAML changes — /reload is fine here, no restart needed) and re-trigger — check-upstreams’s confidence_threshold is still 0.99 from step 2, so this forces the same escalation again:

Terminal window
curl -X POST http://localhost:8000/reload
curl -X POST "http://localhost:8000/webhook?source=alertmanager&allow_testing=true" \
  -H "Content-Type: application/json" \
  -d @tests/fixtures/alertmanager_critical.json

What you should see

Section titled “What you should see”

Nothing arrives in Telegram — and that’s correct, not broken. The pipeline is still stage: testing, so the telegram entry is forced to log too, exactly like step 2: the run log shows a second notification_suppressed_testing event, this time ... → would have been telegram, with your 🚨 template rendered in the message but sent through the log notifier instead of the real bot API.

The wiring is genuinely correct at this point — nothing further to change here. It starts actually reaching Telegram the moment this pipeline is promoted to stage: production, which the capstone tutorial covers deliberately, readiness evidence and all, rather than just flipping the flag early to prove a point here.

Revert check-upstreams’s confidence_threshold to 0.70 once you’re done — 0.99 only ever existed to manufacture the escalation for this tutorial.

A brief word on executor: notify

Section titled “A brief word on executor: notify”

Everything above reacts to a state transition — escalate, abort, stop. For sending something as a first-class step in the pipeline’s own sequence (a structured Slack/PagerDuty/Teams payload, independent of whether anything escalated), reach for executor: notify instead — it takes a payload: dict and renders every string value as a Jinja2 template before POSTing structured JSON, rather than reacting to a gate decision the way notifications: does.

Where next

Section titled “Where next”

Go to Gate a pipeline on budget next — the next tutorial in the series.

Once you’re comfortable with the mechanics:

  • Notifications — the full channel reference, including slack and webhook, the two this tutorial didn’t cover.
  • Triggering pipelines from Telegram — the bot you just set up also understands /run <pipeline-name> from the same chat, no new credentials needed.
  • Human-in-the-loop — a different mechanism for when escalation should mean “wait for someone to actually decide,” not just “tell someone.”
  • Testing vs production stages — the full reference for what else stage: testing mutes besides notifications.