Browse documentation

Respond

Manage incidents and on-call

Turn an active customer-impacting problem into an owned, traceable response.

Use an incident when a problem needs coordination, explicit ownership, or a durable timeline. A single Issue can often be handled directly without declaring an incident.

Prepare on-call

Before relying on escalation:

  1. add personal contact methods in Settings → Notifications;
  2. create an on-call schedule;
  3. add rotations and any temporary overrides;
  4. create an escalation policy; and
  5. send a test through the complete route.

Keep schedule ownership current. An escalation policy cannot compensate for stale contact methods.

Build an on-call schedule

Open Respond → On-Call and create a schedule with the timezone where its handoffs should be interpreted. Add members to a layer in rotation order, choose the rotation length and start time, and use active-window restrictions when a layer covers only particular days or hours.

Layers stack: a later active layer overrides an earlier one. Review the preview, then check the live on call: badge because the preview does not apply restrictions or temporary overrides.

Use an override for a dated coverage change such as leave or a shift swap. Set both start and end; the underlying rotation resumes when the override expires.

Create escalation policies

Open Respond → Escalation Policies and create ordered levels. Each level waits its configured delay, then notifies one or more channels, members, or on-call schedules. Every level needs at least one target.

Use later levels to broaden the audience when an alert remains unacknowledged. Repeating escalation requires a repeat interval and a maximum escalation count. Attach the policy to an alert rule and test the complete route. If a policy is deleted, rules using it fall back to their own notification channels.

Declare an incident

When an Issue triggers the response, open that Issue and choose Declare incident. Teml carries the Issue link into the new case file. For a manual incident, open Respond → Incidents, choose Declare incident, and enter a short customer-facing symptom as the title.

The dashboard’s manual declaration starts with the title and default severity. Use the CLI when you need to set the description, severity, and primary signal in one operation:

teml incidents create "Checkout unavailable" \
  --description "Customers cannot complete payment" \
  --severity 1 \
  --link-type error_group \
  --link-ref <issue-id>

Supported incident links are alert, error_group (Issue), and anomaly. Add another supported signal with teml incidents link <incident-id> --type <type> --ref <signal-id>. Use the linked Issue to reach affected customers, traces, replays, releases, and external tickets.

Work from the case file

Record material changes as timeline comments:

  • what customers are experiencing;
  • current impact and affected scope;
  • evidence supporting the working cause;
  • mitigation and rollback actions; and
  • decisions that change the response.

Use Customer Detective to gather cited evidence, then link the relevant underlying objects rather than pasting an unsupported summary.

Close deliberately

Resolve the incident when customer impact has ended and monitoring confirms recovery. Capture any remaining follow-up work in the linked ticket system. Reopen the incident if the same impact returns before the response is complete.

Verify the response path

Schedule a safe test window, confirm the expected member is on call, trigger a test alert, and verify the correct escalation levels stop after acknowledgement. Test again after changing rotations, provider connections, or contact methods.