Set up out-of-hours call routing for serious security alerts

TL;DR — To get serious alerts routed correctly out of hours, you need three things in your alerting provider: a high-severity rule, an on-call schedule, and a notification policy that calls the primary person first and escalates to a backup if they do not acknowledge (confirm they received it). The most common fix is linking the schedule to the high-severity policy but forgetting to add a phone call action, so the alert only sends email or chat. Reading time: ~5 min

Goal

When you finish, a serious out-of-hours alert will place a phone call to the person on duty, wait for acknowledgement, then call the backup contact if the first person does not respond.

Prerequisites

• Access to your alerting provider's dashboard with permission to edit notification rules, schedules, and contacts (often called Admin, Owner, or Manager)
• The mobile phone numbers for the primary and backup on-call people, in full international format, for example +44 7700 900123
• Your definition of a serious alert, for example Severity = Critical or Priority = P1
• A test alert method in your provider, such as Send test alert, Create test incident, or a webhook (an HTTP callback)
• If your provider supports acknowledgement by keypad, the on-call people should know which key confirms the call

Steps

Step 1: Add the people who can receive out-of-hours calls

In your provider's dashboard, open the contacts or users area and add both phone numbers.

Use this menu path in your provider's dashboard:

Dashboard → Users or Contacts → Add user/contact

Enter these literal values for each person:

What you should see: both contacts appear as active, and the phone number status shows verified or ready to use.

Step 2: Create the out-of-hours schedule

Create a schedule (the rota for who is on duty) that covers nights, weekends, and holidays if your provider supports holiday calendars.

Use this menu path in your provider's dashboard:

Dashboard → On-call schedules or Schedules → New schedule

Enter these literal values:

If your provider does not support a secondary assignee inside the schedule, leave only Primary On-Call here. You will add the backup in the escalation policy in Step 3.

What you should see: a calendar or timeline showing covered hours outside business hours.

Step 3: Create the escalation policy for serious alerts

Create a policy (the rule that decides who gets contacted and in what order) for critical alerts.

Use this menu path in your provider's dashboard:

Dashboard → Alerting or Notifications → Escalation policies → New policy

Enter these literal values:

If your provider uses drag-and-drop rules, add them in exactly this order: Critical filter → Out-of-hours schedule → Call primary → Wait 15 minutes for acknowledgement → Call backup.

What you should see: the policy summary shows voice call as the first action and backup escalation after 15 minutes.

Step 4: Link the policy to your serious alert source

Now connect the policy to the alerts coming from your security tooling, such as your XDR (extended detection and response), SIEM (security event monitoring), or monitoring system.

Use this menu path in your provider's dashboard:

Dashboard → Alert sources or Integrations → <your source> → Notification rules

Set these literal values:

If your provider uses labels instead of severity, use the exact label your alerts already send, for example:

{
  "severity": "critical"
}

What you should see: the alert source shows Critical out-of-hours call policy as the destination for critical alerts.

Step 5: Add a business-hours rule so daytime alerts do not call people unnecessarily

Create a separate rule for business hours if you want critical alerts to go to chat, email, or a service desk during the day.

Use this menu path in your provider's dashboard:

Dashboard → Alerting or Notifications → Rules → New rule

Enter these literal values:

What you should see: two non-overlapping rules, one for business hours and one for out of hours.

Step 6: Run a live test call

Send a test alert that matches your critical rule.

Use this menu path in your provider's dashboard:

Dashboard → Alert sources or Integrations → <your source> → Send test alert

Use these literal test values:

{
  "title": "TEST - Critical out-of-hours call routing",
  "severity": "critical",
  "source": "manual-test",
  "body": "This is a test of the out-of-hours critical call policy."
}

If your provider has no test button but supports webhook tests, send this from any machine with curl installed:

curl -X POST "https://YOUR-ALERT-ENDPOINT" \
  -H "Content-Type: application/json" \
  -d '{"title":"TEST - Critical out-of-hours call routing","severity":"critical","source":"manual-test","body":"This is a test of the out-of-hours critical call policy."}'

What you should see: the primary on-call phone rings within 1-2 minutes, and the alert timeline shows voice call sent.

[Critical Alert] ---- 1. matches severity rule ----> [Policy]
[Policy] ---- 2. checks out-of-hours schedule ----> [Schedule]
[Schedule] ---- 3. call primary ----> [Primary On-Call]
[Primary On-Call]
     |
     | 4. no acknowledgement in 15 min
     v
[Backup On-Call] <---- 5. escalate call ---- [Policy]

Verify it works

Use both of these checks:

1. In the dashboard, open the test alert timeline:

Dashboard → Alerts or Incidents → TEST - Critical out-of-hours call routing

Expected result: you see matched policy: Critical out-of-hours call policy, then voice call sent to Primary On-Call.

1. Confirm escalation works by not acknowledging the first test, then watching for the backup call after 15 minutes.

Expected result: the backup phone rings after the acknowledgement timeout, and the timeline shows escalated to Backup On-Call.

If you want a final proof for your records, take a screenshot of the alert timeline showing the policy match and escalation events.

Common pitfalls

The policy sends email or chat, but not a phone call

Mistake: the notification policy was linked to the schedule, but the action is email or message instead of voice call.

Symptom: the alert appears in the dashboard, but nobody gets a call.

Fix: edit the policy and set Step 1 action = Voice call Primary On-Call immediately.

The phone number is not in international format

Mistake: entering a local number like 07700 900123 instead of +44 7700 900123.

Symptom: the contact saves, but calls fail or never start.

Fix: edit the contact and replace the number with the full +countrycode format.

The critical alert does not match the rule

Mistake: the rule expects Critical but the source sends critical, P1, or a different label.

Symptom: test alerts arrive, but the out-of-hours policy never triggers.

Fix: open one real or test alert, copy the exact severity value shown in the payload, and use that exact value in the match condition.

The schedule uses the wrong time zone

Mistake: the schedule was created in UTC instead of your local time zone.

Symptom: calls happen an hour early or late, especially around daylight saving changes.

Fix: edit the schedule and set Time zone to your team's local time zone, then save and retest.

The primary person answered, but the system still escalated

Mistake: answering the call does not count as acknowledgement in some providers; the person must press a key or confirm in the app.

Symptom: the backup gets called even though the primary picked up.

Fix: tell on-call staff the exact acknowledgement action your provider uses and run the test again with that action.

The business-hours and out-of-hours rules overlap

Mistake: both rules are active at the same time, often at exactly 08:00 or 18:00.

Symptom: duplicate notifications or inconsistent routing at handover time.

Fix: set business hours to 08:00-17:59 and out of hours to 18:00-07:59, then rerun the test at the boundary time.

This article was written by an AI system and published pending human review. Verify anything you intend to act on.