Alerting
What It Is
Alerting means notifying the right people when system behavior moves away from expected conditions.
It connects monitoring data with human action so problems can be handled before they become bigger incidents.
Alerting = Detect abnormal behavior -> Notify team -> Take action
Why It Matters
Monitoring only shows what is happening.
Alerting makes sure important problems are not missed.
It helps teams respond when there is:
- performance degradation
- resource issue
- user action anomaly
- latency spike
- error rate increase
- traffic anomaly
- crash-free rate drop
- users experiencing errors
Monitoring watches the system.
Alerting wakes up the team when something needs attention.
End-to-End Flow
The basic alerting flow is:
Events / Metrics -> Thresholds -> Breach -> Alert -> Incident
This means the system collects signals, compares them with rules, triggers alerts when conditions are breached, and then turns serious alerts into incidents.
Monitoring Flow
Alerting starts with events and metrics.
The system continuously watches collected data and checks whether it crosses the configured threshold.
1. Events and metrics are collected
2. Thresholds are configured
3. System checks continuously
4. Threshold is exceeded
5. Alert is triggered
Key components:
| Component | Purpose |
|---|---|
| Events / Metrics | Raw signals from the app or system |
| Thresholds | Rules that define expected limits |
| Threshold Breach | Condition goes above or below the limit |
| Alert | Notification is sent to the team |
| Incident | Alert becomes a trackable issue |
Events and Metrics
Events and metrics are the input for alerting.
They can come from different parts of the system.
Examples:
- application errors
- API latency
- server response status
- resource usage
- user actions
- performance metrics
- logs
No metrics -> no alerting decision
Wrong metrics -> noisy or useless alerts
Good alerting starts with collecting the right signals.
Thresholds
A threshold is a condition that decides when an alert should fire.
Examples:
Users experiencing errors > X
Crash-free rate drops
API latency goes above limit
Error rate increases
A threshold can be based on:
- performance
- resources
- user actions
- errors
- availability
Threshold = Expected boundary for healthy behavior
Alert Rule Configuration
Alert rule configuration defines when an alert should fire.
In a tool like Sentry, this means selecting a metric and defining a condition.
Examples:
- users experiencing errors crosses a limit
- crash-free rate drops below expected value
- performance degradation crosses a threshold
What happens during configuration:
Select metric
Define condition
Set threshold
Configure severity
Choose notification target
Severity Levels
Alerts usually need severity levels so teams can understand how urgent the issue is.
Common severity states:
| Severity | Meaning |
|---|---|
| Critical | Needs urgent attention |
| Warning | Needs attention but may not be urgent |
| Resolved | Issue condition is no longer active |
Severity helps avoid treating every alert with the same priority.
Not every alert is an emergency.
Severity tells how serious the issue is.
Alert Detection Engine
An alert detection engine continuously evaluates metrics against configured rules.
Grafana is an example of a tool that can evaluate time-series data and change alert states.
What it does:
- runs queries on time-series data
- checks thresholds
- evaluates rules continuously
- changes alert states
- triggers alert notifications
Detection Engine = Continuously checks whether metrics are healthy
Alert States
Alert systems usually move through different states.
Common states:
| State | Meaning |
|---|---|
| OK | Metric is healthy |
| Pending | Condition may be becoming unhealthy |
| Firing | Alert condition is active |
This helps teams understand whether the problem is stable, starting, or already active.
Alert Examples
Common alert examples:
Latency spike
Error rate increase
Traffic anomaly
Crash-free rate drop
Users experiencing errors above threshold
These alerts are useful because they point to conditions that may affect users or system reliability.
Notification Channels
Once an alert fires, it must reach the right people.
Common notification channels:
- Slack
- SMS
- PagerDuty
- Squadcast
- Zenduty
- On-call notifications
Alert without notification = problem may still be missed
The channel should match the severity.
A critical alert may need on-call escalation, while a warning may only need Slack or email.
Incident Management
When an alert fires and needs action, it becomes an incident.
Incident management helps track who owns the issue and whether it is resolved.
Alert fired -> Incident created -> Team responds
Zenduty is an example of an incident management tool.
Incident Lifecycle
A basic incident lifecycle looks like this:
Triggered -> Acknowledged -> Resolved
| Stage | Meaning |
|---|---|
| Triggered | Alert created an incident |
| Acknowledged | Someone takes ownership |
| Resolved | Issue is fixed |
This makes response clear and trackable.
Incident Features
Incident tools help teams manage real production issues.
Common features:
- assign to teams
- set priority
- track SLA
- manage incident ownership
- follow alert status
- coordinate response
Priority examples:
P0
P1
P2
Higher priority usually means faster response is required.
Complete Architecture
A full alerting architecture connects application signals to team response.
[App / System]
↓
[Metrics & Logs]
↓
[Monitoring: Grafana / Sentry]
↓
[Alert Rules Engine]
↓
[Alert Triggered]
↓
[Incident Tools: Zenduty]
↓
[Team Responds]
This turns raw system behavior into action.
Alerting Tools
| Tool | Role |
|---|---|
| Sentry | Alert rules for errors, crash-free rate, and performance degradation |
| Grafana | Metric evaluation and alert detection |
| AWS CloudWatch | Cloud metric monitoring and alarms |
| AWS SNS | Notification delivery |
| Slack / Email / SMS | Alert notification channels |
| PagerDuty / Squadcast / Zenduty | Incident management and on-call response |
Alerting Table
| Area | Explanation |
|---|---|
| Main goal | Notify the right people when conditions become unhealthy |
| Input | Events, metrics, logs |
| Rule | Threshold condition |
| Breach | Metric crosses expected limit |
| Alert | Notification is triggered |
| Incident | Trackable issue created for response |
| States | OK, Pending, Firing |
| Severity | Critical, Warning, Resolved |
| Response | Acknowledge, investigate, resolve |
Basic Checklist
Collect useful events and metrics
Define thresholds for important signals
Track performance, resources, and user actions
Create alert rules for serious failures
Set severity levels like Critical and Warning
Avoid alerting on every small noise
Use Grafana or Sentry to evaluate rules
Send alerts to the right channel
Escalate critical alerts to on-call tools
Convert important alerts into incidents
Assign incident ownership
Track priority and SLA
Resolve the incident and confirm recovery
Interview Style Answer
Alerting ensures that when system behavior deviates from expected conditions, the right people are notified and action is taken. The basic flow is events or metrics, thresholds, breach, alert, and incident. First, the system collects metrics and logs. Then alert rules define conditions such as users experiencing errors above a limit, crash-free rate dropping, latency spikes, error rate increases, or performance degradation. Tools like Sentry help configure alert rules, while Grafana continuously evaluates time-series data and changes alert states such as OK, Pending, and Firing. Once an alert fires, it can be sent through channels like Slack, email, SMS, PagerDuty, Squadcast, or Zenduty. If action is required, the alert becomes an incident that is triggered, acknowledged, assigned, tracked, and resolved.
One-Line Summary
Alerting = Convert abnormal metrics into notifications and incidents so teams can respond quickly.
Final Mental Model
Metrics show signals
Thresholds define healthy limits
Breaches trigger alerts
Alerts notify people
Incidents organize response
Monitor first, alert only when action is needed.