Alerting

Alerting means notifying the right people when system behavior moves away from expected conditions.

It connects monitoring data with human action so problems can be handled before they become bigger incidents.

Alerting = Detect abnormal behavior -> Notify team -> Take action

One-line idea: Alerting = Convert abnormal metrics into notifications and incidents so teams can respond quickly.

Core Concepts

Why It Matters

Monitoring only shows what is happening.

Alerting makes sure important problems are not missed.

It helps teams respond when there is:

• performance degradation
• resource issue
• user action anomaly
• latency spike
• error rate increase
• traffic anomaly
• crash-free rate drop
• users experiencing errors

Monitoring watches the system.
Alerting wakes up the team when something needs attention.

End-to-End Flow

The basic alerting flow is:

Events / Metrics -> Thresholds -> Breach -> Alert -> Incident

This means the system collects signals, compares them with rules, triggers alerts when conditions are breached, and then turns serious alerts into incidents.

Monitoring Flow

Alerting starts with events and metrics.

The system continuously watches collected data and checks whether it crosses the configured threshold.

1. Events and metrics are collected
2. Thresholds are configured
3. System checks continuously
4. Threshold is exceeded
5. Alert is triggered

Key components:

Component	Purpose
Events / Metrics	Raw signals from the app or system
Thresholds	Rules that define expected limits
Threshold Breach	Condition goes above or below the limit
Alert	Notification is sent to the team
Incident	Alert becomes a trackable issue

Practical Examples

Events and Metrics

Events and metrics are the input for alerting.

They can come from different parts of the system.

Examples:

• application errors
• API latency
• server response status
• resource usage
• user actions
• performance metrics
• logs

No metrics -> no alerting decision
Wrong metrics -> noisy or useless alerts

Good alerting starts with collecting the right signals.

Thresholds

A threshold is a condition that decides when an alert should fire.

Examples:

Users experiencing errors > X
Crash-free rate drops
API latency goes above limit
Error rate increases

A threshold can be based on:

• performance
• resources
• user actions
• errors
• availability

Threshold = Expected boundary for healthy behavior

Alert Rule Configuration

Alert rule configuration defines when an alert should fire.

In a tool like Sentry, this means selecting a metric and defining a condition.

Examples:

• users experiencing errors crosses a limit
• crash-free rate drops below expected value
• performance degradation crosses a threshold

What happens during configuration:

Select metric
Define condition
Set threshold
Configure severity
Choose notification target

Severity Levels

Alerts usually need severity levels so teams can understand how urgent the issue is.

Common severity states:

Severity	Meaning
Critical	Needs urgent attention
Warning	Needs attention but may not be urgent
Resolved	Issue condition is no longer active

Severity helps avoid treating every alert with the same priority.

Not every alert is an emergency.
Severity tells how serious the issue is.

Alert Detection Engine

An alert detection engine continuously evaluates metrics against configured rules.

Grafana is an example of a tool that can evaluate time-series data and change alert states.

What it does:

• runs queries on time-series data
• checks thresholds
• evaluates rules continuously
• changes alert states
• triggers alert notifications

Detection Engine = Continuously checks whether metrics are healthy

Alert States

Alert systems usually move through different states.

Common states:

State	Meaning
OK	Metric is healthy
Pending	Condition may be becoming unhealthy
Firing	Alert condition is active

This helps teams understand whether the problem is stable, starting, or already active.

Alert Examples

Common alert examples:

Latency spike
Error rate increase
Traffic anomaly
Crash-free rate drop
Users experiencing errors above threshold

These alerts are useful because they point to conditions that may affect users or system reliability.

Notification Channels

Once an alert fires, it must reach the right people.

Common notification channels:

• Mail
• Slack
• SMS
• PagerDuty
• Squadcast
• Zenduty
• On-call notifications

Alert without notification = problem may still be missed

The channel should match the severity.

A critical alert may need on-call escalation, while a warning may only need Slack or email.

Incident Management

When an alert fires and needs action, it becomes an incident.

Incident management helps track who owns the issue and whether it is resolved.

Alert fired -> Incident created -> Team responds

Zenduty is an example of an incident management tool.

Incident Lifecycle

A basic incident lifecycle looks like this:

Triggered -> Acknowledged -> Resolved

Stage	Meaning
Triggered	Alert created an incident
Acknowledged	Someone takes ownership
Resolved	Issue is fixed

This makes response clear and trackable.

Incident Features

Incident tools help teams manage real production issues.

Common features:

• assign to teams
• set priority
• track SLA
• manage incident ownership
• follow alert status
• coordinate response

Priority examples:

P0
P1
P2

Higher priority usually means faster response is required.

Complete Architecture

A full alerting architecture connects application signals to team response.

[App / System]
↓
[Metrics & Logs]
↓
[Monitoring: Grafana / Sentry]
↓
[Alert Rules Engine]
↓
[Alert Triggered]
↓
[Incident Tools: Zenduty]
↓
[Team Responds]

This turns raw system behavior into action.

Alerting Tools

Tool	Role
Sentry	Alert rules for errors, crash-free rate, and performance degradation
Grafana	Metric evaluation and alert detection
AWS CloudWatch	Cloud metric monitoring and alarms
AWS SNS	Notification delivery
Slack / Email / SMS	Alert notification channels
PagerDuty / Squadcast / Zenduty	Incident management and on-call response

Alerting Table

Area	Explanation
Main goal	Notify the right people when conditions become unhealthy
Input	Events, metrics, logs
Rule	Threshold condition
Breach	Metric crosses expected limit
Alert	Notification is triggered
Incident	Trackable issue created for response
States	OK, Pending, Firing
Severity	Critical, Warning, Resolved
Response	Acknowledge, investigate, resolve

Summary

Alerting = Convert abnormal metrics into notifications and incidents so teams can respond quickly.

Metrics show signals
Thresholds define healthy limits
Breaches trigger alerts
Alerts notify people
Incidents organize response

Monitor first, alert only when action is needed.