Monitoring
Monitoring exists to answer three questions quickly: is the service healthy, if not where is it failing, and what changed? A dashboard that does not answer those questions is decoration.
The standard. Cloud and network monitoring — metrics, logs, alarms and dashboards — designed around the customer's operational questions, not a service inventory.
What "established" means
| Property | Test |
|---|---|
| Complete | Every critical component emits metrics and logs; nothing important is unmonitored |
| Relevant | Metrics map to workload behaviour, not only to infrastructure health |
| Alarmed | Every metric that indicates a user-visible problem has an alarm with an owner |
| Actionable | Every alarm has a runbook entry and a known response |
| Owned | Each alarm routes to a named team, not a shared inbox |
| Reviewed | Thresholds and coverage are reviewed on a cadence, not set once |
Layers of telemetry
┌─ Business KPIs (orders, signups, exam submissions)
├─ Application (request rate, errors, latency)
├─ Runtime (CPU, memory, GC, connections)
├─ Platform (ALB, RDS, cache, queue)
└─ Network (flow logs, VPC endpoints, DNS)
An alarm on CPU utilisation tells you a symptom. An alarm on checkout success rate tells you the problem.
Metrics
Infrastructure
- EC2:
CPUUtilization,StatusCheckFailed, disk and network I/O, memory via the CloudWatch agent. - ECS / Fargate: CPU and memory utilisation, task count, deployment state.
- EKS: pod CPU and memory, container restarts, pending pods, available replicas.
- Lambda: invocations, errors, duration, throttles, concurrent executions, iterator age.
Platform
- Elastic Load Balancer: request count, 4xx and 5xx,
TargetResponseTime, healthy and unhealthy hosts, active connections. - API Gateway: count, 4xx, 5xx,
Latency,IntegrationLatency. - CloudFront: requests, bytes, error rate, cache hit rate, latency.
- RDS: CPU, free storage, connections, IOPS, latency,
DatabaseLoad, replica lag. - DynamoDB: consumed capacity, throttle events, throttled requests.
- ElastiCache: connections, hits and misses, evictions, memory, CPU.
- SQS: visible messages, age of oldest message, DLQ depth.
Application
- Request rate, error rate and duration per endpoint (RED).
- Saturation: thread pool, connection pool, queue depth.
- Business events: successful checkouts, failed payments, completed jobs.
See Monitoring metrics for the per-tier reference.
Logs
- Centralised. CloudWatch Logs with a consistent naming convention:
/<service>/<environment>/<component>. - Structured. JSON with timestamp, level, service, correlation id, request id, message and relevant fields.
- Retained deliberately. 30 days hot by default; longer for audit and security logs. Retention is a cost decision, made explicitly.
- Queryable. CloudWatch Logs Insights queries saved and shared for common investigations.
- Not a dumping ground. No secrets, tokens, personal data or full request bodies.
Alarms
Alarm design rules:
- Alarm on user impact first. Error rate, latency, success rate, queue age.
- Then on leading indicators. Saturation, connection pool, disk growth.
- Every alarm has an owner and a runbook. An alarm with no response path is noise.
- Composite alarms to reduce noise. A single "service degraded" alarm from several signals is more actionable than five separate ones.
- Review alarm volume. If an alarm fires more than it is acted on, its threshold is wrong or its response is missing.
- Snooze deliberately. A maintenance window disables expected alarms; an alarm disabled permanently should be deleted.
| Alarm class | Example | Response |
|---|---|---|
| Availability | 5xx rate above 1% for 3 minutes | Incident process |
| Latency | p95 above target for 5 minutes | Investigate, scale, or roll back |
| Saturation | CPU above 85% for 10 minutes | Scale out; review capacity |
| Data | Replica lag above 30 seconds | Failover decision |
| Queue | Oldest message older than 5 minutes | Consumer health check |
| Security | Root login, guard duty finding | Security incident |
Dashboards
Dashboards are organised by audience, not by service inventory.
| Dashboard | Audience | Content |
|---|---|---|
| Service health | On-call | Request rate, error rate, latency, saturation, deployment markers |
| Platform | Platform team | Cluster, database, network, queue health |
| Business | Product and leadership | Workload KPIs with trend |
| Security | Security team | Findings, Config compliance, GuardDuty |
| Cost | Finance and platform | Spend by service, tag, environment |
Every dashboard shows the same time range and annotates deployments. Correlating a latency change with a deployment is the most common first diagnostic step.
Network monitoring
- VPC Flow Logs enabled on every VPC, delivered to a central bucket.
- Rejected traffic reviewed for unexpected patterns.
- VPC endpoints monitored for failures.
- DNS query logging where the workload depends on internal resolution.
- Reachability checks from outside the environment for public endpoints.
Evidence
- Metric and alarm inventory with owners.
- Dashboard exports or screenshots.
- Alarm-to-runbook mapping.
- Log group and retention configuration.
- Sample CloudWatch Logs Insights query results.