Improve operational excellence
Use ownership, observability, automation, safe change, runbooks, feedback, and measurable operations to reduce toil and failure.
- Lesson
- d3-lesson
- Practice pool
- d3-questions
- Application
- sap-l07, sap-l08
Use evidence to improve operational excellence, security, performance, reliability, and cost without destabilizing production.
Use ownership, observability, automation, safe change, runbooks, feedback, and measurable operations to reduce toil and failure.
Assess and prioritize identity, data, network, vulnerability, detection, response, compliance, and governance improvements.
Profile bottlenecks and change the constrained layer using representative load, telemetry, and controlled experiments.
Analyze incidents and dependency risks, improve fault isolation and recovery, and validate through exercises and controlled failure.
Use usage and ownership evidence to remove waste and right-size architecture without violating quality or resilience requirements.
Improvement starts with evidence about a current workload and a stated objective. Do not redesign from a diagram that ignores traffic, failures, team capability, dependencies, and operating cost. Establish the baseline, identify the constrained or risky layer, propose the smallest change that tests the hypothesis, deploy safely, compare results, and preserve rollback.
Use the AWS Well-Architected pillars together. Operational excellence, security, reliability, performance efficiency, cost optimization, and sustainability influence one another. An improvement in one pillar is not complete if it silently violates another requirement.
| Objective | Required judgment | Proof |
|---|---|---|
| 3.1 Improve operational excellence | Strengthen ownership, observability, automation, runbooks, change, feedback, and learning | Operators detect, diagnose, and recover from a known failure faster |
| 3.2 Improve security | Prioritize identity, data, network, vulnerability, detection, response, compliance, and governance gaps | Risk is reduced by an implemented control and verified test |
| 3.3 Improve performance | Find the constrained layer with representative telemetry and experiments | Tail latency or throughput improves under the same workload |
| 3.4 Improve reliability | Reduce failure probability or blast radius and improve recovery | A controlled failure remains within objective and recovery is timed |
| 3.5 Identify cost optimization | Remove waste and right-size architecture using ownership and usage evidence | Unit cost falls without breaching quality or resilience |
Inventory workload owners, users, business criticality, dependencies, data, Regions, accounts, deployment path, quotas, availability targets, RTO/RPO, security obligations, traffic, cost, and known risks. Review architecture and operations with the relevant Well-Architected lenses. Distinguish observed facts from assumptions and backlog wishes.
Collect useful telemetry: business transactions, request rates, error rates, tail latency, saturation, queue age, throttles, retries, deployment changes, dependency health, recovery time, security findings, and cost allocation. High-cardinality traces and logs need privacy, access, retention, and cost design. A dashboard without an owner or action threshold does not improve operations.
Prioritize by business impact, risk, recurrence, effort, reversibility, and dependencies. Fix dangerous unknowns and unsupported components before optimizing cosmetic metrics. Avoid a large rewrite when a bounded change can prove the desired outcome.
Give each workload and alarm an owner. Maintain versioned runbooks for frequent and high-impact events. Automate repeated safe operations while keeping approval for ambiguous or destructive actions. Use infrastructure as code, peer review, CI/CD, small reversible changes, and post-deploy verification. Track toil, false alerts, failed changes, time to detect, time to restore, and recurring incident causes.
Game days and controlled exercises expose missing permissions, stale documentation, unavailable backups, quota gaps, and unclear authority before a real event. Feed lessons into code, architecture, runbooks, and training. A retrospective that assigns no corrective owner and retest does not close the loop.
Start with attack paths and obligations, not a generic list of services. Review workforce and workload identity, cross-account trust, public exposure, data classification, key and secret access, software supply chain, vulnerability exposure, logging, findings, incident response, backup protection, and privileged change.
Prioritize conditions that combine exploitability, privilege, data sensitivity, blast radius, and weak detection. Replace long-term credentials with temporary sessions, reduce wildcard access, restrict public paths, harden build and deployment identities, and centralize evidence where justified. Use GuardDuty, Security Hub CSPM, Inspector, Config, Access Analyzer, CloudTrail, and other controls as parts of an owned detection and remediation system, not as a product checklist.
Measure the full request path and locate the bottleneck. CPU, memory, connections, locks, storage IOPS, cache miss, database query, partition skew, network transfer, downstream latency, queue depth, and quota limits require different responses. Optimize p95 and p99 where tail behavior affects users; an average can hide saturation.
Change one layer and compare under representative load. Right-size compute, adjust autoscaling signals, improve indexes and access patterns, cache safe repeated reads, introduce asynchronous processing, use content delivery, choose a better storage tier, or change concurrency only when evidence points there. Larger instances can hide a query or locking problem without improving scalability.
Analyze incidents and dependency graphs. Remove single points, distribute across Availability Zones where supported, isolate tenants or cells, add timeouts and bounded retries, make operations idempotent, add queues and dead-letter handling, protect quotas, and design graceful degradation. Review shared identity, DNS, networks, artifacts, secrets, and observability as dependencies.
Improve recovery as well as prevention. Protect backups across appropriate accounts or Regions, retain known-good artifacts, and test restoration. Define detection, declaration, failover, validation, failback, and communication. Controlled failure testing should have clear scope, stop conditions, safety, and rollback.
Attribute spend to owners and workloads. Identify idle resources, overprovisioning, low utilization, unattached storage, stale snapshots, excessive logs, NAT and data-transfer paths, expensive queries, duplicate environments, and commitments that do not match usage. Relate cost to business units such as request, customer, transaction, or dataset.
Rightsize after observing cycles and peak behavior. Use autoscaling, serverless, lifecycle policies, storage classes, Spot where interruption is acceptable, and purchase commitments for stable demand. Include engineering and operational complexity. Removing redundancy, logs, or tests may create a false saving.
| Observation | First move | Avoid |
|---|---|---|
| High p99 with low average | Trace tail requests and saturation | Scaling everything equally |
| Frequent failed deployments | Improve artifact, test, rollout, and rollback path | Adding manual approvals with no evidence |
| Recurring account compromise path | Fix identity issuance and privilege boundaries | Rotating credentials after each incident only |
| Recovery misses RTO | Time each recovery dependency and automate bottlenecks | Buying more backup storage without restore tests |
| Rising cost with stable demand | Attribute unit cost and inspect changed layers | Cutting availability before finding waste |
| Alert fatigue | Tie alerts to user impact, owner, and runbook | Raising thresholds until alerts disappear |
An API has rising p99 latency, monthly cost increased 40%, and two recent deployments required manual rollback. Availability remains within target.