Question 1

How do you approach a production incident you have never seen before?

Accepted Answer

Stabilise first, diagnose second. Establish scope — which users, which region, which service. Look for a recent change, because most incidents follow a deployment or configuration change, and roll back if one correlates. Then work down the stack from user symptom to component using dashboards and traces, communicating status while you work. The single most valuable habit is "what changed?", which resolves the majority of incidents faster than root-cause analysis. Interviewers listen for whether you restore service before fully understanding it, which is the correct priority.

Question 2

An ALB returns 502 but the target responds fine when you curl it directly. What is wrong?

Accepted Answer

Check target group health first — the health check path, port, protocol and expected status code, and whether the security group allows the load balancer to reach the target port. Then check for a keepalive mismatch, where the target closes idle connections faster than the load balancer expects, producing intermittent 502s. The keepalive mismatch is the subtle cause: the target's idle timeout must exceed the load balancer's, or the ALB sends a request on a connection the target is closing. That produces intermittent, hard-to-reproduce 502s that look like nothing is wrong.

Question 3

An ALB returns 503. What does that indicate?

Accepted Answer

503 from an ALB generally means no healthy targets are registered in the target group — either every target is failing health checks, or the group is empty because the Auto Scaling group or ECS service has no running instances. It is a capacity or health problem rather than an application error. Distinguishing 502 from 503 quickly is the diagnostic value: 502 means a target responded badly, 503 means there was nobody to send it to. That single distinction routes the investigation correctly in seconds.

Question 4

A Lambda function is being throttled. What do you check and what do you do?

Accepted Answer

Check whether the account concurrency limit or the function's reserved concurrency is the binding constraint, and whether another function is consuming the shared pool. Short term, raise reserved concurrency or the account limit; long term, reduce duration, batch records, or buffer with SQS so bursts are absorbed rather than rejected. The behaviour differs by invocation type — synchronous callers get a 429 immediately while asynchronous invocations are retried for up to six hours — and knowing that determines whether users are affected. Reserved concurrency on other functions is the cause people miss.

Question 5

DynamoDB is throttling at well below the provisioned capacity. Why?

Accepted Answer

A hot partition: traffic concentrated on one partition key value exceeds the per-partition limit even though the table total is within provisioned capacity. Use CloudWatch Contributor Insights to identify the hot key, then redesign with a higher-cardinality key or write sharding. Adaptive capacity mitigates moderate skew, so a table throttling despite headroom almost always has a genuinely extreme key distribution. Naming Contributor Insights turns a capacity mystery into a specific data-model fix.

Question 6

A user reports AccessDenied but you can see they have the right IAM policy. What are the possibilities?

Accepted Answer

An SCP on the account or OU denies it; a permissions boundary excludes it; a resource-based policy does not allow this principal; a VPC endpoint policy restricts it; a session policy narrowed the session; the resource is encrypted with a KMS key they cannot use; or the policy scopes an ARN that does not match. The KMS case is the one that catches strong candidates — S3 or EBS permissions are not enough when a customer-managed key is involved; the principal also needs kms:Decrypt. Reading the AccessDenied message for whether the deny is explicit or implicit is the fastest triage.

Question 7

An EC2 instance is unreachable over SSH. Walk through the diagnosis.

Accepted Answer

Confirm the instance is running and both status checks pass; use the EC2 Serial Console or an instance screenshot for boot output; verify the security group allows port 22 from your source; check the NACL allows both request and ephemeral return traffic; confirm the route table and gateway; check the key pair and OS user; and verify sshd is running in the guest. The Serial Console is the strongest move because it works when the network stack or sshd is broken. Naming Session Manager as the alternative that avoids this whole class of problem is the preventive half of the answer.

Question 8

An ECS task fails to start. How do you diagnose it?

Accepted Answer

Read the stopped-task reason, which usually names the cause: image pull failure from a missing execution role permission or wrong ECR path, insufficient CPU or memory on any instance, no ENI capacity in awsvpc mode, a failed health check, or a container exiting immediately. Then check container logs and the exit code. The stopped reason field is the single most useful thing and many candidates never mention it. ENI exhaustion in awsvpc mode is the non-obvious one — instances have limited ENIs, capping task density independently of CPU and memory.

Question 9

A pod is stuck in Pending on EKS. What do you check?

Accepted Answer

kubectl describe pod and read the Events. Usual causes are insufficient CPU or memory on any node, a node selector, affinity or taint no node satisfies, an unbound PersistentVolumeClaim, or IP exhaustion in the VPC CNI. Check the autoscaler or Karpenter logs for why capacity was not added. IP exhaustion is the EKS-specific cause: the VPC CNI assigns real VPC addresses to pods, so a small subnet caps pod count. Prefix delegation, which assigns /28 prefixes rather than individual IPs, is the mitigation worth naming.

Question 10

Pods are in CrashLoopBackOff. What do you check?

Accepted Answer

kubectl logs --previous for the crashed container, then describe for the exit code. Exit code 137 means OOM-killed, so the memory limit is too low or there is a leak. Other causes are a liveness probe with too short an initial delay, a missing ConfigMap or Secret, a bad entrypoint, or an unreachable startup dependency. Naming exit code 137 and the initialDelaySeconds mistake specifically is what makes this read as operational experience. Startup probes exist precisely to handle slow-starting applications without weakening the liveness check.

Question 11

A newly created private EKS cluster cannot pull images. Why?

Accepted Answer

Private nodes have no route to the internet, so without a NAT gateway they cannot reach public registries. Fix with NAT for general internet access, or better, ECR with VPC endpoints for the API, DKR and S3 so image pulls stay entirely inside the VPC. This is the most common first-day problem on a private cluster and the symptom is ImagePullBackOff on every pod. The ECR pull also needs the S3 gateway endpoint, because layers are stored in S3 — that is the detail people miss.

Question 12

An application intermittently fails with database connection errors. What is likely?

Accepted Answer

Connection exhaustion. Calculate maximum concurrency multiplied by pool size against the instance connection limit — an autoscaling event multiplies connections. Other causes are a failover dropping connections without client retry, stale connections not being recycled, or NAT gateway port exhaustion if traffic routes through NAT. The arithmetic is what makes this concrete: 500 concurrent Lambda executions each holding a connection will exhaust most RDS instances. NAT port exhaustion is the second-order cause that looks identical from the application side.

Question 13

A deployment succeeded but users see errors. How do you investigate?

Accepted Answer

Establish scope — all users or a subset, all regions or one. Check for new error groups introduced by this version, compare metrics split by version, and look at traces for a changed span breakdown. If it correlates with the deploy, roll back first and diagnose afterwards. Check whether a database migration accompanied the release. Rolling back before completing diagnosis is the correct instinct. The migration question matters because a rollback that leaves a changed schema may not restore service, which is the failure mode people discover mid-incident.

Question 14

The bill doubled this month with no traffic increase. Where do you look?

Accepted Answer

Query the Cost and Usage Report grouping by service and usage type for both months and compute the delta, then drill into account, tag and resource. Common causes are a new always-on resource, a recursive Lambda or pipeline, log or flow-log volume, cross-region or NAT data transfer from a deployment change, and storage accumulation from missing lifecycle rules. Usage-type-level delta analysis rather than service totals is the method, because service totals rarely explain the change. Naming recursive invocation as a specific cause shows awareness of the serverless failure mode that produces exactly this symptom.

Question 15

An S3 bucket suddenly costs far more than the data it holds. Why?

Accepted Answer

Usually requests or retrieval rather than storage: a job listing millions of objects repeatedly, frequent Glacier retrievals, or cross-region transfer. Storage-side causes are noncurrent versions with no expiry rule and incomplete multipart uploads, neither of which appears in a normal object listing. Incomplete multipart uploads are the invisible cost and S3 Storage Lens surfaces them directly. Adding an abort-incomplete-multipart-upload lifecycle rule to every bucket is the hygiene step that prevents it recurring.

Question 16

A Kinesis consumer is falling behind. What do you check?

Accepted Answer

Iterator age is the primary signal. Check whether the consumer is erroring, too slow, or bounded by shard read throughput; whether one shard is hot because of a low-cardinality partition key; and whether the record processing does something synchronous and slow. Options are more shards, enhanced fan-out, or a faster consumer. Iterator age matters because it is the only metric that warns of impending data loss when it approaches the retention period. A hot shard from a poor partition key is the cause people miss when they focus on consumer capacity.

Question 17

An SQS queue backlog is growing. How do you diagnose it?

Accepted Answer

Look at the age of the oldest message and the visible message count. Determine whether consumers are erroring, too slow or too few; whether visibility timeouts are expiring and causing redelivery that makes throughput worse; whether FIFO message groups are limiting concurrency; and whether the publish rate genuinely increased. Alerting on age of oldest message rather than depth is the operational point, because depth alone is misleading during a legitimate spike. The visibility-timeout death spiral — redelivery while still processing — is the non-obvious cause.

Question 18

A Lambda that worked in testing fails in production with permission errors. Why?

Accepted Answer

Locally or in testing it ran with broader credentials than the execution role it uses in production. Check the execution role against the APIs the code calls, and whether a VPC attachment, VPC endpoint policy, SCP or resource policy is blocking it. The credentials delta between development and deployed identity is the most common cause of this exact symptom. Naming it immediately shows you understand how serverless identity works rather than only how to write the function.

Question 19

A Lambda attached to a VPC can no longer reach a third-party API. What happened?

Accepted Answer

Attaching a function to a VPC removes its default internet access, so outbound calls now need a NAT gateway in a route table the function's subnets use, or a VPC endpoint for AWS services. The function should be in private subnets with a NAT route, not public ones. Placing the function in a public subnet does not help, because a Lambda ENI has no public IP — this is the specific misunderstanding that wastes time. Private subnets plus NAT is the only configuration that works.

Question 20

Latency is fine at p50 but terrible at p99. What causes that and how do you find it?

Accepted Answer

Tail latency usually comes from cold starts, garbage collection pauses, lock contention, a slow dependency affecting a subset of requests, cache misses, or one overloaded backend. Find it with tracing filtered to slow traces, comparing the span breakdown of a slow request against a fast one. Filtering traces to the slow tail rather than looking at averages is the technique, because p99 requests are structurally different from p50 ones. Naming cold starts and garbage collection as specific candidates shows you have chased this before.

Question 21

Users in one region report slowness while others are fine. How do you investigate?

Accepted Answer

Check whether traffic is being routed to a distant region because the nearest is unhealthy or at capacity; check target health and capacity settings there; look at cross-region dependency latency such as a database in another region; and use synthetic checks from multiple locations to confirm it is regional rather than client-side. A cross-region dependency introduced by a recent change is the cause worth naming, because it looks like a network problem and is actually a configuration one. Synthetic canaries from multiple locations are what turn user reports into evidence.

Question 22

You are paged that a service is down but all metrics look healthy. What now?

Accepted Answer

Trust the user report over your instrumentation. Check from outside with a synthetic check or a manual request from a different network — the failure may be DNS, certificate expiry, load balancer misconfiguration or a CDN issue, none of which server-side metrics see. Also check whether the metrics pipeline itself is broken. Internal metrics are collected by the system that may be broken, which is why black-box monitoring exists. Certificate expiry is the classic cause of a service that is perfectly healthy and completely unreachable.

Question 23

CloudFront serves stale content after a deployment. What do you do?

Accepted Answer

Check the Cache-Control headers on the origin and the distribution's cache policy and TTLs. Invalidate the affected paths as an immediate fix, but the durable answer is versioned or content-hashed asset filenames so a new deployment references new paths and old objects simply age out. Relying on invalidation as the normal release mechanism is a design smell — it is rate-limited, slow and a single point of failure in the deploy. Content-hashed filenames make cache correctness automatic, which is why build tools produce them.

Question 24

A CloudFront distribution has a very low cache hit rate. What do you investigate?

Accepted Answer

Origin Cache-Control headers, since CloudFront honours them; the cache policy's included query strings, headers and cookies, because unnecessary components fragment the cache; whether responses vary by cookie; and whether content is being invalidated too frequently. Tracking query parameters in the cache key is the most common specific cause, creating a separate cache entry per user for identical content. Excluding them can transform hit rate, which makes it a satisfying and concrete fix.

Question 25

A batch job that normally takes 30 minutes has run for 4 hours. What do you check?

Accepted Answer

Look at the execution graph for a stage consuming disproportionate time; check for data skew where one task does most of the work; check whether input volume grew or the small-file count exploded; check whether scaling is constrained by a quota; and check for a downstream dependency throttling writes. Skew shows as a large gap between median and maximum task duration in a stage, which is the specific diagnostic signature. The small-file problem is invisible unless you look at input file counts rather than total bytes.

Question 26

An Athena query that used to be fast is now scanning far more data. Why?

Accepted Answer

Partition pruning was lost — a filter now wraps the partition column in a function, filters on a different column, or partitions were not registered after new data landed. Also check whether the data format changed from Parquet to something row-oriented, or whether small files have accumulated. The lost-pruning case is subtle because the query still returns correct results, just after scanning everything. Naming partition projection as the fix for missing partition registration is the current best-practice answer.

Question 27

An RDS instance is at 100% CPU. What do you do?

Accepted Answer

Open Performance Insights to find the queries and wait events dominating load; check for missing indexes and full scans; look for lock contention and long-running transactions; correlate with recent deployments or batch jobs; and check whether the load is connection churn. Scale up as a mitigation, then fix the query or index. Diagnosing before scaling is what interviewers assess, because scaling a badly-indexed query just makes the same problem more expensive. Naming Performance Insights and wait events rather than generic database debugging is the AWS-specific part.

Question 28

You get "Quota exceeded" or "InsufficientInstanceCapacity" during a deployment. What do you do?

Accepted Answer

For a quota error, identify which quota, free capacity by removing unused resources, deploy to another AZ or a smaller shape as an immediate workaround, and request an increase — which takes days. For insufficient capacity, try a different instance type or AZ, and use capacity reservations for critical workloads in future. The lead time on quota increases is the practical point: discovering the limit during a launch is already too late. Adding quota headroom checks to launch planning is the preventive answer after the immediate fix.

Question 29

A CloudFormation stack is stuck in UPDATE_ROLLBACK_FAILED. How do you recover?

Accepted Answer

Identify which resources failed to roll back, usually because they were modified outside CloudFormation so the rollback cannot reconcile them. Fix the underlying issue manually, then call ContinueUpdateRollback, optionally skipping the problematic resources so the stack returns to a usable state. Knowing ContinueUpdateRollback with a resources-to-skip list is the specific recovery step, and drift from manual changes is the usual cause. That connection between drift and stuck stacks is what makes the answer complete.

Question 30

A Terraform apply failed halfway. What is the state of the world?

Accepted Answer

Terraform recorded what it created, so re-running continues from there. If a resource was created but the state write failed, it is orphaned — it exists in AWS but not in state — so the next apply tries to create it again and fails on a name conflict. Import the orphan or delete it, then re-run. Recognising the orphaned-resource symptom from a name-conflict error is the practical skill. The other check is a stale lock left by a killed process, requiring force-unlock — which must only be done when no apply is genuinely running.

Question 31

A scheduled job silently stopped running and nobody noticed for a week. How do you prevent that?

Accepted Answer

Alert on absence, not just on failure — a heartbeat or dead-man's-switch that fires when an expected completion signal does not arrive within a window. Monitor data freshness downstream so a stale output triggers an alert regardless of cause, and track job success as a metric with an expected rate. Alerting on absence is the key concept, because a job that never starts produces no errors to alert on. Data-freshness monitoring is the complementary control that catches the failure from the consumer's perspective.

Question 32

A service works for some users and not others, with no regional pattern. What could it be?

Accepted Answer

A partial rollout — a canary serving a percentage of traffic; a feature flag enabled for a segment; sticky sessions pinning some users to a broken target; a cache serving stale content to some; data-dependent failure affecting particular accounts; or a client-version difference. Correlate failures by user attribute rather than by infrastructure. The instinct to correlate by user attribute is what solves these, and it requires logging user, tenant and version identifiers on every request. That is the observability design decision that makes this debuggable at all.

Question 33

API Gateway returns 429 to clients. What is happening?

Accepted Answer

Throttling at one of several levels: the account-level limit, the stage or method throttle, or a usage plan attached to the caller's API key. Check which by looking at the throttle configuration and the CloudWatch throttle metrics per stage and per usage plan. The layered throttling model means the fix differs by level — raising a usage plan limit for one consumer versus raising the stage limit for everyone. Identifying which limit is binding before changing anything is the diagnostic discipline.

Question 34

An SNS to Lambda integration is losing events. Why might that be?

Accepted Answer

Direct SNS-to-Lambda has limited retry behaviour, so if the function is throttled or erroring beyond the retry policy the event is dropped unless a dead-letter queue is configured. The robust pattern is SNS to SQS to Lambda, so the queue buffers and retries independently. This is a very common real-world data-loss cause and the fix is architectural rather than a setting. Naming the SNS-to-SQS-to-Lambda pattern as the resilient form is the answer interviewers are looking for.

Question 35

A NAT gateway is dropping connections. What is likely?

Accepted Answer

Port exhaustion: a NAT gateway supports a bounded number of simultaneous connections per destination per IP, so a workload opening very many concurrent outbound connections to one endpoint exhausts them. Mitigate by adding NAT IPs, distributing across destinations, using VPC endpoints to remove AWS traffic from NAT, or reusing connections. The symptom is intermittent connection failures under load with no clear application cause, and the ErrorPortAllocation metric confirms it. VPC endpoints for S3 and DynamoDB often remove enough traffic to solve it outright.

Question 36

You suspect a credential has been compromised. What is your first action?

Accepted Answer

Contain — deactivate the access key or attach a deny-all policy, and revoke active role sessions with a token-issue-time condition. Preserve CloudTrail evidence before deleting anything. Investigate what the credential did and could reach, rotate everything in that blast radius, then remediate the leak source. Preserving evidence before deleting is the step people skip under pressure, and deleting the identity destroys the history needed to scope the blast radius. Contain, preserve, investigate, remediate is the sequence interviewers assess.

Question 37

An EC2 instance keeps failing status checks. What do you check?

Accepted Answer

Distinguish the two checks: a system status check failure is an underlying host problem, remedied by stopping and starting the instance to move it to new hardware. An instance status check failure is inside the guest — exhausted memory, a corrupt filesystem, a kernel issue — diagnosed through the serial console and system logs. Knowing which check maps to which responsibility makes triage immediate. Auto recovery handles the system-check case automatically on most current instance types, which is worth naming as the preventive configuration.

Question 38

S3 uploads from your application are intermittently slow. What do you investigate?

Accepted Answer

Whether the bucket is in the same region as the compute, since cross-region writes add latency and egress; whether large uploads use multipart with parallelism; whether many small objects are written sequentially rather than concurrently; and whether a very high sustained rate with sequential key prefixes is limiting distribution. Bucket and compute co-location is the most frequent real cause and easy to get wrong when a bucket predates the workload. The sequential-prefix issue only matters at very high rates, and saying so rather than treating it as universal is the accurate answer.

Question 39

Two instances in the same VPC cannot communicate. How do you troubleshoot?

Accepted Answer

Check they are in the same or connected VPCs with routes both ways; check security groups allow the traffic in the right direction; check NACLs allow both request and ephemeral return traffic; check the OS firewall; then use VPC Reachability Analyzer, which traces the configured path and names the blocking component. Reachability Analyzer is the strongest move because it identifies the exact rule blocking the path rather than requiring you to reason through four layers. The OS firewall is the layer people forget when AWS configuration looks perfect.

Question 40

Route 53 failover is not switching traffic. Why?

Accepted Answer

Check the health check is actually failing — it may be probing a path that still returns 200 while the application is broken. Check the record type is failover with primary and secondary correctly associated, and remember DNS TTL means clients keep the old answer until it expires, so failover is never instant. A shallow health check that only pings the load balancer is the most common cause of failover not triggering. Naming calculated health checks, which combine several signals, is the fix that makes failover meaningful.

Question 41

A CloudWatch alarm did not fire during an outage. What went wrong?

Accepted Answer

Most likely the metric stopped being published entirely, so the alarm sat in INSUFFICIENT_DATA rather than ALARM, because the default treatment of missing data is not to alarm. Also check the evaluation periods, whether the threshold was on an average that masked the problem, and whether the notification target was working. The missing-data configuration is the specific gap: a total outage produces no metrics at all, which is exactly when you need the alarm. Setting missing data to breaching for availability metrics is the fix.

Question 42

A Glue or EMR job fails with out-of-memory errors. What do you investigate?

Accepted Answer

Data skew concentrating a partition on one executor; a broadcast join with a side too large; collecting results to the driver; too many partitions causing overhead or too few causing large partitions; and insufficient executor memory for the workload. Check the Spark UI for task-level memory and duration distribution. Collecting to the driver is the anti-pattern that produces sudden driver OOM regardless of executor sizing, and it is a common bug in code written against small test data. Skew is the other dominant cause, visible as one task far slower than the rest.

Question 43

How do you handle an incident where you cannot find the root cause quickly?

Accepted Answer

Restore service by whatever safe means — roll back, fail over, shed load, disable the feature — and continue investigating with the pressure off. Preserve logs, metrics and affected state for analysis. Communicate that service is restored while investigation continues, then run a blameless postmortem. The priority order is what is being tested: mitigation before understanding. Engineers who keep debugging a live outage because they want to know why are optimising for curiosity over users, and interviewers watch for exactly that.

Question 44

You inherit an AWS account with no documentation. How do you understand what is running?

Accepted Answer

AWS Config for resource inventory and configuration history; the Cost and Usage Report to see what actually costs money, which reveals what matters; CloudWatch for what receives traffic; CloudTrail for who has been changing things; and VPC topology tooling for the network. Then map resources to owners and start documenting. Following the money is the practical shortcut — expensive resources are almost always the important ones, and idle resources with no traffic are removal candidates. That heuristic orients you faster than reading configuration.

Question 45

An AWS API call intermittently returns 503 or a throttling error. What is the correct response?

Accepted Answer

Transient errors and throttling are expected in a distributed system, so the first question is whether the client retries with exponential backoff and jitter — most SDKs do by default, but hand-written HTTP calls often do not. Then check whether a quota or rate limit is being hit, and the service health dashboard for an actual event. The expectation that clients handle transient errors is the substance: a design treating every 503 as an outage reports constant failures. Jitter matters because synchronised retries cause the thundering herd that prolongs the problem.

Question 46

Describe a production incident you handled and what you learned.

Accepted Answer

A strong answer states the symptom and user impact, how it was detected, the diagnostic steps and what was eliminated, the mitigation and its timing, the actual root cause, and — most importantly — the systemic change that prevents recurrence, whether a code fix, a guardrail, an alert or a process change. Interviewers are assessing whether you distinguish mitigation from root cause and whether you drive follow-through. An answer ending at "we restarted it and it worked" with no preventive action is the weak version, however dramatic the incident.

AWS Troubleshooting & Scenario Questions Interview Questions and Answers

How to use this set