Question 1

How do you approach a production incident you have never seen before?

Accepted Answer

Stabilise first, diagnose second. Check the scope — which users, which region, which service. Look for a recent change, because most incidents follow a deployment or configuration change, and roll back if one correlates. Then work down the stack from user symptom to component, using dashboards and traces rather than guessing, and communicate status while you work. The single most valuable habit is "what changed?" — it resolves the majority of incidents faster than root-cause analysis does. Interviewers are listening for whether you restore service before understanding it, which is the correct priority order.

Question 2

Your load balancer returns 502 but the backend responds fine when you curl it directly. What is wrong?

Accepted Answer

Almost always health checks. Verify the firewall allows 130.211.0.0/22 and 35.191.0.0/16 to the health-check port, that the check path returns 200, and that the backend service port and protocol match. Also check the backend service timeout against actual response time, and whether the backend closes idle connections faster than the load balancer expects, which causes intermittent 502s. Reciting the health-check source ranges immediately is the strongest possible signal here. The keepalive mismatch is the subtle second cause and is worth naming because it produces intermittent rather than total failure, which is much harder to diagnose.

Question 3

A GKE pod is stuck in Pending. Walk me through the diagnosis.

Accepted Answer

kubectl describe pod and read the Events. The usual causes are insufficient CPU or memory on any node, a node selector, affinity rule or taint no node satisfies, an unbound PersistentVolumeClaim, or pod IP range exhaustion in a VPC-native cluster. If cluster autoscaling is on, check its events — it logs precisely why it did or did not add a node. Going to Events first rather than guessing is what is being assessed. The autoscaler detail that impresses is that it will not add a node if the pod could never fit on the largest node in any pool, and it says so explicitly in its events.

Question 4

Pods are in CrashLoopBackOff. What do you check?

Accepted Answer

kubectl logs --previous for the crashed container's output, then kubectl describe for the exit code. Exit code 137 means OOM-killed, so the memory limit is too low or there is a leak. Other common causes are a liveness probe with too short an initial delay, a missing ConfigMap or Secret, a bad entrypoint, or a dependency unreachable at startup. Naming exit code 137 and the initialDelaySeconds mistake specifically is what makes this read as operational experience. Startup probes exist precisely to handle slow-starting applications without weakening the liveness check.

Question 5

A user reports "permission denied" but you can see they have the right role. What are the possibilities?

Accepted Answer

The role is granted on a different project or resource; an IAM condition on the binding is not satisfied; a VPC Service Controls perimeter blocks access from their location; an organisation policy or deny policy overrides it; the resource uses CMEK and they lack the KMS key permission; the change has not propagated yet; or a service account access scope on a VM is blocking the API despite correct IAM. The CMEK case is the one that separates strong candidates — Storage Admin alone is not enough when a customer-managed key is involved. Naming Policy Troubleshooter as the tool that decides between these hypotheses definitively is the right close.

Question 6

A deployment succeeded but users see errors. How do you investigate?

Accepted Answer

Confirm the scope — all users or a subset, all regions or one. Check Error Reporting for new error groups introduced by this version, compare metrics split by revision, and look at Cloud Trace for a changed span breakdown. If it correlates with the deploy, roll back first and diagnose afterwards. Check whether a database migration accompanied the release. Rolling back before completing diagnosis is the correct instinct and interviewers watch for it. The migration question matters because a rollback that leaves a changed schema may not restore service, which is the failure mode people discover mid-incident.

Question 7

Cloud Run latency spiked but CPU is low. What is happening?

Accepted Answer

Likely cold starts or downstream blocking. Check instance count churn against request rate for cold starts, then check concurrency — if requests are queueing on instances at their concurrency limit, latency rises without CPU saturation. Also check downstream dependency latency, database connection pool exhaustion, and whether CPU is being throttled between requests. Low CPU with high latency almost always means waiting rather than computing, so the diagnosis should go to queueing and downstream dependencies rather than scaling up. Recognising that pattern immediately is the point of the question.

Question 8

A Cloud SQL instance is at 100% CPU. What do you do?

Accepted Answer

Open Query Insights to find the queries contributing most load, look for missing indexes and full scans, check for lock contention and long-running transactions, and correlate with recent deployments or batch jobs. Short term, scale up or throttle the offending workload; long term, fix the query or index. Also check whether the load is connection churn rather than query cost. Diagnosing before scaling is what interviewers assess — scaling a badly-indexed query just makes the same problem more expensive. Naming Query Insights specifically, rather than describing generic database debugging, is the GCP-flavoured answer.

Question 9

A BigQuery dashboard query that took 3 seconds now takes 90. What changed?

Accepted Answer

Compare job execution details between runs. Check whether partition pruning was lost because a filter now wraps the partition column in a function or filters a different column; whether the table grew significantly; whether slot availability dropped because another workload is consuming the reservation; whether caching stopped applying; or whether a schema or clustering change altered the plan. The lost-pruning case is the subtle and most common one — the query still returns correct results, just after scanning everything. Naming CAST or DATE() applied to the partition column as the specific pattern is a strong practical detail.

Question 10

A Pub/Sub subscription backlog is growing. How do you diagnose it?

Accepted Answer

Look at oldest unacknowledged message age and undelivered message count. Determine whether consumers are erroring, too slow, or too few; whether ack deadlines are expiring and causing redelivery that makes throughput worse; whether ordering keys are causing head-of-line blocking; and whether the publish rate genuinely increased. Alerting on oldest_unacked_message_age rather than backlog count is the operational point, because a large backlog during a legitimate spike is fine while a growing age means you are permanently behind. The ack-deadline death spiral is the non-obvious cause worth naming.

Question 11

The bill doubled this month with no traffic increase. Where do you look?

Accepted Answer

Query the billing export in BigQuery, grouping by service and SKU for both months and computing the delta, then drill into project, label and resource. Common causes are a new always-on resource, a recursive function or pipeline, log or flow-log volume, BigQuery scan growth from a new dashboard, storage class or versioning accumulation, and cross-region egress introduced by a deployment change. SKU-level delta analysis rather than service-level totals is the method, because service totals rarely explain the change. Naming recursive triggers as a specific cause shows awareness of the serverless failure mode that produces exactly this symptom.

Question 12

A VM cannot reach the internet. What do you check?

Accepted Answer

Whether it has an external IP or a Cloud NAT configured for its subnet; whether a route to the internet gateway exists; whether an egress firewall rule denies the traffic; whether DNS resolution works; and whether the guest OS firewall or routing table is the problem. Network Intelligence Center Connectivity Tests will trace the path and name the blocking rule. The most common real cause on a private cluster or VM is simply missing Cloud NAT, which manifests as failing package installs and ImagePullBackOff. Naming Connectivity Tests turns a list of hypotheses into a definitive one-step answer.

Question 13

You get "Quota exceeded" during a deployment. What do you do?

Accepted Answer

Identify which quota from the error — commonly regional CPU, in-use IP addresses, or an API rate limit. Check current usage against the limit in the quotas page, free capacity by removing unused resources, deploy to another region or a smaller shape as an immediate workaround, and request an increase, which can take days. Then add quota headroom checks to your pre-launch process. The lead-time point is the practical one: quota increases are not instant, so discovering the limit during a launch is already too late. Adding quota review to capacity planning is the preventive answer interviewers want after the immediate fix.

Question 14

A GKE cluster stops adding nodes even though pods are pending. Why?

Accepted Answer

Most likely IP exhaustion in the VPC-native cluster's pod secondary range — each node consumes a pod CIDR block, so the range size caps the node count. Other causes are hitting the node pool maximum, regional CPU or quota limits, or the pending pod requesting more resources than the largest available machine type. The pod-range limit is a genuinely non-obvious constraint decided at cluster creation and hard to change later. Naming the mitigation — reducing maximum pods per node to shrink the per-node block — shows you understand the mechanism rather than just the symptom.

Question 15

An application intermittently fails with connection errors to Cloud SQL. What is likely?

Accepted Answer

Connection exhaustion. Calculate maximum instances multiplied by pool size against the instance connection limit — an autoscaling event multiplies connections. Other causes are a failover dropping connections without client retry, missing connection recycling so stale connections are used, or Cloud NAT port exhaustion if traffic goes through NAT. The arithmetic is what makes this answer concrete: 100 Cloud Run instances with a pool of 10 is 1,000 connections, which exhausts most instances. Cloud NAT port exhaustion is the second-order cause that catches people because it looks identical from the application side.

Question 16

Cloud Function invocations are being retried forever. What happened?

Accepted Answer

Retry is enabled on an event-driven function and the function fails deterministically on a particular message, so it is redelivered with backoff for up to seven days. The fix is a guard on event age inside the function, a dead-letter configuration on the underlying subscription, and making the function distinguish transient failures from permanent ones. The event-age guard is the specific idiom: check the event timestamp and acknowledge instead of failing once it exceeds a threshold. Without it, a poison message consumes resources and generates cost for a week.

Question 17

A Terraform apply failed halfway. What is the state of the world and what do you do?

Accepted Answer

Terraform recorded the resources it created, so re-running continues from there. If a resource was created but the state write failed, it is orphaned — it exists in GCP but not in state — so the next apply tries to create it again and fails on a name conflict. Import the orphan, or delete it manually if it is safe, then re-run. Recognising the orphaned-resource symptom from a name-conflict error is the practical skill. The other check is whether a lock was left behind by the killed process, requiring force-unlock — which must only be done when no apply is genuinely running.

Question 18

Users in one region report slowness while others are fine. How do you investigate?

Accepted Answer

Check whether the global load balancer is routing them to a distant region because their nearest region is unhealthy or at capacity; check backend health and capacity settings in that region; look at regional dependency latency such as a database in another region; and check Performance Dashboard for packet loss between zones. Uptime checks from multiple locations confirm whether it is regional. Capacity spillover is the specific cause worth naming, because a region at its configured maximum silently sends traffic elsewhere, which looks like a network problem and is actually a configuration one.

Question 19

A batch job that normally takes 30 minutes has been running for 4 hours. What do you check?

Accepted Answer

Look at the execution graph for a stage consuming disproportionate time, check for data skew where one worker does most of the work, check whether input data volume grew or the small-file count exploded, check whether autoscaling is constrained by quota, and check for a downstream dependency that is throttling writes. Skew shows as a large gap between median and maximum worker time in a stage, which is the specific diagnostic signature. The small-file problem is the other frequent cause and is invisible unless you look at input file counts rather than total bytes.

Question 20

You are paged that a service is down but all your metrics look healthy. What now?

Accepted Answer

Trust the user report over your instrumentation. Check from outside with an uptime check or a manual request from a different network — the failure may be DNS, certificate expiry, load balancer configuration, or a CDN issue, none of which your server-side metrics see. Check whether the metrics pipeline itself is broken. The insight is that internal metrics are collected by the system that may be broken, which is why black-box monitoring exists. Certificate expiry is the classic cause of a service that is perfectly healthy and completely unreachable.

Question 21

A newly created private GKE cluster cannot pull images. Why?

Accepted Answer

Private nodes have no external IP, so without Cloud NAT they cannot reach public registries. The fixes are configuring Cloud NAT for outbound access, or better, using Artifact Registry with Private Google Access enabled on the subnet so image pulls stay on Google's network entirely. This is the single most common first-day problem on a private cluster and the symptom is ImagePullBackOff on every pod. Preferring Private Google Access with Artifact Registry over Cloud NAT is the better answer because it avoids the internet path altogether.

Question 22

Cloud Storage uploads from your application are intermittently slow. What do you investigate?

Accepted Answer

Check whether the bucket is in the same region as the compute, since cross-region writes add latency and egress; whether uploads are resumable and correctly chunked for large objects; whether many small objects are being written sequentially rather than in parallel; and whether a very high sustained write rate with sequential key prefixes is limiting distribution. Bucket and compute co-location is the most frequent real cause and is easy to get wrong when a bucket was created earlier for a different purpose. The sequential-prefix issue only matters at very high rates, and saying so rather than treating it as universal is the accurate answer.

Question 23

Your Dataflow streaming job is falling behind. What do you do?

Accepted Answer

Check system lag and data freshness to confirm, then look at the execution graph for the bottleneck stage. Common causes are skew on a key, an expensive per-element operation such as a synchronous external call, insufficient workers or a maximum-worker cap, an undersized sink, or a side input that has grown too large for worker memory. The synchronous external call inside a DoFn is a frequent and fixable cause — batching those calls or caching the results transforms throughput. Enabling Streaming Engine so autoscaling responds properly is the platform-level fix.

Question 24

An engineer accidentally deleted a production Cloud Storage bucket. What now?

Accepted Answer

If soft delete is enabled, restore within the retention window. If not, check whether the bucket name has been reclaimed, look for cross-region or cross-project backups, and check whether any consumer has a cached copy. Then fix the systemic issue: soft delete, versioning, deny policies on delete permissions, and separate backups in another project. The honest part of the answer is that without soft delete or an independent backup the data is gone, and saying so rather than implying magic recovery is correct. The systemic remediation is what the interviewer is really assessing.

Question 25

A Cloud Run service is returning 429 errors. What is happening?

Accepted Answer

The service has hit its maximum instance limit and requests are being rejected, or Cloud Armor rate limiting is throttling the client, or a downstream API is returning 429 and the service is propagating it. Check instance count against the configured maximum first, then Cloud Armor logs, then downstream dependency responses. Hitting the max-instances cap is the most common cause and it is a deliberate configuration, so the fix is deciding whether to raise it or whether the cap is correctly protecting a downstream database. That trade-off is the real answer.

Question 26

A service works for some users and not others, with no regional pattern. What could it be?

Accepted Answer

A partial rollout — a canary revision serving a percentage of traffic; a feature flag enabled for a segment; session affinity pinning some users to a broken instance; a cache serving stale content to some users; data-dependent failure affecting particular accounts; or a client-version difference. Correlate failures by user attribute rather than by infrastructure. The instinct to correlate by user attribute rather than by server is what solves these, and it requires logging user, tenant and version identifiers on every request. That is the observability design decision that makes this debuggable at all.

Question 27

The Terraform plan shows changes you did not make. What does that mean and what do you do?

Accepted Answer

It is drift — someone changed the resource outside Terraform, or another controller manages that attribute. Determine which by checking audit logs for who changed it. If it was an unauthorised manual change, apply to revert and remove the access that allowed it. If another system legitimately owns the attribute, add ignore_changes. Checking audit logs to identify the actor before deciding is the step that distinguishes a considered response from reflexively applying. Blindly reverting a change another controller manages produces a plan that never converges.

Question 28

A scheduled job silently stopped running and nobody noticed for a week. How do you prevent that class of failure?

Accepted Answer

Alert on absence, not just on failure — a heartbeat or dead-man's-switch that fires when an expected completion signal does not arrive within a window. Monitor data freshness downstream so a stale output triggers an alert regardless of why. Track job success as a metric with an expected rate rather than relying on error notifications. Alerting on absence is the key concept, because a job that never starts produces no errors to alert on. Data-freshness monitoring is the complementary control that catches the failure from the consumer's perspective rather than the producer's.

Question 29

A Cloud Function that worked in testing fails in production with permission errors. Why?

Accepted Answer

Locally it ran with your developer credentials, which are far broader than the runtime service account it uses in production. Check the function's service account roles against the APIs it calls, and whether an access scope, VPC Service Controls perimeter or organisation policy is blocking it. The credentials delta between local and deployed is the most common cause of this exact symptom, and naming it immediately shows you understand how serverless identity works rather than only how to write the function.

Question 30

Latency is fine at p50 but terrible at p99. What causes that and how do you find it?

Accepted Answer

Tail latency usually comes from cold starts, garbage collection pauses, lock contention, a slow dependency affecting a subset of requests, cache misses, or one overloaded backend. Find it with distributed tracing filtered to slow traces, and by comparing the span breakdown of a slow request against a fast one. Filtering traces to the slow tail rather than looking at averages is the technique, because the p99 requests are structurally different from the p50 ones. Naming garbage collection and cold starts as specific candidates shows you have chased this before.

Question 31

A deployment is stuck and will not complete on GKE. What do you check?

Accepted Answer

kubectl rollout status and describe the deployment for the reason. Common causes are new pods failing readiness so the rollout never progresses, a PodDisruptionBudget too strict to allow eviction, insufficient cluster capacity for the surge, an image that cannot be pulled, or progressDeadlineSeconds not yet reached. The over-strict PodDisruptionBudget is the non-obvious cause and it blocks node drains and cluster upgrades too. Naming it shows you understand that the same mechanism protecting availability can also prevent all progress.

Question 32

How would you debug a slow API call that passes through five services?

Accepted Answer

Use distributed tracing to see the span breakdown across all five and identify where time is actually spent — that single view usually answers it. If tracing is missing, add correlation identifiers to logs and reconstruct the path, then instrument properly. Check each hop for serial calls that could be parallel and for retries amplifying latency. The requirement that makes tracing work is context propagation through every service, and a broken propagation chain is the usual reason traces are useless when you need them. Retry amplification is the second cause worth naming — three services each retrying three times is nine attempts.

Question 33

A VM is unreachable over SSH. Walk through the diagnosis.

Accepted Answer

Confirm the instance is RUNNING; read the serial console output for boot or disk errors; verify a firewall rule allows port 22 from your source or from IAP's 35.235.240.0/20; check the route to the internet or on-premises; verify OS Login IAM roles or metadata keys; and check whether sshd is running inside the guest, using the serial console to get in if it is not. The serial console is the strongest single move because it is the only way in when the network stack or sshd is broken, and it works when nothing else does. A candidate who names it first has clearly recovered a broken VM before.

Question 34

You suspect a service account has been compromised. What is your first action?

Accepted Answer

Contain — disable the service account or remove its role bindings and disable its keys — then preserve evidence before anything is deleted. Investigate with audit logs to see what it did, from where, and over what period, using Cloud Asset Inventory time travel for state changes. Rotate everything it could reach, then remediate the root cause. Preserving evidence before deleting is the step people skip under pressure, and deleting the service account outright destroys the binding history you need to scope the blast radius. Contain, preserve, investigate, remediate is the order.

Question 35

A GKE node keeps going NotReady. What are the likely causes?

Accepted Answer

Resource pressure — memory or disk exhaustion on the node, often from log files or images filling the boot disk; kubelet crashing; network issues preventing heartbeats; or an underlying VM problem. Check node conditions with kubectl describe node, node logs in Cloud Logging, and disk utilisation. Disk pressure from accumulated images and logs is a common and easily-missed cause, and it manifests as pods being evicted before the node goes fully NotReady. Node auto-repair handles the symptom; finding what fills the disk handles the cause.

Question 36

Your CDN hit rate is very low. What do you investigate?

Accepted Answer

Check origin Cache-Control headers, since Cloud CDN follows them by default and a missing or no-store header prevents caching entirely. Check the cache key for unnecessary query parameters or headers that fragment the cache. Check whether responses vary by cookie, and whether content is being invalidated too frequently. Tracking query parameters in the cache key is the most common specific cause, creating a separate cache entry per user for identical content. Excluding them can transform hit rate, which makes it a satisfying and concrete fix.

Question 37

A BigQuery scheduled query has been failing silently. How would you have caught it?

Accepted Answer

Scheduled queries have limited failure notification, so the robust approach is monitoring the freshness of the output table and alerting when it is stale, plus a log-based alert on job failure events from INFORMATION_SCHEMA.JOBS or Cloud Logging. Better still, move dependent transformations to an orchestrator that alerts on task failure. Monitoring the output rather than the job is the general principle — it catches every reason the data is stale, not just an error the job reported. That reframing applies to any pipeline component and is what the question is teaching.

Question 38

How do you handle an incident where you cannot find the root cause quickly?

Accepted Answer

Restore service by whatever safe means — roll back, fail over, shed load, disable the feature — and continue investigating afterwards with the pressure off. Preserve logs, metrics and any affected state for analysis. Communicate that service is restored while investigation continues, and follow with a blameless postmortem. The priority order is what is being tested: mitigation before understanding. Engineers who keep debugging a live outage because they want to know why are optimising for curiosity over users, and interviewers watch for exactly that.

Question 39

What would you check if a GCP API call intermittently returns 503?

Accepted Answer

Transient service errors are expected in any distributed system, so the first question is whether the client retries with exponential backoff and jitter — most client libraries do, but hand-written HTTP calls often do not. Then check for a quota or rate limit being hit, and check the service health dashboard for an actual incident. The expectation that clients handle transient errors is the substance: a design that treats every 503 as an outage will report constant failures. Naming jitter alongside exponential backoff matters because synchronised retries cause the thundering herd that prolongs the problem.

Question 40

You inherit a GCP project with no documentation. How do you understand what is running?

Accepted Answer

Cloud Asset Inventory to list every resource and its IAM policy; billing export to see what actually costs money, which reveals what matters; Cloud Monitoring for what receives traffic; audit logs for who has been changing things; and network topology in Network Intelligence Center. Then map resources to owners and start documenting. Following the money is the practical shortcut — the expensive resources are almost always the important ones, and idle resources with no traffic are candidates for removal. That heuristic gets you oriented faster than reading configuration.

Question 41

Describe a production incident you handled and what you learned.

Accepted Answer

A strong answer states the symptom and user impact, how you detected it, the diagnostic steps and what you eliminated, the mitigation and its timing, the actual root cause, and — most importantly — the systemic change that prevents recurrence, whether that is a code fix, a guardrail, an alert or a process change. Interviewers are assessing whether you distinguish mitigation from root cause and whether you drive follow-through. An answer ending at "we restarted it and it worked" with no preventive action is the weak version, regardless of how dramatic the incident was.

GCP Troubleshooting & Scenario Questions Interview Questions and Answers

How to use this set