Question 1

What is the Google Cloud Operations suite?

Accepted Answer

It is the observability stack formerly called Stackdriver: Cloud Monitoring for metrics, dashboards, uptime checks and alerting; Cloud Logging for log ingestion, search, sinks and log-based metrics; Cloud Trace for distributed tracing; Cloud Profiler for continuous CPU and memory profiling; and Error Reporting for grouped exception tracking. Naming all five and what each is *for* is the answer. The connection worth drawing is that they share resource labels, so a metric, a log entry and a trace for the same Cloud Run revision can be correlated automatically — that correlation is the value of an integrated suite.

Question 2

What is the difference between metrics, logs and traces?

Accepted Answer

Metrics are numeric time series — cheap to store, good for trends, dashboards and alerting, but they cannot tell you about a specific request. Logs are discrete records with detail, good for investigating a specific event, but expensive at volume. Traces follow a single request across services, showing where time was spent. The way to use them together is the substance: alert on metrics because they are cheap and aggregate, use traces to find which service is slow, then use logs to find out why. Candidates who reach for logs first for every problem are describing an expensive and slow workflow.

Question 3

What is an SLI, an SLO and an SLA?

Accepted Answer

A service level indicator is a measurement of behaviour — request success rate, latency at the 95th percentile. A service level objective is a target for that indicator over a window, such as 99.9% of requests succeeding over 28 days. A service level agreement is a contract with consequences, usually financial, and it is normally looser than the internal SLO. The relationship to state is that the SLO should be stricter than the SLA, so you detect and fix degradation before you breach a contract. The other point is that an SLI must be measured from the user's perspective — server-side success rate misses failures that never reached your server.

Question 4

What is an error budget and how do you use it?

Accepted Answer

The error budget is the acceptable amount of unreliability implied by the SLO — a 99.9% target over 30 days allows about 43 minutes of failure. It is used as a decision-making tool: while budget remains, the team can ship features and take risk; when it is exhausted, the priority shifts to reliability work until the budget recovers. The organisational point is what interviewers actually want: the error budget converts an argument about "are we moving too fast" into a measurable, agreed policy. Without a pre-agreed consequence for exhausting it, the budget is just a number on a dashboard.

Question 5

What is burn rate alerting and why is it better than threshold alerting?

Accepted Answer

Burn rate measures how fast you are consuming the error budget relative to the rate that would exhaust it exactly at the window's end. Alerting on a fast burn over a short window catches sudden outages quickly, while a slower burn over a longer window catches gradual degradation, and combining both gives high precision with low false-positive rates. The advantage over a static threshold — "alert if error rate is above 1%" — is that burn rate accounts for both severity and duration, so a brief 5% spike does not page anyone but a sustained 2% does. Multi-window multi-burn-rate alerting is the standard SRE pattern to name.

Question 6

What makes a good alert?

Accepted Answer

It should be actionable, urgent and user-visible: it fires only when a human needs to do something now, about something that is affecting users, with a runbook link and enough context to start. Anything that is informative but not urgent belongs on a dashboard or a ticket, not a page. The anti-pattern to name is alerting on causes rather than symptoms — paging on high CPU when users are unaffected. Symptom-based alerting on SLIs is what keeps the pager quiet and trusted, and alert fatigue from noisy causes is the reason real incidents get missed.

Question 7

What is a log sink in Cloud Logging?

Accepted Answer

A sink routes matching log entries to a destination — Cloud Storage for cheap archival, BigQuery for analysis, Pub/Sub for streaming to a SIEM or third-party tool, or another log bucket. Sinks use an inclusion filter and optional exclusion filters, and can be created at project, folder or organisation level with an aggregated sink. The aggregated organisation-level sink is the enterprise pattern: one sink captures audit logs from every project into a locked logging project, rather than configuring hundreds of per-project exports that drift. Naming it shows you have thought about scale.

Question 8

What is a log-based metric?

Accepted Answer

A log-based metric extracts a numeric signal from log entries — a counter of entries matching a filter, or a distribution built from a numeric field in the log. It lets you alert on something that only appears in logs, such as a specific application error string, using the cheaper metrics pipeline. The design advice is to log structured JSON so fields can be extracted reliably, rather than parsing free-text with regular expressions that break when a message changes. Structured logging is the enabling practice behind most log-based metrics working well.

Question 9

How do you control Cloud Logging costs?

Accepted Answer

Exclusion filters on high-volume, low-value logs such as health-check requests and successful load-balancer entries; shorter retention on the default bucket with longer retention only for what you must keep; routing archival copies to Cloud Storage, which is far cheaper than log storage; and being selective about enabling Data Access audit logs and VPC Flow Logs, which are the usual volume drivers. The point to make is that ingestion is where the cost is, so an exclusion filter saves money while a retention change saves less. Excluding successful health checks alone often removes a large share of volume in a Kubernetes environment.

Question 10

What is Cloud Trace and what is a span?

Accepted Answer

Cloud Trace collects latency data for requests as they move through services. A trace is the whole request; a span is one timed operation within it — an HTTP call, a database query — with a parent-child relationship forming a tree. Together they show exactly where a slow request spent its time. The practical requirement is context propagation: services must pass the trace context header downstream or the trace breaks into disconnected fragments. OpenTelemetry is the standard instrumentation to name, and knowing that propagation is the hard part shows real tracing experience.

Question 11

What is Cloud Profiler?

Accepted Answer

Cloud Profiler continuously samples CPU time and memory allocation in production with very low overhead, and presents flame graphs so you can see which functions consume resources. Because it runs continuously in production, it reveals the real hot paths rather than the ones a synthetic benchmark exercises. The value over local profiling is representativeness: production traffic patterns, data sizes and cache states differ from a developer machine. It is the tool for "our service is expensive and we do not know why", which is a cost question as much as a performance one.

Question 12

What is an uptime check?

Accepted Answer

An uptime check probes an endpoint from several geographic locations at an interval, verifying status code, response content and latency, and feeding a metric you can alert on. It provides external, black-box monitoring that catches failures your internal metrics cannot see — DNS, certificate expiry, load-balancer misconfiguration. The reason it matters is that internal metrics are collected by the very system that may be broken. An uptime check is independent, which is exactly why it catches the class of outage where the service is fine but nobody can reach it.

Question 13

What is Google Cloud Managed Service for Prometheus?

Accepted Answer

It collects Prometheus-format metrics at scale without you operating Prometheus servers, storing them in Monarch — the same backend as Cloud Monitoring — with global querying via PromQL, long retention and no sharding or federation to manage. The problem it solves is the operational burden of self-managed Prometheus at scale: retention, high availability, sharding and federation are genuinely difficult. Keeping PromQL and existing exporters and dashboards while removing that burden is the value proposition.

Question 14

What is a golden signal and what are the four?

Accepted Answer

The four golden signals from the SRE book are latency, traffic, errors and saturation. Latency is how long requests take, traffic is demand, errors is the rate of failed requests, and saturation is how full the most constrained resource is. Monitoring these four covers most user-visible failure modes. The point about latency worth adding is to measure successful and failed requests separately, because fast failures can make average latency look excellent during an outage. That distinction is a classic SRE detail interviewers listen for.

Question 15

Why should you alert on percentiles rather than averages?

Accepted Answer

An average hides the tail. If 99% of requests take 50 milliseconds and 1% take 10 seconds, the average looks fine while one user in a hundred has a terrible experience. Percentiles — p50, p95, p99 — show the distribution, and the tail is where user-visible problems live. The extension worth making is that in a system where one page makes twenty backend calls, a p99 backend latency affects a large share of *pages*. That fan-out amplification is why tail latency matters more than it first appears.

Question 16

What is a dashboard versus an alerting policy in Cloud Monitoring?

Accepted Answer

A dashboard is for investigation and situational awareness — a human looking at charts. An alerting policy is a condition evaluated continuously that notifies a channel when met. Dashboards answer "what is happening"; alerts answer "does someone need to act now". The discipline to state is that not everything on a dashboard deserves an alert, and everything that alerts should have a dashboard and a runbook. A team with fifty alerts and no runbooks has a pager that nobody trusts.

Question 17

What is a notification channel and what escalation would you configure?

Accepted Answer

A notification channel is where an alert is delivered — email, SMS, Slack, PagerDuty, webhook, or Pub/Sub for automation. A sensible escalation sends urgent, user-affecting alerts to a paging service with an on-call rotation, and non-urgent ones to a chat channel or ticket queue, with a defined escalation path if unacknowledged. The Pub/Sub channel is the interesting one because it enables automated remediation: an alert can trigger a function that restarts a component or scales a resource. Naming auto-remediation with a safety limit shows maturity beyond notification.

Question 18

How do you monitor a GKE cluster effectively?

Accepted Answer

Use the built-in GKE monitoring for system and workload metrics plus Managed Service for Prometheus for application metrics. Alert on pending pods, node not-ready, container restart rate, memory working set approaching limits, and SLO burn rate for the services. Dashboards should be per-service rather than per-node, because in Kubernetes nodes are fungible and services are what users care about. The per-service framing is the insight: alerting that a node went away is noise if the workload rescheduled successfully. Alerting when a *service* degrades is what matters, and that reframing is what distinguishes a Kubernetes-native monitoring approach.

Question 19

What is toil and how do you reduce it?

Accepted Answer

Toil is manual, repetitive, automatable work that scales linearly with the size of the service and produces no lasting value — restarting a stuck process, manually approving a routine change, copying data by hand. You reduce it by automating the task, removing the need for it, or changing the system so it self-heals. The SRE convention worth citing is capping toil at roughly half of an SRE's time, with the rest spent on engineering that reduces future toil. Being able to name that boundary shows familiarity with the actual discipline rather than the job title.

Question 20

What is a blameless postmortem and why does it matter?

Accepted Answer

A postmortem that focuses on the systemic causes of an incident — what made the failure possible and why it was not caught — rather than on who made a mistake. It matters because blame suppresses information: people who fear consequences do not report near-misses, and you lose the data that prevents the next outage. The concrete practice is asking "what about the system allowed this action to cause an outage" instead of "why did you do that". Naming action items with owners and deadlines is the other half, because a postmortem without follow-through is a document nobody reads twice.

Question 21

What is MTTR and what reduces it?

Accepted Answer

Mean time to recovery — how long from the start of an incident to service restoration. It is reduced by good detection (fast, accurate alerting), good diagnosis (dashboards, tracing, structured logs, runbooks), and fast, safe remediation (one-command rollback, traffic shifting, feature flags, tested failover). The lever most under-invested in is fast rollback: if reverting takes forty minutes, every incident has a forty-minute floor. Investing in progressive delivery and instant rollback often reduces MTTR more than better monitoring does.

Question 22

How would you structure logs for a production service on GCP?

Accepted Answer

Emit structured JSON to stdout, which Cloud Logging parses automatically on Cloud Run, GKE and Cloud Functions. Include a severity field, a trace identifier for correlation, and consistent business fields such as tenant and request identifiers. Never log secrets or personal data, and use log levels deliberately so debug volume can be turned down. The trace correlation field is the highest-value detail: it lets you pivot from a slow trace directly to the log lines for that request. Setting the logging.googleapis.com/trace field is the specific mechanism on GCP, and knowing it is a strong practical signal.

Question 23

What is the difference between white-box and black-box monitoring?

Accepted Answer

White-box monitoring uses internal signals the system exposes — queue depth, garbage collection time, connection pool utilisation — and is good for diagnosis and prediction. Black-box monitoring probes from outside as a user would and is good for detecting that something is actually broken from the user's perspective. The rule to give is to page on black-box and symptom-based signals, and to use white-box signals for diagnosis and for capacity prediction. Paging on white-box internals is how teams end up with alerts that fire while users are perfectly happy.

Question 24

How do you set an SLO for a new service?

Accepted Answer

Start from what users actually need rather than from what the system currently does, but validate it against historical data so the target is achievable. Pick SLIs measured at the user boundary — request success rate and latency at a percentile — choose a rolling window, define the error budget, and agree in advance what happens when it is exhausted. The mistake to avoid is setting an SLO of 99.99% because it sounds good: each additional nine costs disproportionately more, and an unachievable target trains everyone to ignore the error budget. Choosing a target you will actually enforce is more valuable than a strict one you will not.

Question 25

What is Cloud Monitoring's metric scope / metrics scope project?

Accepted Answer

A metrics scope lets one project view metrics from several projects, giving a single pane of glass across an estate without duplicating dashboards. The scoping project holds the dashboards and alerting policies while the monitored projects contribute metrics. It is the practical answer to "we have forty projects, how do we monitor them together" — otherwise you would be switching context per project. Setting it up early in a multi-project landing zone avoids retrofitting dashboards later.

Question 26

What is a custom metric and when would you create one?

Accepted Answer

A custom metric is application-defined data written to Cloud Monitoring — queue depth, business events per minute, cache hit ratio — that no platform metric exposes. Create one when you need to alert or autoscale on something specific to your domain, such as the number of unprocessed orders. The autoscaling connection is the strongest use case: scaling a worker fleet on queue depth is far more responsive than scaling on CPU, because queue depth rises before CPU does. Naming that use case makes the answer concrete.

Question 27

What is Error Reporting and how does it differ from searching logs?

Accepted Answer

Error Reporting automatically groups exceptions by stack trace fingerprint across services and versions, showing occurrence counts, first and last seen, affected versions and a trend. Searching logs finds individual entries; Error Reporting tells you which distinct error is new, which is growing, and which was introduced by the latest release. The deployment workflow it enables is the point: after a release, you look at new error groups rather than at raw log volume. That turns "are there errors" — there always are — into "are there errors we have not seen before".

Question 28

How would you investigate a latency regression after a deployment?

Accepted Answer

Compare latency percentiles before and after in Cloud Monitoring, split by revision or version label. Use Cloud Trace to compare span breakdowns between the two versions and find which operation grew. Check Error Reporting for new error groups, Profiler for a change in CPU hot paths, and the dependency metrics in case the cause is downstream rather than in the deployed code. The version-label split is the key technique: without labelling metrics by revision, before-and-after comparison is guesswork. On Cloud Run and GKE that labelling is automatic, which is a good reason to name those platforms specifically.

Question 29

What is the difference between availability and reliability?

Accepted Answer

Availability is the proportion of time or requests the service is usable. Reliability is broader — the service does the right thing correctly and consistently. A service can be 100% available while returning wrong answers, which is reliable-sounding and useless. The practical extension is that SLIs should include correctness where it matters — data freshness, completeness, or a quality metric — not only success rate. A data pipeline that runs successfully but produces stale numbers is available and unreliable.

Question 30

How do you monitor a data pipeline as opposed to a request-serving service?

Accepted Answer

The user-facing property is freshness and completeness rather than latency and error rate. Monitor data freshness — how old the newest processed record is — record counts against expected volume, dead-letter and rejected-record counts, and reconciliation between source and destination totals. Alert on freshness exceeding the business tolerance. The reconciliation check is what catches silent data loss, which is the characteristic failure of pipelines: everything reports success while records are quietly dropped. Naming reconciliation separates data-platform experience from generic monitoring knowledge.

Question 31

What is on-call and what makes a healthy rotation?

Accepted Answer

A rotation where engineers take responsibility for responding to pages, with enough people that the load is sustainable, a documented handover, runbooks for every alert, an escalation path, compensation or time off in lieu, and a feedback loop so noisy alerts are fixed rather than tolerated. The measurable health signal is pages per shift — a rotation averaging more than a couple of actionable pages per shift is unsustainable and will burn people out. Tracking that number and treating it as a bug when it rises is what a mature team does.

Question 32

What is chaos engineering and would you use it on GCP?

Accepted Answer

Deliberately injecting failure — terminating instances, adding latency, failing a zone — into a system to verify that resilience mechanisms work before a real incident tests them. On GCP you might drain a zone, delete a MIG instance, or simulate a Cloud SQL failover, ideally starting in a staging environment with a clear hypothesis and a stop condition. The precondition to state is that you need good monitoring and a rollback path first, otherwise you are just causing outages. Starting with a documented hypothesis — "we believe traffic will shift within 30 seconds" — is what makes it an experiment rather than vandalism.

Question 33

How do you handle alert fatigue?

Accepted Answer

Audit every alert against whether it is actionable and user-affecting; delete or downgrade those that are not. Move cause-based alerts to dashboards and keep symptom-based ones. Add or tune thresholds using burn-rate windows. Track pages per shift and treat a noisy alert as a defect with an owner, not an inevitability. The cultural point is that tolerating noisy alerts is itself the risk: once responders assume a page is probably noise, the real incident is answered slowly. Framing alert quality as a reliability control rather than a convenience is the mature position.

Question 34

What is an incident severity classification and why have one?

Accepted Answer

A defined scale — for example SEV1 for total outage or data loss, SEV2 for major degradation, SEV3 for minor or partial impact — that determines who is notified, how quickly, whether an incident commander is appointed, and whether a postmortem is required. It removes negotiation during the incident. The benefit is speed under pressure: nobody argues about whether to wake the database team when the criteria are written down. Defining the roles — incident commander, communications lead, operations lead — is the other half that prevents the common failure of everyone debugging and nobody coordinating.

Question 35

What is the difference between monitoring and observability?

Accepted Answer

Monitoring is watching known signals for known failure modes — you decide in advance what to measure and alert on. Observability is the property of being able to answer questions you did not anticipate, from the data the system already emits, without shipping new code. High-cardinality structured events and traces enable it; a fixed set of pre-aggregated metrics does not. The practical consequence is designing for unknown-unknowns: log a request identifier, tenant, region and version on every event so you can slice by any of them later. A system where every investigation requires a new deployment to add a log line has monitoring but not observability.

Question 36

How do you correlate logs, metrics and traces for one request on GCP?

Accepted Answer

Propagate a trace context through every service and include the trace identifier in structured log entries using the logging.googleapis.com/trace field. Cloud Logging then links the entry to the trace, the Trace UI shows associated logs for each span, and resource labels tie both to the metrics for that service and revision. The instrumentation to name is OpenTelemetry, which handles propagation and exports to all three backends. Without propagation the three data sources remain unlinked, and correlating by timestamp alone is unreliable under load — which is precisely when you need it.

Question 37

What would you put on a service dashboard?

Accepted Answer

The SLIs at the top — success rate and latency percentiles with the SLO target and remaining error budget — then traffic volume, error breakdown by type, saturation of the constrained resource, dependency health and latency, and deployment markers so you can see which change coincided with a change in the graphs. Deployment markers are the underrated element: the majority of incidents follow a change, and a dashboard that overlays deploy times answers "what happened" in seconds. Ordering the dashboard by user impact first is the other principle worth stating.

Question 38

How do you monitor cost as a reliability concern?

Accepted Answer

Export billing to BigQuery, build dashboards by project, service and label, set budgets with alerts at percentage thresholds, and alert on anomalous daily spend rather than only on monthly totals. Treat a runaway cost event — a recursive function, an unbounded autoscale — as an incident with the same urgency as an outage. Framing cost as an incident class is the insight: a misconfigured pipeline can spend a quarter's budget in a weekend, and monthly budget alerts arrive far too late. Daily anomaly detection is the control that actually catches it.

Question 39

What is a runbook and what should it contain?

Accepted Answer

A runbook is the documented response for a specific alert: what the alert means, what user impact it implies, the first diagnostic steps with the exact queries or commands, known causes and their remedies, escalation contacts, and how to verify recovery. It should be linked directly from the alert. The quality test is whether someone unfamiliar with the service can follow it at 3am. Runbooks that say "investigate the issue" are decorative; ones with copy-pasteable queries and decision points are what actually reduce MTTR.

Question 40

Design the observability strategy for a platform of 40 microservices on GCP.

Accepted Answer

Standardise instrumentation with OpenTelemetry exporting metrics to Managed Service for Prometheus, traces to Cloud Trace and structured logs to Cloud Logging with trace correlation. Define SLIs and SLOs per user-facing service with error budgets and multi-window burn-rate alerting, paging only on symptoms. Provide a templated per-service dashboard with SLIs, traffic, errors, saturation and deploy markers, plus a metrics scope for the whole estate. Route audit and security logs through an aggregated organisation sink to a locked project, exclude health-check noise, and archive to Cloud Storage. Maintain runbooks linked from every alert, a severity model with defined incident roles, and blameless postmortems with tracked action items. The closing scenario. What marks it senior is standardising instrumentation so every service is observable the same way, paging only on user-visible symptoms, and treating the human process — severity, roles, postmortems — as part of the observability strategy rather than an afterthought.

GCP Cloud Monitoring, Logging & SRE Interview Questions and Answers

How to use this set