Question 1

What is Cloud Dataflow?

Accepted Answer

Dataflow is a fully managed service for running Apache Beam pipelines, handling both batch and streaming with the same code. It provisions and autoscales workers, rebalances work dynamically, and manages checkpointing and exactly-once state so you write transformation logic rather than cluster management. The unified batch-and-streaming model is the headline: the same pipeline can read a bounded file collection or an unbounded Pub/Sub stream. That is the Beam idea Dataflow implements, and naming Beam as the programming model versus Dataflow as the runner is the distinction interviewers check.

Question 2

What is Apache Beam and what are its core abstractions?

Accepted Answer

Beam is a unified programming model for data processing. Its core abstractions are the Pipeline (the whole job), the PCollection (a distributed, possibly unbounded dataset), the PTransform (an operation applied to a PCollection), and I/O connectors as sources and sinks. Windowing, triggers and watermarks handle time in streaming. The property to state about PCollections is that they are immutable — a transform produces a new PCollection rather than mutating the input, which is what allows the runner to parallelise and retry freely. That immutability is the basis of the whole execution model.

Question 3

What is the difference between a bounded and an unbounded PCollection?

Accepted Answer

A bounded PCollection has a known, finite size — a set of files, a database table — and the pipeline can complete. An unbounded PCollection is a continuous stream with no end, such as Pub/Sub, so the pipeline runs indefinitely and must use windowing to produce results at intervals. The consequence to draw out is that aggregations over an unbounded collection are meaningless without windowing — you cannot sum an infinite stream. That is precisely why windowing exists and why streaming pipelines look different from batch ones.

Question 4

What windowing strategies does Beam support?

Accepted Answer

Fixed (tumbling) windows of equal, non-overlapping duration; sliding windows that overlap so each element can be in several windows; session windows that group activity separated by gaps of inactivity; and the global window, which is the default and holds everything. Session windows are the one worth explaining concretely — grouping a user's clicks into a browsing session separated by 30 minutes of inactivity — because it demonstrates you understand data-driven rather than clock-driven windowing. Sliding windows multiply the data volume by the overlap ratio, which is a cost consideration.

Question 5

What is a watermark in Beam?

Accepted Answer

A watermark is the runner's estimate of how complete the data is for a given event time — effectively "we believe no more data older than time T will arrive". When the watermark passes the end of a window, that window is considered complete and its result is emitted. The watermark is a heuristic, not a guarantee, which is exactly why late data handling exists. Being able to say "the watermark is an estimate and allowed lateness plus triggers decide what happens when it is wrong" is what separates real understanding from repeating the definition.

Question 6

What is the difference between event time and processing time?

Accepted Answer

Event time is when the event actually occurred, carried in the data itself. Processing time is when the pipeline observed it. They diverge because of network delays, mobile devices being offline, and backlogs. Correct analytics almost always requires event time, because processing-time windows produce different results on every replay. The concrete example that lands: a user's phone is offline for three hours, then uploads events. In processing time those events land in today's window; in event time they land in the correct three-hour-old window. Replayability — getting the same answer when you reprocess — is the strongest argument for event time.

Question 7

What are triggers and allowed lateness?

Accepted Answer

A trigger decides when to emit results for a window — by default when the watermark passes the window end, but also early (speculative results before the window closes) and late (updates after the watermark). Allowed lateness defines how long after the watermark the runner keeps window state to accommodate late arrivals; after that, late data is discarded. The trade-off to state is state size versus completeness: a long allowed lateness keeps windows in memory for longer and costs money, while a short one silently drops late data. The accumulation mode — accumulating versus discarding fired panes — is the follow-up that determines whether late updates are deltas or full restatements.

Question 8

What is a side input in Beam?

Accepted Answer

A side input is an additional, usually small, PCollection made available to every worker processing the main PCollection — typically a lookup table or configuration used to enrich each element. It is broadcast rather than shuffled, so it must fit comfortably in worker memory. The failure mode is using a large side input, which causes memory pressure and slow workers. For a large lookup, the right answer is a CoGroupByKey join or an external lookup with caching. Knowing the size boundary is what makes this a practical answer.

Question 9

What is a ParDo and how does it differ from a Map?

Accepted Answer

ParDo is the general parallel-processing transform: it applies a DoFn to each element and can emit zero, one or many outputs, access side inputs, use state and timers, and write to multiple output tags. MapElements is a simpler one-to-one transform. ParDo is the primitive from which most higher-level transforms are built. Multiple output tags are the practical feature to name, because that is how you implement a dead-letter path — valid records to the main output, malformed ones to a side output that lands in a quarantine table instead of failing the pipeline.

Question 10

What is GroupByKey and why is it expensive?

Accepted Answer

GroupByKey collects all values for each key into one collection, which requires a shuffle — redistributing data across workers by key over the network. It is expensive because of that data movement, and in streaming it must wait for a window to close. Combine transforms are cheaper because they aggregate partially on each worker before the shuffle. The optimisation to name is CombinePerKey instead of GroupByKey plus a reduce: the combiner runs before and after the shuffle so far less data crosses the network. That is the single most effective Beam performance change and it is the answer interviewers are looking for.

Question 11

What is data skew in a Dataflow pipeline and how do you fix it?

Accepted Answer

Skew is when one key holds a disproportionate share of the data, so one worker does most of the work while others idle and the job takes far longer than it should. Fixes include salting the key with a random suffix and aggregating in two stages, using a Combine with a hot-key fanout, or filtering the dominant key into a separate path. Beam offers withHotKeyFanout on Combine transforms specifically for this, which is the GCP-flavoured detail. Diagnosing skew from the Dataflow job graph — one stage with a huge gap between median and maximum worker time — is the practical half of the answer.

Question 12

What is Dataflow Streaming Engine?

Accepted Answer

Streaming Engine moves the pipeline's shuffle and state storage off the worker VMs into a Google-managed backend service. That makes workers stateless and much smaller, improves autoscaling responsiveness because there is no state to move when scaling, and reduces the disk attached to each worker. The practical benefit to name is autoscaling: without Streaming Engine, scaling a streaming job means redistributing persistent state across workers, which is slow and disruptive. With it, workers can be added and removed quickly. Dataflow Shuffle is the equivalent for batch.

Question 13

What is Dataflow Prime?

Accepted Answer

Dataflow Prime is a serverless execution platform for Dataflow that adds vertical autoscaling — automatically increasing worker memory when a job is memory-constrained rather than failing — right-fitting, where different stages get different resource shapes, and job visualisation with recommendations. The problem it solves is the traditional one of sizing workers for the most demanding stage and wasting that capacity everywhere else. Vertical autoscaling also removes a common class of out-of-memory failures that previously required a manual restart with a bigger machine type.

Question 14

What is Cloud Dataproc?

Accepted Answer

Dataproc is GCP's managed Hadoop and Spark service. It provisions clusters in about 90 seconds, supports Spark, Hadoop, Hive, Presto, Flink and related tools, integrates with Cloud Storage as the filesystem instead of HDFS, and bills per second so ephemeral job-scoped clusters are practical. The ephemeral cluster pattern is the key idea: instead of a long-running cluster, you create a cluster for a job and delete it afterwards, which is far cheaper and removes the state-management problem. That works because data lives in Cloud Storage rather than HDFS.

Question 15

When would you choose Dataproc over Dataflow?

Accepted Answer

Dataproc when you have existing Spark, Hive or Hadoop code and want to lift and shift without rewriting; when you need a specific library from the Spark ecosystem such as MLlib or GraphFrames; or when the team's expertise is Spark. Dataflow when you are building new pipelines, want a serverless model with no cluster to size, or need sophisticated streaming semantics. The honest framing is migration cost versus operational cost: Dataproc minimises rewriting, Dataflow minimises operating. A migration answer that assumes everything should be rewritten in Beam ignores the real economics of an existing codebase.

Question 16

What is Dataproc Serverless?

Accepted Answer

Dataproc Serverless runs Spark batch workloads and interactive sessions without you creating or sizing a cluster — you submit the job and Google provisions and scales the resources automatically, billing for what the job used. It removes cluster lifecycle management while keeping the Spark API. It closes most of the operational gap between Dataproc and Dataflow while keeping Spark, which makes it the natural landing place for teams with Spark code who do not want to manage clusters. That positioning is the useful part of the answer.

Question 17

Why use Cloud Storage instead of HDFS with Dataproc?

Accepted Answer

Because it decouples storage from compute: data persists independently of any cluster, so clusters become disposable, several clusters can read the same data, and you pay object-storage rates rather than for persistent disks attached to always-on nodes. The connector makes gs:// paths work like HDFS paths. The performance caveat to acknowledge is that object storage has higher per-operation latency than HDFS, so workloads with many small files or heavy random access suffer. Compacting small files into larger columnar files is the standard mitigation and a common interview follow-up.

Question 18

What are Dataflow templates?

Accepted Answer

Templates package a pipeline so it can be launched with parameters and without a development environment — classic templates stage a serialised execution graph, while Flex templates package the pipeline as a container image and build the graph at launch, supporting dynamic parameters. Google provides many pre-built templates for common movements such as Pub/Sub to BigQuery. The operational value is that non-developers or automation can launch pipelines with parameters, and that Flex templates avoid the classic template restriction where parameters could not affect the pipeline structure. Naming a Google-provided template shows practical familiarity.

Question 19

How does Dataflow autoscaling work?

Accepted Answer

Dataflow monitors backlog and CPU utilisation and adjusts worker count within configured bounds. For streaming it responds to the backlog growth rate and processing latency; for batch it uses dynamic work rebalancing, splitting remaining work among workers so stragglers do not extend the job. Streaming Engine makes streaming autoscaling far more responsive. Dynamic work rebalancing is the batch feature to name specifically: it re-splits the work of a slow worker onto idle ones mid-job, which is why Dataflow batch jobs do not suffer the long tail that fixed-partition systems do.

Question 20

How do you handle bad records in a Dataflow pipeline?

Accepted Answer

Use a dead-letter pattern: in the DoFn, catch parsing or validation errors and emit the offending record with its error to a side output tagged separately from the main output, then write that side output to BigQuery or Cloud Storage for inspection. The pipeline continues rather than failing. The alternative — throwing an exception — causes the bundle to be retried, and after repeated failures the whole job fails, so one malformed record can stop a production pipeline. Recognising that a pipeline must be resilient to bad input is the point of the question.

Question 21

What does exactly-once processing mean in Dataflow?

Accepted Answer

Dataflow provides exactly-once semantics for the effect of each element on pipeline state and on supported sinks, by deduplicating records and committing state atomically with checkpoints, even though sources may deliver duplicates and workers may retry. External side effects performed inside a DoFn are not covered. The caveat is the substance: if your DoFn calls a payment API, retries can call it more than once regardless of Dataflow's guarantees. Exactly-once applies to the pipeline's internal state and to sinks it controls, not to arbitrary external calls — that distinction is what interviewers check.

Question 22

What is Cloud Composer?

Accepted Answer

Cloud Composer is managed Apache Airflow. You define workflows as DAGs in Python with dependencies between tasks, and Composer runs the scheduler, workers and web UI on GKE, integrated with GCP IAM, logging and monitoring. It is the standard orchestrator for multi-step data pipelines. The positioning to state is that Composer orchestrates, it does not process — the heavy work is done by Dataflow, Dataproc or BigQuery, and Airflow triggers and monitors those jobs. Using Airflow workers to process large data is a well-known anti-pattern worth calling out.

Question 23

When would you use Cloud Composer versus Workflows versus scheduled queries?

Accepted Answer

Scheduled queries for a single independent SQL statement on a timer. Workflows for a serverless sequence of API and service calls with retries and conditionals, billed per step and with no infrastructure. Composer for complex data pipelines with many dependencies, backfills, sensors, a large operator ecosystem and a team that knows Airflow — accepting that it runs on a cluster you pay for continuously. The cost distinction matters: Composer has a meaningful always-on cost, while Workflows costs almost nothing when idle. For a handful of steps, Composer is over-provisioned, and saying so shows cost awareness rather than tool preference.

Question 24

What is Cloud Data Fusion?

Accepted Answer

Data Fusion is a managed, graphical data integration service built on the open-source CDAP project, letting you build ETL and ELT pipelines by dragging and connecting components, with a large connector library and built-in lineage and metadata. It executes pipelines on Dataproc under the hood. Its audience is data analysts and integration teams who are not writing Beam or Spark code. The trade-offs to name are cost — it runs Dataproc clusters — and that version-controlling and reviewing a visual pipeline is harder than reviewing code.

Question 25

What is a Dataflow shuffle and why does it matter for cost?

Accepted Answer

A shuffle redistributes data across workers by key, which is required for grouping and joining. Dataflow Shuffle moves this operation into a managed backend service rather than on the worker VMs, which improves performance and autoscaling but is billed by the volume of data shuffled. Reducing shuffle volume directly reduces both time and cost. The optimisation to name is filtering and combining before the shuffle rather than after — pushing predicates early and using CombinePerKey means less data crosses the network. That connection between a code choice and a billing line is what an interviewer wants.

Question 26

How do you optimise a slow Dataflow batch job?

Accepted Answer

Inspect the job graph to find the slow stage. Look for skew, excessive shuffle, expensive per-element operations that should be batched, small-file problems on the source, and side inputs that are too large. Use CombinePerKey rather than GroupByKey plus reduce, filter early, choose a suitable machine type, and enable Dataflow Shuffle or Prime. The method matters more than the list: identify the dominant stage from the execution graph before changing anything. Candidates who start by increasing worker count are treating the symptom, and interviewers watch for exactly that.

Question 27

What is the difference between ETL and ELT, and which does GCP favour?

Accepted Answer

ETL transforms data before loading it into the warehouse; ELT loads raw data first and transforms inside the warehouse with SQL. GCP favours ELT for most analytics because BigQuery's compute is elastic and cheap enough to transform at query or scheduled-transform time, which preserves the raw data for reprocessing when requirements change. The argument for ELT is reprocessability: if you transform before loading and later find a bug, the original data is gone. ETL still wins when transformation must happen before storage for compliance reasons, such as tokenising PII before it lands.

Question 28

How do you build a streaming pipeline from Pub/Sub to BigQuery?

Accepted Answer

The simplest path is a BigQuery subscription, which writes directly with no processing. If you need validation, enrichment or aggregation, use Dataflow with the Pub/Sub source and the BigQuery Storage Write API sink, with windowing for aggregates, a dead-letter side output for bad records, and a Google-provided template if the transformation is standard. Naming the direct subscription first shows you do not over-engineer. The criterion for reaching for Dataflow — you need per-element logic, joins with reference data, or windowed aggregation — is what makes the answer decision-oriented.

Question 29

What is the small-files problem and how do you address it?

Accepted Answer

Many small files force a distributed engine to spend most of its time on per-file overhead — listing, opening, and scheduling a task per file — rather than on processing. The fix is compaction into larger files, typically a few hundred megabytes each, in a columnar format such as Parquet, partitioned sensibly by date. It is especially acute on object storage, where each file open is a network request. A compaction step in the pipeline, or writing with a controlled number of output shards rather than letting the writer default, is the practical remedy.

Question 30

What file formats would you use in a GCP data lake and why?

Accepted Answer

Parquet or ORC for analytical data, because they are columnar, compressed and support predicate pushdown so engines read only the columns and row groups they need. Avro for row-oriented streaming and schema-evolution-heavy pipelines. Avoid CSV and JSON for large analytical datasets — they are uncompressed, untyped and force full scans. The Apache Iceberg angle is worth adding: table formats layered over Parquet add ACID semantics, time travel and schema evolution to a data lake, and BigLake supports Iceberg tables. That shows awareness of where lakehouse architecture has moved.

Question 31

How do you handle late-arriving data in a streaming aggregation?

Accepted Answer

Set an allowed lateness appropriate to the source's observed delay, use a trigger that fires on late data, and choose an accumulation mode — accumulating so each firing restates the full window total, or discarding so each firing is a delta. Downstream, write with an upsert keyed on the window so restatements overwrite rather than duplicate. The downstream half is the part candidates forget: emitting a corrected window total is useless if the sink appends it as a second row. Designing the sink for restatement — a MERGE in BigQuery or an upsert — completes the answer.

Question 32

What is Datastream and where does it fit in a data platform?

Accepted Answer

Datastream is serverless change data capture from MySQL, PostgreSQL, Oracle and SQL Server into BigQuery or Cloud Storage. It reads the source database's transaction log rather than querying tables, so it captures inserts, updates and deletes continuously with minimal load on the source. It replaces the nightly full-extract pattern, which is both slow and heavy on the operational database. The detail worth adding is that its BigQuery destination applies changes directly, so you do not have to write merge logic to reconstruct current state.

Question 33

How do you test a Dataflow pipeline?

Accepted Answer

Use Beam's testing utilities: TestPipeline with Create to supply fixed input, PAssert to assert on the output PCollection, and TestStream to simulate event-time advancement, watermarks and late data for streaming logic. Unit test DoFns independently, then run integration tests against a small real dataset with the DirectRunner before deploying to Dataflow. TestStream is the piece that distinguishes a real answer, because it is the only way to test windowing and late-data behaviour deterministically. Without it, streaming logic is effectively untested until production surprises you.

Question 34

What is the difference between Dataflow FlexRS and standard workers?

Accepted Answer

Flexible Resource Scheduling schedules a batch job within a six-hour window using a mix of preemptible and standard VMs, at a significantly reduced price. The job may not start immediately, so it suits non-urgent batch work with a tolerant deadline. The precondition is that the job must be delay-tolerant, which excludes anything on a tight SLA. Naming FlexRS alongside Spot VMs for Dataproc shows you know the cost levers specific to data processing rather than generic advice.

Question 35

How would you migrate an on-premises Hadoop estate to GCP?

Accepted Answer

Assess workloads and dependencies first. Move data to Cloud Storage, replacing HDFS paths with gs:// through the connector. Move Hive metastore to Dataproc Metastore or BigQuery. Run existing Spark and Hive jobs on ephemeral Dataproc clusters with minimal change, then incrementally modernise — moving SQL workloads to BigQuery and stream processing to Dataflow — rather than rewriting everything at once. The sequencing is the answer: lift and shift onto Dataproc first to get off the on-premises hardware, then modernise selectively where it pays. Attempting a full rewrite to BigQuery and Dataflow as the migration is how these projects run years over schedule.

Question 36

What monitoring would you put on a production Dataflow streaming pipeline?

Accepted Answer

System lag and data freshness as the primary health signals, backlog size on the Pub/Sub source, worker CPU and memory, the count of records written to the dead-letter output, and element throughput per stage. Alert on data freshness exceeding the business tolerance rather than on worker metrics alone. Data freshness is the metric that maps to the business requirement — "our dashboard is at most five minutes stale" — while CPU tells you nothing a stakeholder cares about. Framing alerts around the user-visible property is what makes this a strong operational answer.

Question 37

What is a Combine transform and when should you use it over GroupByKey?

Accepted Answer

A Combine applies an associative, commutative function to aggregate values — sum, max, mean, or a custom CombineFn. Use it whenever the aggregation can be computed incrementally, because the runner applies it partially on each worker before the shuffle and again after, drastically reducing shuffled data compared with GroupByKey followed by a reduce. The associativity and commutativity requirement is what makes partial aggregation valid, and stating it shows you understand why the optimisation is possible rather than just that it exists. A custom CombineFn with an accumulator is the answer for non-trivial aggregates.

Question 38

What are stateful DoFns and timers in Beam?

Accepted Answer

A stateful DoFn keeps per-key, per-window state across elements, and timers let it schedule callbacks in event time or processing time. Together they enable custom logic that windowing cannot express — deduplication over a long horizon, detecting the absence of an expected event, or building a custom session model. The example that demonstrates understanding is alerting when an expected heartbeat does not arrive: you cannot detect a non-event with a normal transform, but a timer set on each heartbeat and reset on the next one can. State size and cleanup are the operational concerns to mention.

Question 39

How do you control Dataflow costs?

Accepted Answer

Reduce shuffle volume by filtering and combining early; right-size machine types rather than defaulting; use FlexRS or Spot-backed Dataproc for delay-tolerant batch; enable Streaming Engine and Dataflow Prime so workers are smaller and scale better; set maximum worker counts; avoid unnecessary streaming where micro-batch would do; and compact small files at the source. The largest single lever is usually choosing batch over streaming when the business does not actually need sub-minute freshness, because a streaming pipeline runs continuously while a batch job runs for minutes. Challenging the freshness requirement is a legitimate and valuable engineering move.

Question 40

Design a real-time analytics pipeline for clickstream data on GCP.

Accepted Answer

Clients publish events to Pub/Sub with an event-time field and an ordering key where needed. Dataflow reads the stream, validates and enriches with a side input of reference data, applies session and fixed windows for aggregates, routes malformed records to a dead-letter table, and writes raw events and aggregates to BigQuery with the Storage Write API. Bigtable serves low-latency per-user lookups for personalisation. Composer orchestrates daily backfills and reconciliation, and dashboards read materialised views rather than raw tables. The closing scenario. The senior markers are separating raw and aggregate layers, handling late data explicitly with allowed lateness and upserts, providing a dead-letter path, and adding a reconciliation job — because a streaming pipeline without a batch reconciliation eventually drifts and nobody notices.

GCP Dataflow, Dataproc & Data Processing Interview Questions and Answers

How to use this set