1h 2m read

Unified Batch and Streaming Lakehouse in One Engine

A vendor-neutral guide to evaluating single-engine batch and streaming lakehouse claims: the four layers vendors conflate, six convergence mechanisms, co-location tests, and a runnable acceptance checklist.

Unified Batch and Streaming Lakehouse in One Engine

Most data platforms that claim to handle batch and streaming still run two stacks underneath: one engine for historical processing, another for real-time, two codebases expressing the same business logic, and eventually two versions of the truth that someone has to reconcile. "One engine" is usually offered as the fix, but the phrase is rarely defined precisely enough to evaluate.

This guide treats single-engine operation as a testable architectural property rather than a marketing claim. It separates the four layers that vendors routinely conflate — engine, platform, catalog and table format — so you can tell which layer a given convergence claim actually lives in. It then works through the six mechanisms that genuine convergence requires, gives you a way to distinguish co-location from real convergence, and closes with a runnable acceptance checklist you can carry into an evaluation.

Scope: batch and streaming convergence at the engine layer, with table format and catalog treated as supporting layers. Capabilities in this space change quickly, so treat the vendor documentation you consult as dated material and re-verify it before you commit. This guide was refreshed in November 2026.

What "One Engine" Actually Means: Engine, Platform, Catalog and Table Format

Before any vendor claim can be evaluated, the reader needs a precise four-layer distinction. Each layer owns something different, and convergence can happen at any one of them — but only engine-layer convergence removes duplicated logic.

The engine owns execution and semantics. It parses a query, plans it, executes it and defines what the result means: how nulls compare, how time is handled, what a window boundary includes, what a failure guarantees. When people say two systems have "different semantics," they are describing two engines, regardless of what console they sit behind.

The platform owns packaging, scheduling and operations. It handles how jobs are submitted, how clusters are provisioned, how credentials are issued, how metrics are surfaced and how failures are alerted. A platform can present a single console over several engines without changing any of those engines.

The catalog owns naming and discovery. It maps a table name to a location, tracks schema, and increasingly holds the permission model. A shared catalog means both modes see the same namespace; it does not mean they execute the same way.

The table format owns storage layout and the commit protocol. This is the layer most often mistaken for the engine. Apache Iceberg describes itself as "an open table format for huge analytic datasets" that "adds tables to compute engines including Spark, Trino, PrestoDB, Flink, Hive and Impala" — the format supplies tables to engines, it does not supply an engine. Delta Lake's documentation similarly describes ACID transaction guarantees between reads and writes and an optimistic concurrency protocol in which a writer validates its staged changes against concurrently committed ones before committing a new versioned snapshot. Both are storage contracts: they define how files are laid out and how a commit becomes visible.

That distinction matters because a shared format is genuinely valuable — it lets many engines read the same data consistently — while leaving every one of those engines with its own planner, runtime and semantics. Convergence at the format layer reduces storage duplication. Only convergence at the engine layer reduces logic duplication.

Why Separate Batch and Streaming Stacks Produce Duplicated Logic and Semantic Drift

The failure mode is well documented in the architecture literature, and it is worth naming precisely because teams often feel its symptoms without being able to point at the cause.

Jay Kreps's 2014 critique of the Lambda architecture describes the mechanism directly: an immutable sequence of records is fed into a batch system and a stream processing system in parallel, transformation logic is implemented twice, and results from both systems are stitched together at query time. His assessment is blunt — maintaining code that must produce the same result in two complex distributed systems is "exactly as painful as it seems like it would be," and code inevitably ends up engineered toward the framework it runs on. Ericsson's account of the Kappa architecture names the same motivation from the other direction: one of the important reasons for inventing Kappa was to avoid maintaining two separate codebases for the batch and speed layers, handling real-time processing and reprocessing with a single stream processing engine.

Duplicate code is the visible cost. The expensive part is the drift that follows, and it shows up in specific places you can inspect:

  • Watermark and late-data handling. How late is too late, and what happens to a record that arrives after the threshold, is a policy each stack implements separately. The watermark literature treats this as a first-class semantic decision, not an implementation detail.

  • Exactly-once boundaries. Where the guarantee starts and stops differs by system and by connector. Spark's documentation, for example, notes that output operations carry at-least-once semantics by default and that exactly-once requires idempotent or transactional sinks.

  • Type and null semantics. Two engines can disagree on coercion, null ordering and time zone handling without either being wrong.

  • Reconciliation logic. The code that stitches batch and streaming results together is rarely designed as a system. It accretes, and it becomes an undocumented dependency that only its author understands.

A useful diagnostic: ask which engineer can explain, without reading code, why a given number differs between the batch and streaming paths. If the answer requires tracing two implementations, the drift is already structural.

The Six Convergence Mechanisms and What Each One Technically Requires

Convergence is not one capability. It is six, each with its own prerequisites, and satisfying one tells you nothing about the others. A vendor can have a unified API and separate runtimes, or one runtime and two state models. Evaluate each mechanism on its own evidence.

Spark Structured Streaming: documented latency floor by execution mode Horizontal bar chart comparing the documented best-case end-to-end latency floor of Spark's default micro-batch engine (about 100 milliseconds, exactly-once) against its Continuous Processing mode (about 1 millisecond, at-least-once). Both are vendor-stated floors, not measured results for any particular workload. Spark Structured Streaming: documented latency floor by execution mode. horizontal bar data: Micro-batch (default) — exactly-once, ~100 ms 100; Continuous Processing — at-least-once, ~1 ms 1.Source: Apache Spark, Structured Streaming Programming Guide (Overview) . Spark Structured Streaming:documented latency floor byexecution mode Micro-batch(default) —exactly-once,... 100 ContinuousProcessing —at-least-once,... 1 Source: Apache Spark, Structured StreamingProgramming Guide (Overview)

Latency floors and delivery guarantees documented for Spark Structured Streaming's two execution modes. These are the vendor's stated best-case figures for each mode, not a ranking: the modes trade latency against guarantee strength. Values are as published in the Spark documentation; actual latency depends on workload, cluster and configuration.

1. A unified programming model. The same query should express bounded and unbounded input. Spark's Structured Streaming guide states the model plainly: treat a live data stream as a table that is being continuously appended, and express the streaming computation as a standard batch-like query on a static table. The guide also notes that DataFrames and Datasets can represent both static, bounded data and streaming, unbounded data. What to verify: whether the operations you actually use — your joins, aggregations and stateful operators — are supported in both modes, since Spark's own documentation lists operations that are not supported on streaming Datasets, including limit, distinct, and sorting outside an aggregation in complete mode.

2. A single execution engine underneath both modes. Flink's execution mode documentation is the clearest public example of one runtime serving both: BATCH execution is enabled for bounded inputs, and because the input is known to be bounded, Flink can apply additional optimizations that are unavailable in STREAMING mode. The runtime is shared; the optimization strategy differs by boundedness. What to verify: that the batch and streaming paths produce plans from the same planner, not from two planners that happen to accept the same SQL.

3. Unified state and checkpointing semantics. State handling is where guarantees are actually made or lost. Spark's Structured Streaming guide documents that its default micro-batch engine achieves end-to-end latencies as low as 100 milliseconds with exactly-once fault tolerance, while its Continuous Processing mode reaches latencies as low as 1 millisecond with at-least-once guarantees — a trade the documentation states explicitly. Flink's stateful stream processing documentation describes checkpointing as the mechanism for restoring state and consistency. What to verify: one checkpoint model, and a documented guarantee per execution mode rather than a single guarantee quoted for the product.

4. A shared table format with a commit protocol that makes streaming writes and concurrent batch reads consistent. Iceberg's specification describes serializable isolation, with readers isolated from concurrent writes and always using a committed snapshot, and writes adding and removing files in a single operation that is never partially visible. Delta's documentation makes the equivalent commitment for its tables. What to verify: that a streaming writer and a batch backfill can target the same table under one concurrency protocol, and what the documented failure behavior is when they conflict.

5. A shared catalog and metadata layer. One namespace, one schema registry, one place where a table's identity is resolved. What to verify: that a table created by a streaming job is immediately discoverable and queryable by a batch job without a registration step.

6. Shared governance, security and lineage. Permissions, masking and lineage should apply to the table, not to the engine that touches it. What to verify: that a policy defined once is enforced identically whether the read is batch or streaming, and that lineage captures both paths into the same graph.

How to Tell Co-Location on One Platform from Genuine Single-Engine Operation

Co-location is a real convenience: separate engines packaged under one console, one bill and one catalog. It reduces procurement friction and vendor sprawl. It does not reduce duplicated logic, because two execution paths, two query planners and two sets of semantics remain underneath.

Genuine single-engine operation means one planner, one runtime and one semantic contract serving both bounded and unbounded input. The distinction is architectural, and you can reason about it from observable signals rather than from a vendor's description:

  • Plan shape. Run the same SQL or DataFrame against a bounded source and an unbounded one. In a single-engine system, the plans should share structure and differ only where boundedness legitimately changes the strategy — as Flink's BATCH mode does. Two unrelated plan shapes suggest two planners.

  • State and checkpoint mechanisms. Look for one checkpoint format and one recovery procedure. Parallel, separately configured state backends are the signature of two runtimes.

  • Concurrent writes to one table. Attempt a batch backfill and a streaming write against the same table and observe whether one concurrency protocol governs both. Delta's documentation, for instance, warns that two streaming queries should never share the same checkpoint location and run simultaneously — a constraint that belongs to the commit protocol, and one you should be able to read about rather than discover.

  • Operational tooling. Metrics, failure recovery and resource management should be one system. If batch jobs and streaming jobs report to different metric stores with different alerting rules, you are operating two systems.

None of these signals accuses any particular product of anything. They are questions about architecture, and the answers should be available in documentation or observable in a trial.

How to Validate Vendor Performance and Convergence Claims

Headline speedup figures do not transfer between environments. Benchmark workload, data shape, concurrency, hardware and configuration all move the number, and vendor-published pages typically omit enough of the setup that the figure cannot be reproduced from the page alone.

Onehouse's 2026 survey of Spark accelerators illustrates the discipline even from a vendor with its own product in the comparison. The article notes that two of the engines it covers "publish headline figures without a full benchmark setup on those pages" and advises choosing one only if you "will validate the claims against your own jobs." It also separates a 1 TB TPC-DS result validated by NetApp in a specific lab from the 10 TB TPC-DS benchmarks published on its own site, explicitly noting that the smaller workload is a different scale and environment and does not reproduce the comparison. That is the standard to hold every vendor to, including any lakehouse vendor publishing figures about itself.

A workable validation protocol:

  1. Reproduce on your own data. Use a representative slice of your largest and most skewed tables, not a synthetic benchmark whose shape differs from yours.

  2. Hold configuration constant. Same cluster size, same file layout, same concurrency. Change one variable per run.

  3. Separate cold from warm. First-run costs — metadata loading, cache warming, JIT — belong in their own measurement, not averaged into steady state.

  4. Record the full setup. If you cannot write down enough detail for a colleague to repeat the run, the result is not evidence.

  5. Treat every number as a claim. A vendor figure is a hypothesis about your environment, not a premise you can adopt.

The same discipline applies to convergence claims. "Unified" is a claim; a plan you generated, a checkpoint you inspected and a concurrent write you attempted are evidence.

A Runnable Acceptance Checklist for Single-Engine Claims

The checklist below converts the preceding sections into pass/fail items a data architect or procurement lead can execute in a bounded evaluation window. It is vendor-neutral and applies to any candidate. For each item, the pass evidence is stated alongside the evidence that should be treated as inconclusive.

Semantic equivalence

  • Test: Run the same query against the same logical dataset in batch and streaming modes.

  • Pass: Identical results, including null handling and time zone behavior, with differences documented where boundedness legitimately changes the answer.

  • Inconclusive: A demo on a curated dataset, or a claim that the API is shared.

Execution unity

  • Test: Capture query plans and runtime logs for both modes.

  • Pass: One planner and one runtime visible in the plans and logs.

  • Inconclusive: A single console, or a shared SQL dialect.

State and recovery

  • Test: Inspect the checkpoint model and the documented guarantee for each mode.

  • Pass: One checkpoint mechanism, with the guarantee stated separately per execution mode and confirmed by a failure test.

  • Inconclusive: A single guarantee quoted for the whole product.

Table and concurrency

  • Test: Run a streaming write and a batch backfill against the same table concurrently.

  • Pass: Both governed by one commit protocol, with the conflict behavior documented and observed.

  • Inconclusive: Successful sequential writes, or a shared table format alone.

Catalog and governance

  • Test: Create a table via streaming, query it via batch, and apply one permission policy to both.

  • Pass: One namespace and one permission model, enforced identically.

  • Inconclusive: A shared catalog with separately administered permissions.

Operations

  • Test: Review the metrics, alerting and resource management surfaces for both modes.

  • Pass: One metrics and alerting surface covering both.

  • Inconclusive: Two dashboards behind one login.

Score each item independently. A candidate that passes four of six has partial convergence, and you should know which two gaps you are accepting before you sign.

Where the Unified Lakehouse Fits — and Where It Does Not

A single engine genuinely simplifies several common architectures. Incremental pipelines benefit because the transformation logic exists once. Batch backfill and streaming ingestion into shared tables benefit because both write under one commit protocol instead of being reconciled afterward. Late-arriving data benefits because one watermark policy governs the answer, rather than two policies whose disagreement surfaces as a discrepancy report.

It does not simplify everything. Specialized engines still earn their place in at least two situations. The first is extreme low-latency operational streaming with strict millisecond requirements, where the trade between latency and guarantee is the whole design problem — Spark's own documentation frames this as a choice between micro-batch exactly-once at roughly 100 milliseconds and continuous processing at roughly 1 millisecond with at-least-once guarantees. The second is an environment with heavy investment in a specific ecosystem, where the migration cost of consolidating outweighs the duplication cost of staying.

The more important shift is what a unified engine does to the architecture question. It moves the burden from "how do we keep two implementations in sync" to "how do we govern one implementation well." That burden lands on the catalog, the table format and the governance layer — which is exactly why the four-layer distinction from the opening section matters. Consolidating the engine does not consolidate the catalog or the permission model for you.

Singdata positions its lakehouse on a single-engine architecture, and it is one candidate among several to be evaluated against the checklist above. Any performance figure a vendor publishes, including Singdata's, should be treated as a claim to test in your own environment rather than a premise to adopt.

Conclusion: From Two Truths to One Testable Architecture

The value of "one engine" is not that a vendor says it. It is that the property can be defined, tested, and accepted or rejected on evidence you gather yourself.

That is the move this guide has tried to make. First, separate the four layers — engine, platform, catalog, table format — and ask which one actually converges. A shared table format is a storage contract, not an execution engine; a shared console is packaging, not a shared planner. Only engine-layer convergence removes the duplicated logic that separate batch and streaming stacks force on you.

Second, treat the six mechanisms as independent capabilities rather than a single badge. A unified programming model, one runtime underneath both modes, one state and checkpoint model, a commit protocol that reconciles streaming writes with concurrent batch reads, one catalog, and one governance surface each carry their own prerequisites. Satisfying one tells you nothing about the next.

Third, apply the co-location test. Compare plan shapes across modes, check whether state and recovery are one system or two, and confirm that a backfill and a streaming job can write the same table under one concurrency protocol.

The acceptance checklist is the artifact to carry into an evaluation. Run it against every candidate on your shortlist, record the evidence behind each pass, and mark anything you could not observe as inconclusive rather than assumed. A vendor that welcomes that process is telling you something useful; so is one that does not.

Frequently Asked Questions

Does adopting one table format make our architecture single-engine?

No. A table format is a storage contract: it defines layout, metadata, and commit behavior so that multiple compute engines can read and write the same table safely. Apache Iceberg describes itself as an open table format that adds tables to compute engines including Spark, Trino, PrestoDB, Flink, Hive, and Impala — the engines remain separate. Delta Lake's concurrency documentation likewise describes transactional guarantees between reads and writes, not a shared execution engine. A common format reduces storage-level divergence; it does not remove a second query planner or a second set of runtime semantics.

Does a unified API guarantee identical batch and streaming semantics?

No. A unified API guarantees that you can express both kinds of computation in the same way, which is a real gain in code maintenance. It does not guarantee that every operator behaves identically in both modes. Spark's Structured Streaming guide, for example, lists operations that are unsupported on streaming Datasets, and its output modes are constrained by query type. Verify semantics per mode: run the same logic as a bounded query and as an unbounded one, and compare results, state behavior, and late-data handling rather than assuming equivalence from the shared API.

Can micro-batch streaming meet real-time latency requirements?

It depends on the target and the guarantee you need. Spark's documentation states that its default micro-batch engine achieves end-to-end latencies as low as 100 milliseconds with exactly-once fault tolerance, while its continuous processing mode reaches latencies as low as 1 millisecond with at-least-once guarantees. Those are different operating points, not a ranking. If your requirement is single-digit milliseconds and you can tolerate at-least-once delivery plus downstream deduplication, continuous processing is the relevant mode to test; if you need exactly-once end to end, measure the micro-batch path against your actual latency budget.

How do we start an evaluation without disrupting production?

Begin with a bounded, read-only semantic-equivalence test. Take a representative slice of production data, express one transformation as both a batch query and a streaming query on the candidate engine, and compare outputs, plan shapes, and recovery behavior. This requires no migration commitment and no write access to production tables. Only after that test passes should you move to write-path and concurrency checks, and only after those should you discuss cutover.