Table Format Comparison — Iceberg vs Delta vs Hudi, Head to Head

Overview

Once you know each format’s internals, the interview question becomes comparative: given this workload, which format, and why — and what exactly differs under the hood? This page aligns the three on the same axes: commit protocol, partition evolution, row-level deletes, concurrency semantics, ecosystem, and catalog requirements. The honest headline is convergence — deletion vectors, COW/MoR options, and cross-format readership have closed most gaps since 2023 — but the remaining differences (index, catalog coupling, clustering model) are exactly what decision-makers ask about. Every claim here is justified on the per-format pages: Apache Iceberg, Delta Lake, Apache Hudi.

The Big Table

Commit Protocols in Detail

All three are optimistic concurrency, but the atomic point and the rebase logic differ:

flowchart TD
    A["Writer stages data files off snapshot V"] --> B{"Atomic publish"}
    B -->|Iceberg| C["Catalog CAS on root pointer, Glue/Nessie/JDBC/REST"]
    B -->|Delta| D["Conditional put of next log file NN.json"]
    B -->|Hudi| E["Complete the instant on the timeline"]
    C --> F{"Won?"}
    D --> F
    E --> F
    F -->|Yes| G["Version V+1 visible atomically"]
    F -->|No| H["Rebase: re-read winner, check conflict rules"]
    H --> I{"Conflicts?"}
    I -->|No| B
    I -->|Yes| J["Abort or rerun, e.g. MERGE retry"]

Two subtle points that separate senior answers from junior ones. First, Delta’s atomic point needs no catalog at all — on S3 it is a conditional put on the next log key — which is why Delta tables “work” in a bare directory, while Iceberg requires a catalog with real compare-and-swap semantics (raw Hive Metastore is the known weak spot: HMS locking is coarser than a true CAS). Second, Hudi’s conflict check happens against the timeline and the index, so concurrent upserts of the same key are detected at the file-group level — a stronger guarantee than pure file-list overlap for record-level work.

Partition Evolution and Layout Management

Hidden partitioning remains Iceberg’s clearest differentiator: predicates on source columns prune automatically, there is no way to write a “forgotten partition filter” query, and granularity can change per new data. Delta’s Liquid Clustering has effectively absorbed much of that value on the Databricks side by abandoning manual partition columns, and Hudi’s bucket index doubles as layout. If an interviewer asks “can I change partitions on a Delta table,” the correct answer is “not like Iceberg — you re-cluster (liquid) or rewrite; the directory scheme is static.”

Row-Level Deletes: Three Different Philosophies

The trade is a single curve: where do you pay — at write, at read, or at compaction? COW pays at write; position deletes and DVs pay a little at read; equality deletes and MoR logs pay the most at read and rely on scheduled compaction. What made Hudi distinctive was not MoR itself but the index that makes MoR appends hit one file group; what made Delta’s DVs impactful was making them the default so UPDATE stopped meaning “rewrite the file.”

Concurrency Semantics

The failure mode to name in interviews is the retention-vs-reader race: every format has a GC (expire_snapshots/VACUUM/clean) that can delete files a still-running reader needs if retention is set below the longest query. All three decouple metadata retention from physical file deletion precisely so this can be tuned.

Ecosystem, Vendor Neutrality, and Catalogs

• Iceberg is the neutrality winner by design: the spec defines readers/writers for any engine, the REST catalog spec lets vendors (Unity Catalog, Snowflake Polaris, Gravitino) interoperate, and Snowflake/BigQuery/Trino/Flink/DuckDB all read it natively. Costs: you must operate a catalog, and maintenance has more moving parts.
• Delta is the integration winner: Databricks runtime features (Photon, Liquid Clustering, managed VACUUM/OPTIMIZE), the cleanest Spark semantics, and now UniForm emitting Iceberg metadata for Iceberg-only engines. Costs: best features cluster around Databricks/Spark; self-hosted non-Spark stacks lean on delta-rs.
• Hudi is the ingest winner: record index + MoR + async compaction/clustering as a coherent pipeline for CDC at scale. Costs: more operational state (index, compaction plans), and engine coverage historically narrower (Spark/Flink strongest; Trino reads well, writes less so).

Catalog requirements compared:

What Benchmarks Actually Show

Public head-to-heads (Tabulario’s Iceberg benchmark series, Databricks’ 2021–2022 Delta-vs-Iceberg studies and the Iceberg community’s rebuttals, Fivetran’s ingestion comparisons) agree on one meta-finding: results are dominated by configuration, not format. The knobs that swing benchmarks by 2–10×:

1. Whether MoR tables get compaction (uncompacted MoR loses; compacted MoR is competitive or faster on ingest).
2. Target file sizes and write fan-out (a 128 MB vs 512 MB file setting changes pruning and metadata costs).
3. Whether DVs/deletes are compacted before the read benchmark runs.
4. Caching layers of the engine (Databricks’ disk cache vs OSS Spark) masquerading as format differences.

The defensible conclusions: append-heavy batch tables are a wash; upsert-heavy with random keys favors Hudi (index) unless Delta/Iceberg DVs are enabled and compacted; concurrent multi-writer and partition evolution favor Iceberg; Databricks-centric shops get the most out of Delta. If asked for numbers, give the mechanism (“uncompacted MoR merges logs at read time, so a scan reads base+logs, roughly doubling I/O”) rather than a percentage you cannot defend.

Decision Flowchart

flowchart TD
    Q1{"Primary workload?"}
    Q1 -->|CDC upserts, minute freshness| HUDI["Hudi: index + MoR + async compaction"]
    Q1 -->|Batch and SQL analytics| Q2{"Engines?"}
    Q1 -->|Mixed lakehouse platform| Q2
    Q2 -->|Databricks/Spark centric| DELTA["Delta: deepest integration, liquid clustering"]
    Q2 -->|Trino + Spark + Flink + warehouse readers| ICE["Iceberg: neutral spec, REST catalog"]
    Q1 -->|Massive appends, rare updates| ICE
    ICE --> Q3{"Catalog ready?"}
    Q3 -->|Yes| OK["Ship it"]
    Q3 -->|No| OK2["Adopt Glue/Nessie/REST first, format is secondary"]
    HUDI --> OK
    DELTA --> OK

The flow encodes the practical ordering of concerns: workload first, then engine estate, then governance/catalog. Teams that invert this — picking a format before confirming catalog capability — end up with the classic failure: Iceberg tables no catalog can atomically commit, or Hudi tables nobody operates compaction for.

Maintenance Cadence Compared

Same jobs, different names and mechanisms — a table that doubles as an ops checklist:

The cadence decision is uniform across formats even when the commands differ: compact on a multiple of ingest interval (hourly for streaming, nightly for batch), expire history on the retention your audit requirement dictates, orphan-sweep weekly with an age threshold comfortably above your longest query. What differs is who runs it: Databricks automates much of Delta’s list; Iceberg and OSS Hudi expect you to schedule the procedures yourself (or adopt a managed catalog/pipeline that does). Interviewers hiring data-platform engineers are probing for exactly this awareness — formats do not self-maintain.

Migration Paths Between Formats

The strategic point (and a fair interview answer to “are we locked in?”): data files are portable, metadata is the moat. Conversion tools rewrite metadata trees without touching Parquet, so a migration is an I/O-light metadata rebuild plus a validation pass — hours for hundreds of terabytes, not weeks. What does not migrate cleanly is behavioral state: deletion vectors, ongoing compaction state, indexes, and table properties. Treat feature parity as the migration checklist (DVs on? liquid clustering? bucket index?), not just row counts.

Governance and Security Features

Governance is increasingly the real differentiator as core file/metadata features converge. Iceberg’s catalog-centric design pushes policy (row filters, credential vending, table properties) into the catalog layer, which is why vendors raced to build REST catalogs; Delta’s answer lives inside the Databricks platform (Unity Catalog); Hudi leans on the deployment environment. If an interviewer frames the question as “we need lakehouse governance without a warehouse,” the vendor-neutral answer is Iceberg + a REST-catalog implementation — and the honest caveat is that this stack is younger than warehouse-native governance.

Failure Modes Checklist

Print this table mentally before a “troubleshoot our lakehouse” interview round: each row is a real production incident class, and each fix is a one-liner that demonstrates operational depth. The last row is the quiet killer — teams migrate from Hive-style tables, keep tools that LIST directories instead of using metadata, and blame the format for latency that is actually their planner bypassing the metadata tree entirely.

Interview Questions

1. What is the single biggest architectural difference between Iceberg and Delta? Iceberg commits by swapping one catalog pointer to a new metadata.json tree, so it needs a catalog with true compare-and-swap but gains engine neutrality and cheap planning via manifests. Delta commits by conditionally appending the next numbered JSON file to _delta_log, so the log directory is self-describing and no catalog is strictly required. Everything downstream differs from that choice: Iceberg’s manifests carry per-file stats for planning, Delta replays a log plus checkpoints; Iceberg gets hidden partitioning and evolution, Delta gets simplicity and tight Spark integration. Both remain optimistic-concurrency systems built on immutable Parquet.

2. Compare how each format handles DELETE FROM t WHERE user_id = 42. Delta writes a deletion-vector bitmap for the touched file’s row positions and appends a tiny commit — the file itself is untouched until compaction. Iceberg writes either position delete files (if it resolved target files via stats) or equality deletes keyed on user_id, which apply as an anti-join against all older files in scope. Hudi with MoR appends a delete record to the file group’s log located via its record index, applying it during reads until compaction. Write cost is lowest for the bitmap/log approaches; read cost is lowest for COW and highest for equality deletes; all three need periodic compaction to restore pure-file scans.

3. Why does Hudi historically beat the others on high-rate upserts? Its record index maps keys to file groups, so an upsert touches exactly one file group with no stats-guessing. Iceberg and Delta must infer candidate files from per-file min/max and partition stats, then rewrite or delete-mark every file that might contain each key — expensive for random keys that spread across all files. Hudi’s MoR mode then converts the upsert into a log append, with compaction running asynchronously. Since deletion vectors landed in Delta and Iceberg v3, the raw delete cost has converged, but indexed targeting of random keys remains Hudi’s edge.

4. Which format supports changing partition granularity without a rewrite, and how? Iceberg: SET PARTITION SPEC starts writing new data under a new spec-id while old manifests keep the old tuples, so a table can hold daily partitions for 2022 and hourly for 2025 with no rewrite and no broken queries. Delta’s partition columns are directory structure, so a change is a rewrite — Liquid Clustering sidesteps this by replacing manual partitioning with incremental clustering. Hudi re-keys through clustering or insert_overwrite, which is also effectively a rewrite. If partition evolution is a requirement, it is the strongest single argument for Iceberg.

5. How would you design for 50 concurrent streaming writers on one table? All three formats serialize commits, so the design goal is minimizing conflict probability, not finding a lock-free format. Partition the stream so writers land on disjoint partitions or file groups (by key hash), and batch commits (checkpoint interval, micro-batch duration) so commit rate stays under the catalog/log ceiling. Expect appends to never conflict but merges/deletes to retry; monitor retry rates and back off. If the table truly needs multi-table or cross-table transactions, that pushes toward Iceberg with a Nessie-style catalog, since Delta and Hudi commits are single-table.

6. A vendor benchmark says their format is 4x faster. What do you ask? First: what was compacted and when — uncompacted MoR or unmerged deletion vectors penalize reads enormously, and benchmarks rarely run the maintenance job the vendor’s managed platform runs automatically. Second: file sizes, sort order, and partition alignment, because pruning is stats-driven in every format. Third: which engine and caches were used, since Databricks’ disk cache or vendor-side metadata caching dominates cold-vs-warm scans. The general result across public comparisons is that configuration dominates the format; the format choice should be argued on governance, engines, and workload shape instead.

Key Takeaways

• All three formats: immutable Parquet + atomic metadata commit + optimistic concurrency; the metadata shape (tree vs log vs timeline) is the real difference.
• Delta needs no catalog (the log is self-describing); Iceberg requires a real CAS catalog; Hudi’s timeline is authoritative with a metastore for discovery only.
• Hidden partitioning and no-rewrite partition evolution remain Iceberg’s clearest differentiators.
• Row-level deletes converge on “bitmap or log now, compaction later”; Hudi’s record index still targets random-key upserts best.
• Concurrency differences are about conflict granularity (file overlap vs predicates vs file groups), not locking.
• Benchmarks are config-dominated: compaction cadence, file sizes, and engine caches swing results more than format choice.
• Decide by workload (CDC vs analytics), engine estate (Databricks vs multi-engine), and catalog readiness — in that order.

Cross-References

• Apache Iceberg — snapshot tree, hidden partitioning, catalogs
• Delta Lake — _delta_log, checkpoints, deletion vectors, liquid clustering
• Apache Hudi — timeline, COW/MoR, record index
• Lakehouses — the workload overview this page deepens
• Parquet Internals — the shared data file format and its stats
• Object Storage — why conditional single-key writes anchor every commit
• Trino — the multi-engine story that favors Iceberg

References

• Iceberg table spec — snapshot isolation, delete files, conflict semantics
• Delta Lake protocol — commit log, concurrency, deletion vectors
• Delta Lake docs — concurrency control and best-practice guides
• Apache Hudi docs — table types, timeline, indexing
• Apache Iceberg docs — maintenance procedures and engine matrix
• Vendor benchmark series (Tabulario Iceberg benchmarks; Databricks Delta-vs-Iceberg studies, 2021–2022) — no stable URL relied upon; treat as configuration-sensitive evidence