CMSC724: Delta Lake and Table Formats

Instructor: Amol Deshpande

This chapter follows “Delta Lake: High-Performance ACID Table Storage over Cloud Object Stores” (Armbrust et al., SIGMOD 2020), and closes the storage module.

Where the previous chapters were about laying out bytes, this one is about a different problem: given thousands of Parquet files sitting in an object store, how do you make them behave like a table — one you can update atomically, read consistently, and travel backwards through time.

Passages marked as editorial extensions were written by the notes generator and go beyond the source slides and class discussion.

1. The Setting: What a Data Lake Actually Is

Cloud object stores as data lake storage

Data lakes are extremely common, and the data in them is typically stored in Parquet or ORC columnar formats. The storage underneath is usually Amazon S3, though Azure Blob Storage and Google Cloud Storage serve the same role.

A blunt observation about the terminology:

The phrase “data lake” is basically meaningless. There is nothing to install. You have a big S3 bucket and you call it a data lake. You install systems on top to operate on the data — but the lake itself is just S3.

The abstraction people actually want

The problem is that raw object storage is a poor interface. Suppose you receive sales data every day from many sources, each sending Parquet files. You now have thousands of Parquet files accumulating, perhaps organized by region and then by date.

But when you write a query, you do not want to think about files. You want to write SQL against a sales table that encompasses all of them — and against a logs table made of a different set of files, and so on.

That logical table abstraction over a collection of Parquet files is what Delta Lake, and the formats that followed it, exist to provide.

2. Why This Is Hard

Directories of files, and the problems with them

The naive implementation — a directory of Parquet files per table — runs into properties of cloud object stores that make it hard to guarantee much of anything.

No multi-object atomic updates. You cannot update several objects simultaneously. This turns out to be the central problem: any operation that touches more than one file cannot be made atomic by the storage layer.

Weak consistency models. S3 provides read-your-writes consistency: if a program writes a value and immediately reads it back, it gets what it wrote. That much was guaranteed. But there was a stranger failure: you could check whether an object existed, find it absent, write it, check again — and still be told it does not exist.

Listing is expensive. Running a LIST against S3 can be very costly, and “which files make up this table” is exactly a listing question.

Every object store is different, which is a real problem for a company like Databricks, whose business is the compute layer. They need to run against whichever store the customer already uses, and they have no choice but to absorb the differences.

Important

One of the paper’s premises has since been fixed. The consistency anomaly above was a real problem when the paper was written in 2020 — but AWS made S3 strongly consistent in late 2020, and Jeff Barr’s announcement explicitly addresses this case: you put an object, you get it, and you now receive the new value.

So when you read the Delta Lake paper carefully, keep in mind that some of its observations may no longer be correct. The eventual-consistency behavior was originally there for latency reasons, so one would expect a slight cost to fixing it.

This is a good instance of the general point about reading systems papers: the workarounds are often more durable than the problems that motivated them.

It is worth being precise about why eventual consistency was so painful here, since it is a recurring theme in this course.

The guarantee S3 originally offered was not wrong — it was just weaker than what applications assume by default. And the cost of that gap does not fall on the storage system; it falls on every application built on top, each of which must independently detect and compensate for anomalies that occur rarely and unpredictably.

That is the same dynamic from the abstractions chapter: a guarantee the system declines to provide does not disappear, it relocates to the applications, where it is handled less rigorously and tested less thoroughly. Delta Lake’s read protocol below is, in part, exactly that compensation code — written once, by people who understood the failure mode, so that everyone above them did not have to write it themselves.

3. Two Approaches to the Metadata

Given that you need to know which files constitute a table, where does that metadata live?

Option one: a separate metadata service. This is what Snowflake does — a metadata layer above the object store tracking all the files and their relationship to logical tables. It is designed to be strongly consistent, and it works. Snowflake uses FoundationDB for this.

The drawback: it becomes a choke point, because everything must go through it. And such services are hard to build.

Option two: put the metadata in the lake itself. This is Delta Lake’s approach — the information about what constitutes a table lives inside the data lake, as ordinary objects. This has become quite popular.

4. The Delta Table Format

Keep in mind that everything underneath is just files.

A logical table is typically partitioned — date-based partitioning is very common — and those directories contain the Parquet data files. As each day rolls around, more files are added. A query against the table runs against everything in there.

Alongside the data sits the _delta_log directory, containing small JSON files, numbered in sequence. These are the log records, and each one says which Parquet files were added and which were deleted.

Each JSON file is a transaction. Everything in 3.json is a single atomic operation.

Consider an update. Cloud stores are largely immutable — you cannot modify a file in place, only add a new one. So updating a table means writing new Parquet files and logically removing the old ones, and the log record records all of those additions and deletions together. That is precisely why the record must be atomic: you need every part of it to take effect or none.

How do these logs differ from database logs? A crucial difference:

Database log records are typically per-tuple — one update each — and the system relies on separate transaction start and transaction end markers to group many records into one transaction.

Delta Lake has no such concept. A single JSON file is a multi-object log record, and that one file is the entire transaction.

There is also a checkpoint file (in Parquet) periodically written, which we come to below.

5. The Read Protocol

To read the table, you must determine which Parquet files constitute the current snapshot. Naively that means reading the entire log from the beginning, since only by replaying all the additions and deletions do you know what survives.

That is what checkpoints are for: a checkpoint summarizes the state as of some log record, so you can start there instead of at zero. The protocol:

Read _last_checkpoint, and any log objects after it.
Work out which data objects to read, using the metadata and statistics in those records.
Read the relevant data objects — possibly waiting, if the store is eventually consistent.

And the log’s sequential numbering gives a property that is easy to overlook but genuinely valuable:

Because records are numbered 1, 2, 3, 4, you can detect missing data. If 2.json says there should be an abc.parquet and you do not see it, you know your view is incomplete and can raise an exception rather than silently returning wrong answers. If 4.json is missing entirely, you know that too.

Under a directory-of-files scheme, that same situation is indistinguishable from a table that simply has less data in it.

6. The Write Protocol

Writing is where the whole design turns on a single primitive.

Read up to the latest log record — call it r.
Read the table at version r, and write your new data into new Parquet files.
Attempt to write r+1.json. This write must be atomic. If it fails, retry from step 1.

Everything rests on step 3, because two execution engines may both be trying to write r+1.json at the same time, and only one may succeed. If both succeed, one silently overwrites the other and the table becomes inconsistent.

So the requirement is a put-if-absent operation, and support varies by store:

Store	Mechanism
Google Cloud Storage, Azure	Native atomic put-if-absent
HDFS	Atomic rename — write, then rename into place
S3 (at the time)	Not supported — Databricks had to add their own coordination layer

That last row is worth dwelling on. The entire correctness of a table format used across the industry reduces to whether the underlying object store can do one atomic compare-and-set on a single object name. Everything else — multi-file updates, snapshot reads, time travel — is built on top of that one primitive.

It is the storage-layer analogue of compare-and-swap in the architecture chapter: give the system exactly one atomic operation and it can construct the rest itself.

7. What You Get, and What You Don’t

Writes are serialized; reads get a snapshot. A reader may not see the very latest version — depending on where the client is — but is guaranteed a consistent version. This is snapshot isolation.

No transactions across tables. Updates are confined to a single table, so you cannot atomically add files to two different tables. (Technically a Delta table could correspond to multiple logical tables, which is a partial workaround.)

Transaction rates can be bottlenecked by the put-if-absent operation. In practice I have trouble seeing this bite, since you are adding Parquet files and the rate of those is fairly limited — but the paper discusses it.

Streaming latency. One major Spark use case is streaming, which requires writing many small objects. In theory this format supports it; in practice I am not sure how well it works.

Time travel

Delta Lake lets you go backwards in time. To retrieve the table as of three days ago, stop replaying the log at the corresponding record and you have that snapshot. You also get a complete audit history, since every change is recorded.

The caveat is retention: if 4.json deleted a Parquet file and that file was actually removed from storage, you cannot reconstruct anything before it. Whether old objects are physically deleted determines how far back you can go. This is a standard problem for any versioned store.

8. Optimizations and Schema Evolution

Several things become possible once a system owns the metadata:

Compaction. Small objects get merged in the background — important because streaming and frequent writes produce many small files, and reading thousands of tiny objects is slow.

Z-ordering by multiple attributes, which arranges data so that selective queries on several columns can skip most files.

Statistics in the log records, which is what makes file skipping possible at all — the read protocol consults them to decide which objects it needs.

Schema evolution, handled in a simple form. The problem is real: a Parquet file’s structure is tied to its schema, so if the schema evolves, older Parquet files may not be readable under the new schema — you can read them with their old schema, but not the new one. Delta Lake supports updating older objects in the background to bring them forward.

Indexes are mentioned but not developed in detail. There was a 2022 paper from MIT taking layout optimization to a more extreme conclusion, allowing arbitrary layout optimization, though I have not seen follow-up work.

9. The Broader Point

The paper’s framing is that Delta Lake simplifies enterprise data architectures: one system for many jobs, rather than a separate system for each — and it poses the question directly.

So “one size does fit all” after all?

And the observation the module ends on:

This is almost a throwback to the original motivation for a “shared data bank” — hide the complexity behind a logical abstraction that supports updates, and avoid keeping many copies of the same dataset in different systems.

Which is where this course started, sixty years earlier. The problem that produced databases in the 1960s — many programs, one shared body of data, no coherent way to manage it — reappeared once organizations put their data in object storage, and the answer has the same shape: put a logical abstraction over the physical mess and let the system maintain it.

10. The Competing Formats

This is a rapidly evolving space. In the last few years, two comparable formats have appeared alongside Delta Lake:

Apache Iceberg
Apache Hudi

All three support the same core idea — logical tables over a data lake — with different mechanisms. Delta Lake uses the log records and checkpoints described above; Iceberg uses metadata files and manifest lists. At a high level the differences are not that significant.

Iceberg is currently considered the best of the three and is the one winning out. Part of the reason is likely that it is genuinely open source, whereas Delta Lake was designed by Databricks with parts kept proprietary, which annoyed people.

The commercial evidence is striking: the Iceberg team founded a company, Tabular, which Databricks bought for a couple of billion dollars — a price that has puzzled most observers. How can a table format be worth that? There is no clear answer. (A good comparison of the three formats exists in a blog post by Dremio.)

Two questions worth thinking about, from the closing slides:

Could this be the primary backend for an OLTP system? Why or why not? Consider what OLTP needs — high transaction rates, fine-grained updates, low latency — against what this format provides: one atomic put per transaction, immutable data files, and snapshot isolation only within a table.

What is the impact of partitioning granularity? Too coarse and you read far more than you need; too fine and you accumulate small files, expensive listings, and long log replays.

Worth closing the module by noting what the five chapters together describe, because it is one continuous argument rather than five techniques.

Each step widens the scope of the same decision — how do we group values so that a read touches only what it needs — and pays a correspondingly larger price for it. PAX regroups within a page and pays essentially nothing, because nothing outside the page can tell. Column stores regroup across the whole relation and pay with updates, which is why they need a write-optimized store and a tuple mover. Dremel extends the regrouping to nested data and pays with reconstruction, which is why the automaton exists. Delta Lake regroups at the level of entire files in an object store and pays with a transaction protocol, because at that scale the grouping decision has become a distributed systems problem.

The through-line is that the idea has been stable since the 1980s DSM work; what changes each time is the boundary it is applied at, and the machinery each chapter introduces is the cost of moving that boundary outward one more level.