What are open table formats?

Open table formats, Apache Iceberg, Delta Lake and Apache Hudi are the three most widely used, are open, vendor-neutral specifications that add a management layer on top of plain data lake files (typically Parquet): transactions, versioning, schema evolution, and the ability to query time travel back to earlier versions of the data. It is this layer that turns a data lake, a folder of files, into a data lakehouse, a platform with database-grade guarantees. The three formats solve the same problem with different technical choices (metadata handling, compatibility with existing compute engines, project history), but all share the same underlying goal: bringing database-world transactional guarantees into open, cheap file storage. Iceberg originated at Netflix, Delta Lake at Databricks, Hudi at Uber: each still carries traces of the problem it was built to solve, worth knowing when evaluating which one to adopt for a new project.

The anti-lock-in argument

Before these formats, transactional guarantees at large data volumes required a proprietary database: data lived inside a closed format, readable only by that vendor's engine. Open table formats flip this logic: data stays on cheap storage in a public, documented format, and any compute engine implementing the spec, Spark, Trino, Snowflake, BigQuery, Databricks, can read and write it. The practical result is that your data is no longer hostage to a single platform, but engine parity has to be checked case by case: reading is broadly supported, writing much less so, and Snowflake and BigQuery write Iceberg while handling Delta and Hudi read-only. The residual lock-in has also moved from the format to the catalog: whoever controls the catalog controls who may write, and that is where the contract clause deserves attention. For the same reason the open format belongs on every layer of the medallion architecture, not just gold: the history that makes a migration expensive sits in bronze and silver. None of the three formats is objectively "the best" across the board, the choice depends on the ecosystem you sit in and the dominant engine on your platform.

Why it matters for your business

Adopting an open table format in your medallion architecture is today's default choice for building a data lakehouse without tying yourself to a single cloud vendor for the next decade. The advantage is not just technical, it is negotiating leverage. A vendor who knows you can migrate your data elsewhere with an engine swap, not a months-long migration, treats renewals and pricing differently. It is worth verifying, before signing with a platform, that data is written in one of these open formats and not in a proprietary format dressed up as a standard.