"This module provides the capability to read, write, and manage Delta Lake tables"
OK what's a Delta Lake? Apparently it's "... an open-source storage framework that enables building a format agnostic Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, Hive, Snowflake, Google BigQuery, Athena, Redshift, Databricks, Azure Fabric and APIs for Scala, Java, Rust, and Python."
And there's an image that makes me want to run away screaming:
@itamarst yeah, i think it’s best to think of it as a file management protocol on top of parquet that gives the illusion of insert, update and delete operations. being based on immutable files, it also gives you history of a table. being a protocol instead of a server means multiple uncoordinated writers can all write to the same table (well, coordination is pushed down to the storage layer…S3/blob/etc)
Add comment