GitHub - delta-io/delta-rs: A native Rust library for Delta Lake, with bindings into Python (original) (raw)

delta-rs logo

Delta Lake is an open-source storage format that runs on top of existing data lakes. Delta Lake is compatible with processing engines like Apache Spark and provides benefits such as ACID transaction guarantees, schema enforcement, and scalable data handling.

The Delta Lake project aims to unlock the power of the Deltalake for as many users and projects as possible by providing native low-level APIs aimed at developers and integrators, as well as a high-level operations API that lets you query, inspect, and operate your Delta Lake with ease.

Source Downloads Installation Command Docs
PyPi Downloads pip install deltalake Docs
Crates.io Downloads cargo add deltalake Docs

Table of contents

Quick Start

The deltalake library aims to adopt patterns from other libraries in data processing, so getting started should look familiar.

from deltalake import DeltaTable, write_deltalake import pandas as pd

write some data into a delta table

df = pd.DataFrame({"id": [1, 2], "value": ["foo", "boo"]}) write_deltalake("./data/delta", df)

Load data from the delta table

dt = DeltaTable("./data/delta") df2 = dt.to_pandas()

assert df.equals(df2)

The same table can also be loaded using the core Rust crate:

use deltalake::{open_table, DeltaTableError};

#[tokio::main] async fn main() -> Result<(), DeltaTableError> { // open the table written in python let table = open_table("./data/delta").await?;

// show all active files in the table
let files: Vec<_> = table.get_file_uris()?.collect();
println!("{files:?}");

Ok(())

}

You can also try Delta Lake docker at DockerHub | Docker Repo

Get Involved

We encourage you to reach out, and are committedto provide a welcoming community.

Integrations

Libraries and frameworks that interoperate with delta-rs - in alphabetical order.

Features

The following section outlines some core features like supported storage backendsand operations that can be performed against tables. The state of implementation of features outlined in the Delta protocol is also tracked.

Cloud Integrations

Storage Rust Python Comment
Local done done
S3 - AWS done done
S3 - MinIO done done
S3 - R2 done done
Azure Blob done done
Azure ADLS Gen2 done done
Microsoft OneLake done done
Google Cloud Storage done done
HDFS done done
LakeFS done done Python: Rust engine writer only supported

Supported Operations

Operation Rust Python Description
Create done done Create a new table
Read done done Read data from a table
Vacuum done done Remove unused files and log entries
Delete - predicates done done Delete data based on a predicate
Optimize - compaction done done Harmonize the size of data file
Optimize - Z-order done done Place similar data into the same file
Merge done done Merge a target Delta table with source data
Update done done Update values from a table
Add Column done done Add new columns or nested fields
Add Feature done done Enable delta table features
Add Constraints done done Set delta constraints, to verify data on write
Drop Constraints done done Removes delta constraints
Set Table Properties done done Set delta table properties
Convert to Delta done done Convert parquet table to delta table
FS check done done Remove corrupted files from table
Restore done done Restores table to previous version state

Protocol Support Level

Writer Version Requirement Status
Version 2 Append Only Tables done
Version 2 Column Invariants done
Version 3 Enforce delta.checkpoint.writeStatsAsJson done
Version 3 Enforce delta.checkpoint.writeStatsAsStruct done
Version 3 CHECK constraints done
Version 4 Change Data Feed done
Version 4 Generated Columns done
Version 5 Column Mapping
Version 6 Identity Columns
Version 7 Table Features done
Reader Version Requirement Status
Version 2 Column Mapping
Version 3 Table Features (requires reader V7) done