treeverse /lakeFS

lakeFS - Data version control for your data lake | Git for data

★ 5,589⑂ 491👁 40⑃ 6,516● 3,990▤ 252
GoNOASSERTIONCreated 2019-09-12docs.lakefs.io
More from this author
Download: v1.88.0v1.88.0 ↓1.88.02026-10-01
+0
Daily
5,589
stars
491
forks
40
watchers
6,516
prs
3,990
issues
252
releases

README

lakeFS is Data Version Control (Git for Data)

lakeFS is a source-available tool that transforms your object storage into a Git-like repository. It enables you to manage your data lake the way you manage your code.

With lakeFS you can build repeatable, atomic, and versioned data lake operations - from complex ETL jobs to data science and analytics.

lakeFS supports AWS S3, Azure Blob Storage, and Google Cloud Storage as its underlying storage service. It is API compatible with S3 and works seamlessly with all modern data frameworks such as Spark, Hive, AWS Athena, DuckDB, and Presto.

For more information, see the documentation.

 Getting Started

You can spin up a standalone sandbox instance of lakeFS:

bash
pip install lakefs
python -m lakefs.quickstart


Once you've got lakeFS running, open http://127.0.0.1:8000/ in your web browser.

 Getting Started with Docker

Alternatively, you can run lakeFS a server using Docker:

bash
docker run --pull always \
		   --name lakefs \
		   -p 8000:8000 \
		   treeverse/lakefs:latest \
		   run --quickstart


 Quickstart

👉🏻 For a hands-on walk through of the core functionality in lakeFS head over to the quickstart to jump right in!

Make sure to also have a look at the lakeFS samples. These are a rich resource of examples of end-to-end applications that you can build with lakeFS.

 Why Do I Need lakeFS?

 ETL Testing with Isolated Dev/Test Environment

When working with a data lake, it’s useful to have replicas of your production environment. These replicas allow you to test these ETLs and understand changes to your data without impacting downstream data consumers.

Running ETL and transformation jobs directly in production without proper ETL Testing is a guaranteed way to have data issues flow into dashboards, ML models, and other consumers sooner or later. The most common approach to avoid making changes directly in production is to create and maintain multiple data environments and perform ETL testing on them. Dev environment to develop the data pipelines and test environment where pipeline changes are tested before pushing it to production. With lakeFS you can create branches, and get a copy of the full production data, without copying anything. This enables a faster and easier process of ETL testing.

 Reproducibility

Data changes frequently. This makes the task of keeping track of its exact state over time difficult. Oftentimes, people maintain only one state of their data––its current state.

This has a negative impact on the work, as it becomes hard to:
 Debug a data issue.
 Validate machine learning training accuracy (re-running a model over different data gives different results).
Comply with data audits.

In comparison, lakeFS exposes a Git-like interface to data that allows keeping track of more than just the current state of data. This makes reproducing its state at any point in time straightforward.

 Write-Audit-Publish

Data pipelines feed processed data from data lakes to downstream con

Top issues

Discussions

Latest release

v1.88.0 v1.88.0 2026-10-01
Important note: Action hooks can no longer send requests to internal addresses (loopback, private, link-local) by default. Allow the internal hosts your hooks need with actions.network.allowed_hosts. Hook requests no longer use the HTTP_PROXY/HTTPS_PROXY environment variables.

 Security Fixes

- Fixed a vulnerability that allowed users who can write action files to make the lakeFS server send requests to internal addresses (GHSA-223c-qwgx-wwcg)
- S3 gateway now rejects SigV4 requests with unsigned x-amz- headers (GHSA-96f7-c79c-crg8)

Snapshot: 2026-11-16T09:23:14Z · Data license Apache-2.0 · JSON