Bacalhau is an open-source distributed compute orchestration framework designed to bring compute to the data. Instead of moving large datasets around networks, Bacalhau makes it easy to execute jobs close to the data's location, drastically reducing latency and resource overhead.
- ⚡ Fast job processing: Jobs in Bacalhau are processed where the data was created and all jobs are parallel by default
- 💰 Low cost: Reduce (or eliminate) ingress/egress costs since jobs are processed closer to the source
- 🔒 Secure: Data scrubbing and security can happen before migration, with a granular, code-based permission model
- 🚛 Large-scale data: Process petabytes of data efficiently without massive data transfers
- 🏢 Data sovereignty: Process sensitive data within security boundaries without requiring it to leave your premises
- 🤝 Cross-organizational computation: Allow specific vetted computations on protected datasets without exposing raw data
-
Single Binary Simplicity: Bacalhau is a single self-contained binary that functions as a client, orchestrator, and compute node—making it incredibly easy to set up and scale
-
Modular Architecture: Support for multiple execution engines (Docker, WebAssembly) and storage providers through clean interfaces
-
Orchestrator-Compute Model: A dedicated orchestrator coordinates job scheduling, while compute nodes run tasks
-
Flexible Storage Integrations: Integrates with S3, HTTP/HTTPS, IPFS, and local storage systems
-
Multiple Job Types: Support for batch, ops, daemon, and service job types for different workflow requirements
-
Declarative & Imperative Submissions: Define jobs in YAML (declarative) or pass arguments via CLI (imperative)
-
Publisher Support: Output results to local volumes, S3, or other storage backends
# Install Bacalhau CLI (Linux/macOS)
curl -sL https://get.bacalhau.org/install.sh | bash
# Verify installation
bacalhau versionFor the complete quick start guide, including running your first job, see the Quick Start Documentation.
Bacalhau's distributed compute framework enables a wide range of applications:
- Log Processing: Process logs efficiently at scale by running distributed jobs directly at the source
- Distributed Data Warehousing: Query and analyze data across multiple regions without moving large datasets
- Fleet Management: Efficiently manage distributed nodes across multiple environments
- Distributed Machine Learning: Train and deploy ML models across a distributed compute fleet
- Edge Computing: Run compute tasks closer to the data source for applications requiring low latency
📚 Read the Bacalhau documentation 📚
The Bacalhau documentation contains all the information you need to get started:
Bacalhau has a very friendly community, and we are always happy to help:
- Join the Slack Community and go to the
#generalchannel - it is the easiest way to engage with other members in the community and get help
If you are interested in contributing to the Bacalhau project:
- Set up your local environment
- Check out our Contributing Guide
- For issues and feature requests, please open a GitHub issue
We are excited to hear your feedback!
This repository contains the open-source Bacalhau software, covered under the Apache-2.0 license except where noted. Commercial terms are not required to use, modify, or distribute this code under that license.
The Apache 2.0 license does not grant rights to the Bacalhau name or logos. Distributions may use those trademarks only as permitted by the applicable trademark policy or other written permission.
We explicitly grant permission for you to make a build that includes our trademarks while developing Bacalhau software itself. You may not publish or share the build, and you may not use that build to run Bacalhau software for any other purpose.
We have borrowed the above Open Source clause from the excellent System Initiative