What is OpenETL?

OpenETL copies data between your systems. Source connectors read data from databases and event streams. Destination connectors write that data into warehouses, data lakes and other streams. Pipelines keep the copies up to date, on a schedule or continuously. OpenETL reports failures and other conditions that need attention.

OpenETL is open source under Apache 2.0. You run it on your own machines or in your own cloud account.

Connect your systemsCopy Anchor Link

Connectors provide the connection to each system. A connector can read data, write data, or do both. Which systems you can connect, and which ways you can read or write them, depend on the connectors you install.

The official connectors live in openetl-connectors. Community members and organizations can also build and maintain connectors in their own repositories, including private ones. Connector authors choose their own versions and release cadence, independently of OpenETL.

For setup, permissions, config and limits, read the documentation for the connector version you use. Choose and manage connectors explains how to find, install and upgrade them. Build a connector covers writing your own.

What it does for youCopy Anchor Link

• Runs on a schedule or continuously. A pipeline can copy data at set times or keep reading changes without stopping.
• Checks delivery guarantees. OpenETL determines whether each stream can receive duplicate changes. It refuses to run a pipeline that cannot meet its required guarantee.
• Recovers from failures. After a machine failure, the next attempt resumes from the last checkpoint. If retries cannot resolve the error, OpenETL opens an incident with recovery steps. Some incidents also pause the pipeline.
• Controls schema changes. New columns are added by default. An existing destination column's type changes only if the pipeline permits it. Otherwise, OpenETL holds the affected values where supported, or stops the stream.
• Reports pipeline health. The console shows sync results, data freshness, schema changes and incidents. OpenETL exports metrics and traces through OpenTelemetry. Webhooks send events to your endpoints.
• Transforms data. dbt can run after each sync. Spark and Flink jobs on your clusters can transform data between the source and destination.
• Keeps definitions in git. Pipelines are YAML files. Use openetl plan to review a change before openetl apply makes it.

Who uses itCopy Anchor Link

• Analysts, product managers and other data users check freshness and diagnose stale tables in the console. Start with Use the console.
• Data and analytics engineers write and review pipelines. Start with Pipelines.
• Platform engineers install and run OpenETL for everyone else. Start with Self-host.
• Developers connect it to other tools, or write connectors for systems they need: Integrate and Build a connector.

NextCopy Anchor Link

• Try it on your laptop with the quickstart.
• Read the concepts: a short page that explains every word the rest of the docs use.