Self-host

This section explains how to install, operate and upgrade OpenETL for a team. It assumes experience with Linux, containers, Postgres and object storage. The Kubernetes guide also requires experience with Kubernetes.

To try OpenETL on your computer, follow the quickstart.

The partsCopy Anchor Link

The openetld program runs each component through a separate role:

Runners communicate with the control plane through the API. They can run in the same cluster or on another network near your databases. Each runner needs access to the API and the data systems used by its pipelines.

What you needCopy Anchor Link

• Postgres stores pipelines, versions, syncs, encrypted secrets, users and audit records. The default DBOS backend stores workflows in a second database on the same server. Its name is the metadata database name with _dbos appended. Orchestrators create it if the database user has permission.
• Object storage for checkpoints and staging: an S3 bucket (or one on an S3-compatible store), a GCS bucket or an Azure Blob Storage container. A local directory works only if every runner, and every Spark or Flink cluster, sees it at the same path. A container on the same machine sees the directory only if it is mounted in the container.
• openetld and openetl-engine. A runner searches for the engine beside openetld, then on PATH. The release's image includes both, the CLI and the console. Releases lists what each release has.
• The connectors that your pipelines use. Runners download the packages that you install. A runner also searches for local connector executables in connectors/ beside openetld.

ReleasesCopy Anchor Link

Each release of OpenETL has a version, such as 0.1.0-alpha, and a git tag, such as v0.1.0-alpha. OpenETL is in alpha: the pipeline format and the API can change between releases. The releases page has each release's notes and files:

The official connectors have releases of their own, in openetl-connectors. Their packages are in the registry oci://ghcr.io/openetl-io/openetl-connectors.

The release workflow signs the image and checksums.txt with Sigstore keyless signing. To check the image's signature with cosign 3 or later, run:

cosign verify ghcr.io/openetl-io/openetl:0.1.0-alpha \
  --certificate-identity-regexp '^https://github\.com/openetl-io/openetl/\.github/workflows/release\.yml@' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com

Check the signature shows the same check for checksums.txt.

The settings every installation setsCopy Anchor Link

Each openetld flag has an environment variable equivalent. openetld ROLE -h lists the settings and defaults. Configure these settings for your installation:

OpenETL permits outbound HTTP requests to public addresses and the network ranges listed in OPENETL_EGRESS_ALLOW. This restriction covers webhooks, registry downloads and connector requests made through OpenETL's HTTP service.

BucketsCopy Anchor Link

A runner, and the engine that it starts, use the first S3 credentials that they find:

1. AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. For temporary keys, also set AWS_SESSION_TOKEN.
2. The role that AWS gives the process: an EKS IAM role for a service account, EKS Pod Identity, an ECS task role or an EC2 instance profile. A container on EC2 can use the instance profile only if the instance metadata hop limit is 2 or more.

If a runner cannot get credentials from the source that it finds, it does not start. Its error names that source and what to check.

The runner also reads AWS profiles (AWS_PROFILE). The engine does not read profiles. If a runner uses a profile, also give it keys or a role.

These credentials are for the runner and the engine. By default, connectors do not get them. Run on AWS explains roles on EC2 and EKS, and how a connector gets AWS access.

When a runner starts, it asks S3 for the region of the bucket. To change how runners reach the bucket, add these query parameters to its URL:

For example: s3://openetl?endpoint=http://seaweedfs:8333&anonymous=true.

GCS buckets use the Google application default credentials. Azure containers use AZURE_STORAGE_KEY if it is set. Otherwise, they use the Azure default credential chain.

Connectors are trusted codeCopy Anchor Link

A connector runs as a separate program under the runner's user account, without a sandbox. It has the runner's file and network access, including access to cloud metadata addresses. OPENETL_EGRESS_ALLOW does not restrict sockets opened directly by a connector.

A connector can access the secrets for the sources and destinations it runs. OpenETL does not restrict where the connector sends those values. The descriptor's egress field declares intended destinations but is not enforced.

A connector does not get the runner's environment variables, except PATH, HOME, TMPDIR, TZ, LANG, SSL_CERT_FILE and SSL_CERT_DIR. So it does not get credentials that the runner reads from its environment:

• AWS keys in AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.
• The token of an EKS IAM role for a service account, or of EKS Pod Identity.
• An ECS task role.

By default, the runner also stops the AWS SDKs in a connector from using EC2's instance metadata. A connector gets AWS credentials as Credentials for connectors explains: a role for its source or destination, the runner's own credentials, or keys in its config. For other clouds, a connector needs credentials in its own config, such as a key stored as a secret.

A connector that looks for the runner's credentials on purpose may still find them. What OpenETL can block lists the cases.

Install only connectors you trust. Give runners only the credentials and network access their pipelines need.

Install and upgrade connectorsCopy Anchor Link

External connector packages have their own release cadence. Use the versions your pipelines need, whether maintained by OpenETL, the community or your organization. The release's image includes only the built-in connectors. Install every other connector as a package.

Choose and manage connectors covers applying packages, private registry credentials and controlled upgrades. A runner downloads the applied package's executable for its operating system and architecture. Keep the engine's storage and network requirements separate from each connector's requirements for reaching your data systems.

Sign-in and the first ownerCopy Anchor Link

The server requires password sign-in unless started with --auth none. openetld dev uses this option. With sign-in off, requests without an API key have owner permissions.

In this mode, the server rejects requests from other websites. It accepts host names matching localhost, a loopback address or --public-url. --auth-none-allow-remote permits other host names.

On first start, the server creates the installation's organization and writes an owner API key. The default file is .openetl/admin.key; OPENETL_BOOTSTRAP_KEY_FILE overrides it. Use that key to create a password link for an existing user:

export OPENETL_API_URL=https://openetl.example.com OPENETL_API_KEY="$(cat admin.key)"
openetl principal password-link NAME

The command prints a link for the user to set their password. openetl principal creates users and service accounts and grants roles. openetl key manages API keys. If all owner keys are lost, openetld bootstrap-key creates a replacement. See Manage access and credentials for the full procedure.

Choose an orchestratorCopy Anchor Link

Orchestrators run workflows on one of two backends, chosen when you install:

• DBOS, the default, keeps workflows in Postgres, so an installation needs nothing else.
• Temporal runs them on a Temporal cluster you already have: set OPENETL_ORCHESTRATOR=temporal, OPENETL_TEMPORAL_ADDRESS and OPENETL_TEMPORAL_NAMESPACE.

Choose before the first start, and give every API server and orchestrator the same settings.

Changing it laterCopy Anchor Link

OpenETL does not support changing the orchestration backend while the installation is running. There is no automatic migration. If you must change it, stop the control plane:

1. Pause every pipeline with openetl pause NAME. Wait until openetl syncs NAME shows no active runs.
2. Stop every API server and orchestrator.
3. Change OPENETL_ORCHESTRATOR, and Temporal's address and namespace, on all of them.
4. Start the API servers and orchestrators. Resume the pipelines with openetl resume NAME.

At startup, the control plane rebuilds schedules and job monitors from the metadata database. It does not copy workflow state from the old backend.

Before returning to a previously used backend, remove its old workflow state. For DBOS, drop the installation's _dbos database while the control plane is stopped. For Temporal, use a new namespace. Otherwise, old workflows can restart with outdated pipeline definitions.

MonitoringCopy Anchor Link

openetld exports metrics and traces through OpenTelemetry. Configure an exporter or OTLP endpoint with the standard OTEL_* variables to enable export. OTEL_METRICS_EXPORTER=prometheus serves metrics for Prometheus to scrape. The Kubernetes manifests expose this endpoint on port 9464.

By default, logs use JSON on standard error. OPENETL_LOG_FORMAT and OPENETL_LOG_OUTPUT control their format and destination.

Monitor an installation explains the metrics and how to choose alerts. Troubleshoot a pipeline connects symptoms to checks and recovery.

Operating the installationCopy Anchor Link

• Manage access and credentials: users, scoped service accounts, key and runner-token rotation, and offboarding.
• Manage connection secrets: stored credentials, cloud-secret references, replacement and deletion.
• Back up and restore: the complete state inventory, a stopped Compose backup, and verification after recovery.

InstallCopy Anchor Link

• With Docker Compose, on one machine: an API server with the console, an orchestrator, runners and Postgres.
• On Kubernetes: manifests for API servers, orchestrators, shared runners and a pod launcher. The guide includes KEDA runner scaling and Karpenter node scaling.

These docs have no guide for other platforms. There, run the same roles as processes or containers, with the settings above. The release's Linux archives hold the programs for that.