# OpenETL

- v0.1.0-alpha: The newest release
  - [OpenETL documentation](https://docs.openetl.io/docs/v0-1): Guides and reference for using, running, integrating and extending OpenETL.
  - Get started: What OpenETL is, a first pipeline, the CLI, and the words it uses
    - [What is OpenETL?](https://docs.openetl.io/docs/v0-1/get-started/overview): What OpenETL does, who it's for, and how connectors link your systems.
    - [Quickstart](https://docs.openetl.io/docs/v0-1/get-started/quickstart): Run the OpenETL release on your computer with Docker Compose, copy generated rows into Postgres, and check the result in the console and the database.
    - [Install the CLI](https://docs.openetl.io/docs/v0-1/get-started/install): Install the openetl command-line tool on macOS or Linux, check the download, connect it to an installation, and upgrade or remove it.
    - [Concepts](https://docs.openetl.io/docs/v0-1/get-started/concepts): The words OpenETL uses, what each one means, and how they fit together.
  - Self-host: Install, run, scale and upgrade OpenETL
    - [Self-host](https://docs.openetl.io/docs/v0-1/self-host): What an OpenETL installation is made of, what it needs, and the settings every installation sets.
    - [Install with Docker Compose](https://docs.openetl.io/docs/v0-1/self-host/compose): Run OpenETL on one machine with Docker Compose, with its API server and console, an orchestrator, runners and Postgres.
    - [Install on Kubernetes](https://docs.openetl.io/docs/v0-1/self-host/kubernetes): Run OpenETL's API servers, orchestrators, runners and pod launcher on a Kubernetes cluster, and scale them with KEDA and Karpenter.
    - [Run on AWS](https://docs.openetl.io/docs/v0-1/self-host/aws): Give OpenETL's runners and connectors AWS access through IAM roles on EC2 and EKS, and see what a connector can and cannot get from the runner.
    - [Manage access and credentials](https://docs.openetl.io/docs/v0-1/self-host/access): Add users and service accounts, scope their permissions, rotate keys and runner tokens, and remove access.
    - [Manage connection secrets](https://docs.openetl.io/docs/v0-1/self-host/secrets): Store connection credentials, use cloud-secret references, replace values and remove unused secrets.
    - [Monitor an installation](https://docs.openetl.io/docs/v0-1/self-host/monitoring): Export metrics and traces, interpret pipeline and placement signals, and choose alerts for stalled data delivery.
    - [Back up and restore](https://docs.openetl.io/docs/v0-1/self-host/backup-and-restore): Make a stopped backup of the default Compose installation and restore it into fresh volumes.
  - Pipelines: Write, plan and apply pipelines as code
    - [Pipelines as code](https://docs.openetl.io/docs/v0-1/pipelines): How pipelines are written as YAML, planned and applied with the CLI, versioned, and run day to day.
    - [Your first pipeline](https://docs.openetl.io/docs/v0-1/pipelines/first-pipeline): Copy tables from your own Postgres database into another, from checking the connection to the first sync.
    - [Choose and manage connectors](https://docs.openetl.io/docs/v0-1/pipelines/connectors): Find, install, pin and upgrade official, community and private source and destination connectors.
    - [Delivery guarantees](https://docs.openetl.io/docs/v0-1/pipelines/delivery): Understand delivery guarantees, review their requirements, and remove duplicate changes when reading data.
    - [Schedules](https://docs.openetl.io/docs/v0-1/pipelines/schedules): Configure intervals, cron times and jitter, and understand delayed, paused and missed syncs.
    - [Reading from the start again](https://docs.openetl.io/docs/v0-1/pipelines/positions): Reset stream positions, recover from an unreadable checkpoint, and monitor source retention while a pipeline is stopped.
    - [Handle schema changes and holds](https://docs.openetl.io/docs/v0-1/pipelines/schema-changes): Inspect a schema change, repair and release a hold, or choose a new column or discard its held values.
    - [Troubleshoot a pipeline](https://docs.openetl.io/docs/v0-1/pipelines/troubleshooting): Diagnose waiting jobs, failed syncs, stale data and connection errors, then verify recovery.
  - Use the console: For everyone who depends on the data: freshness, syncs, incidents and schema changes
    - [Use the console](https://docs.openetl.io/docs/v0-1/use): A tour of OpenETL's console, for checking whether your data is fresh, what each sync did, and what needs attention.
  - Integrate: Connect dbt, Spark, Flink, the API, webhooks and your monitoring
    - [Integrate](https://docs.openetl.io/docs/v0-1/integrate): Connect OpenETL to dbt, Spark, Flink, your own code and your monitoring, and read what it writes.
    - [Run dbt after a sync](https://docs.openetl.io/docs/v0-1/integrate/dbt): Package a dbt project, run it against a pipeline's destination, verify its models and diagnose failed steps.
    - [Transform data with Spark](https://docs.openetl.io/docs/v0-1/integrate/spark): Build a Spark transform, configure submission and shared storage, and verify or recover its pipeline.
    - [Transform a stream with Flink](https://docs.openetl.io/docs/v0-1/integrate/flink): Build a Flink job, connect it to a runner, configure checkpoints and verify a continuous pipeline.
    - [Receive and verify webhooks](https://docs.openetl.io/docs/v0-1/integrate/webhooks): Verify event signatures, prevent duplicate processing, inspect delivery failures and rotate webhook signing keys.
  - Build a connector: Write, test, package and publish connectors
    - [Build a connector](https://docs.openetl.io/docs/v0-1/build-connectors): What a connector is to OpenETL, the kinds you can write, and how to build, test, package and publish one.
  - Reference: Specs, commands, settings, events and schemas
    - [Reference](https://docs.openetl.io/docs/v0-1/reference): Where to look up every resource, command, setting, API, event and schema, and the source in the code each one comes from.
    - Resource specs
      - [Pipeline](https://docs.openetl.io/docs/v0-1/reference/specs/pipeline): Every field of a Pipeline document, with its type, required or optional status, default and meaning.
      - [Source](https://docs.openetl.io/docs/v0-1/reference/specs/source): Every field of a Source document, which names the connector a pipeline reads with and holds its config.
      - [Destination](https://docs.openetl.io/docs/v0-1/reference/specs/destination): Every field of a Destination document, which names the connector a pipeline writes with and holds its config.
      - [Connector source](https://docs.openetl.io/docs/v0-1/reference/specs/connector-source): Every field of a ConnectorSource document, which connects a public or private package catalog to an org.
      - [Webhook](https://docs.openetl.io/docs/v0-1/reference/specs/webhook): Every field of a Webhook document, including required signing secrets and optional pipeline filters.
      - [dbt project](https://docs.openetl.io/docs/v0-1/reference/specs/dbt-project): The required project input and bundle format of a DbtProject document.
      - [Spark cluster](https://docs.openetl.io/docs/v0-1/reference/specs/spark-cluster): Every field of a SparkCluster document, with required target choices and configuration defaults.
      - [Flink cluster](https://docs.openetl.io/docs/v0-1/reference/specs/flink-cluster): Every field of a FlinkCluster document, including target choices, credentials and resource defaults.
    - [Pipeline CLI commands](https://docs.openetl.io/docs/v0-1/reference/pipeline-cli): Look up the arguments, flags and defaults for applying, inspecting and operating pipelines with openetl.
    - [openetld settings](https://docs.openetl.io/docs/v0-1/reference/openetld): Every setting of openetld, as a flag and an environment variable, with its default.
    - [openetl-connector.yaml](https://docs.openetl.io/docs/v0-1/reference/connector-file): Every field of the file a connector package is built from.
    - [Connector descriptor](https://docs.openetl.io/docs/v0-1/reference/connector-descriptor): Every field of a Connector document, which describes one version of a connector package.
    - [Declarative source manifest](https://docs.openetl.io/docs/v0-1/reference/manifest): Every field of a DeclarativeSource manifest.
    - [Platform events](https://docs.openetl.io/docs/v0-1/reference/events): Every platform event's type, and every field of its data.
    - [Built-in connectors](https://docs.openetl.io/docs/v0-1/reference/built-in-connectors): The connectors built into OpenETL, the faker source and the stdout and parquet destinations, with every config field and what each one supports.
    - API
      - Audit Service: AuditService reads the audit log: every call that changes something or
 reads secrets' values, who made it, and how it ended.
        - [List audit entries](https://docs.openetl.io/docs/v0-1/reference/api/auditservice/list-audit-entries): Newest first.
      - Access Service: AccessService manages who may call the API: principals, their keys and
 roles. It also makes data planes' runner tokens, which only
 DataPlaneService takes.
        - [Get caller](https://docs.openetl.io/docs/v0-1/reference/api/accessservice/get-caller): Who is calling, and what they may do. Any credential may ask.
        - [Change password](https://docs.openetl.io/docs/v0-1/reference/api/accessservice/change-password): Changes the caller's own password, and ends their other sessions.
        - [List data planes](https://docs.openetl.io/docs/v0-1/reference/api/accessservice/list-data-planes)
        - [List runner tokens](https://docs.openetl.io/docs/v0-1/reference/api/accessservice/list-runner-tokens)
        - [Create runner token](https://docs.openetl.io/docs/v0-1/reference/api/accessservice/create-runner-token): The response has the token itself, which the API never returns again.
        - [Delete runner token](https://docs.openetl.io/docs/v0-1/reference/api/accessservice/delete-runner-token): The token stops working at once.
        - [List principals](https://docs.openetl.io/docs/v0-1/reference/api/accessservice/list-principals)
        - [Create principal](https://docs.openetl.io/docs/v0-1/reference/api/accessservice/create-principal)
        - [Get principal](https://docs.openetl.io/docs/v0-1/reference/api/accessservice/get-principal)
        - [Delete principal](https://docs.openetl.io/docs/v0-1/reference/api/accessservice/delete-principal): Deletes its keys and role bindings with it. Fails with
 FAILED_PRECONDITION for the org's last owner.
        - [List keys](https://docs.openetl.io/docs/v0-1/reference/api/accessservice/list-keys)
        - [Create key](https://docs.openetl.io/docs/v0-1/reference/api/accessservice/create-key): The response has the key itself, which the API never returns again.
        - [Delete key](https://docs.openetl.io/docs/v0-1/reference/api/accessservice/delete-key): The key stops working at once.
        - [Disable principal](https://docs.openetl.io/docs/v0-1/reference/api/accessservice/disable-principal): Its keys stop working until it's enabled again. Fails with
 FAILED_PRECONDITION for the org's last owner.
        - [Enable principal](https://docs.openetl.io/docs/v0-1/reference/api/accessservice/enable-principal)
        - [Grant role](https://docs.openetl.io/docs/v0-1/reference/api/accessservice/grant-role): Granting a role the principal holds already changes nothing.
        - [Create password link](https://docs.openetl.io/docs/v0-1/reference/api/accessservice/create-password-link): A one-use link for a user to set their password, which replaces the
 user's previous link. The user needs an email to sign in with.
        - [Revoke role](https://docs.openetl.io/docs/v0-1/reference/api/accessservice/revoke-role): Fails with FAILED_PRECONDITION for the org's last owner.
        - [Revoke sessions](https://docs.openetl.io/docs/v0-1/reference/api/accessservice/revoke-sessions): Ends every session of a user.
      - Connector Service: ConnectorService lists connected sources of connectors and the catalog of
 packages they hold, and deletes sources. ApplyService connects them, and
 applies connector versions, by reference too.
        - [List catalog](https://docs.openetl.io/docs/v0-1/reference/api/connectorservice/list-catalog): Each package of every connected source, at its highest version, as its
 source last listed it, and each package the org applied that no source
 lists, at its highest applied version.
        - [Get catalog package](https://docs.openetl.io/docs/v0-1/reference/api/connectorservice/get-catalog-package): A package with its roles' config fields, from its highest version's
 descriptor, which the first call fetches.
        - [Refresh catalog](https://docs.openetl.io/docs/v0-1/reference/api/connectorservice/refresh-catalog): Lists every connected source now, as the control plane does every 15
 minutes.
        - [List connector sources](https://docs.openetl.io/docs/v0-1/reference/api/connectorservice/list-connector-sources)
        - [Delete connector source](https://docs.openetl.io/docs/v0-1/reference/api/connectorservice/delete-connector-source): Its packages leave the catalog. Versions applied from it stay.
      - Dbt Project Service: DbtProjectService reads and deletes dbt projects. ApplyService creates and
 changes them.
        - [List dbt projects](https://docs.openetl.io/docs/v0-1/reference/api/dbtprojectservice/list-dbt-projects)
        - [Get dbt project](https://docs.openetl.io/docs/v0-1/reference/api/dbtprojectservice/get-dbt-project)
        - [Delete dbt project](https://docs.openetl.io/docs/v0-1/reference/api/dbtprojectservice/delete-dbt-project): Fails with FAILED_PRECONDITION while a pipeline's current version runs
 it. Older versions keep the bundles they pin.
      - Destination Service: DestinationService reads and deletes destinations. ApplyService creates and
 changes them.
        - [List destinations](https://docs.openetl.io/docs/v0-1/reference/api/destinationservice/list-destinations)
        - [Get destination](https://docs.openetl.io/docs/v0-1/reference/api/destinationservice/get-destination)
        - [Delete destination](https://docs.openetl.io/docs/v0-1/reference/api/destinationservice/delete-destination)
        - [Check destination](https://docs.openetl.io/docs/v0-1/reference/api/destinationservice/check-destination): Runs the destination's Check on a runner of the data plane, and waits
 for it.
      - Flink Cluster Service: FlinkClusterService reads and deletes Flink clusters. ApplyService creates
 and changes them.
        - [List flink clusters](https://docs.openetl.io/docs/v0-1/reference/api/flinkclusterservice/list-flink-clusters)
        - [Get flink cluster](https://docs.openetl.io/docs/v0-1/reference/api/flinkclusterservice/get-flink-cluster)
        - [Delete flink cluster](https://docs.openetl.io/docs/v0-1/reference/api/flinkclusterservice/delete-flink-cluster): Fails with FAILED_PRECONDITION while a pipeline's current version uses
 it. Older versions keep the settings they pin.
      - Pipeline Service: PipelineService reads and deletes pipelines. ApplyService creates and
 changes them.
        - [List pipelines](https://docs.openetl.io/docs/v0-1/reference/api/pipelineservice/list-pipelines)
        - [Get pipeline](https://docs.openetl.io/docs/v0-1/reference/api/pipelineservice/get-pipeline)
        - [Delete pipeline](https://docs.openetl.io/docs/v0-1/reference/api/pipelineservice/delete-pipeline)
        - [List checkpoints](https://docs.openetl.io/docs/v0-1/reference/api/pipelineservice/list-checkpoints): The checkpoints the pipeline's syncs committed, newest first. The
 control plane keeps 30 days of them.
        - [List holds](https://docs.openetl.io/docs/v0-1/reference/api/pipelineservice/list-holds): The columns and cells the pipeline holds back, as its last
 checkpoint left them.
        - [Discard hold](https://docs.openetl.io/docs/v0-1/reference/api/pipelineservice/discard-hold): Discards a hold: its held values are dropped, and a column held whole
 stays as the destination has it.
        - [Release hold](https://docs.openetl.io/docs/v0-1/reference/api/pipelineservice/release-hold): Releases a hold: its held values are patched in, and a column held
 whole is delivered again from the next run. A continuous pipeline's run
 restarts at a checkpoint to take it.
        - [Get pipeline plan](https://docs.openetl.io/docs/v0-1/reference/api/pipelineservice/get-pipeline-plan): How the current version syncs, from the source's latest catalog: each
 stream's delivery level and what sets it. The plan has no changes.
        - [List stream schemas](https://docs.openetl.io/docs/v0-1/reference/api/pipelineservice/list-stream-schemas): Each stream's schema history, by stream, then version.
        - [List pipeline versions](https://docs.openetl.io/docs/v0-1/reference/api/pipelineservice/list-pipeline-versions): Newest first.
        - [Get pipeline version](https://docs.openetl.io/docs/v0-1/reference/api/pipelineservice/get-pipeline-version)
        - [Pause pipeline](https://docs.openetl.io/docs/v0-1/reference/api/pipelineservice/pause-pipeline): Stops scheduled syncs. A sync that is running carries on. With a
 stream, only that stream stops: syncs skip it and keep its state for
 when it's resumed, and a continuous pipeline's run restarts at a
 checkpoint without it. Fails with FAILED_PRECONDITION for a stream that
 can't stop on its own: one read with cdc, which shares the source's
 change log with the pipeline's other cdc streams; a transform's input;
 or the last stream that isn't paused.
        - [Remove pipeline override](https://docs.openetl.io/docs/v0-1/reference/api/pipelineservice/remove-pipeline-override): Removes one of heal's overrides, which gives the spec back its say in
 that field.
        - [Rescale pipeline](https://docs.openetl.io/docs/v0-1/reference/api/pipelineservice/rescale-pipeline): Sets the size of the pipeline's jobs until its next version, or gives
 the spec back its say. A running job stops at a checkpoint and starts
 again at the new size.
        - [Reset pipeline](https://docs.openetl.io/docs/v0-1/reference/api/pipelineservice/reset-pipeline): Reads streams again from the start: the next sync, or a continuous
 pipeline's next run, drops their positions and reads them as it would a
 new stream's, such as a new replication slot and a snapshot. What
 reaches the destination already stays: merge updates rows in place, and
 append adds them again. A continuous pipeline's run restarts at a
 checkpoint to take it. Fails with FAILED_PRECONDITION for a pipeline
 with a Flink transform, whose state Flink keeps.
        - [Resume pipeline](https://docs.openetl.io/docs/v0-1/reference/api/pipelineservice/resume-pipeline)
        - [Rewind pipeline](https://docs.openetl.io/docs/v0-1/reference/api/pipelineservice/rewind-pipeline): Lets the next sync or run restart from the checkpoint before the latest
 one if the latest can't be read. Its epoch was delivered, so it's
 delivered again: streams the destination neither merges nor dedupes get
 its rows twice. Without this, a restart happens by itself only where it
 can't lower a stream's guarantee, under heal mode auto or auto_safe.
        - [Sync pipeline](https://docs.openetl.io/docs/v0-1/reference/api/pipelineservice/sync-pipeline): Starts a sync now. Fails with FAILED_PRECONDITION if the pipeline is
 paused or a sync is running.
      - Incident Service: IncidentService reads incidents and resolves them by hand. The control
 plane opens them, and resolves most once their pipeline makes progress
 again.
        - [List incidents](https://docs.openetl.io/docs/v0-1/reference/api/incidentservice/list-incidents): Newest first. The parent workspaces/{workspace}/pipelines/- lists every
 pipeline's.
        - [Get incident](https://docs.openetl.io/docs/v0-1/reference/api/incidentservice/get-incident)
        - [Resolve incident](https://docs.openetl.io/docs/v0-1/reference/api/incidentservice/resolve-incident): Resolves an open incident, such as one fixed in a way the control plane
 can't see. If it happens again, a new one opens.
      - Sync Service: SyncService reads pipelines' syncs and their logs. A continuous
 pipeline's syncs are its runs: each runs one version until it is stopped
 or fails.
        - [List syncs](https://docs.openetl.io/docs/v0-1/reference/api/syncservice/list-syncs): Newest first.
        - [Get sync](https://docs.openetl.io/docs/v0-1/reference/api/syncservice/get-sync)
        - [Read sync log](https://docs.openetl.io/docs/v0-1/reference/api/syncservice/read-sync-log): A page of the sync's log, oldest first: each attempt's, then each
 post-sync step's. The log is in the data plane that ran the sync, so a
 runner of that data plane reads it; UNAVAILABLE if none does in time.
 Data planes keep logs for 30 days.
      - Secret Service: SecretService stores the secrets that sources' and destinations' configs
 name, as {"secret": "NAME"}, in their workspace. A secret holds a value,
 encrypted at rest, or a reference to a cloud secret store, which runners
 read with their own cloud identity. The API never returns values.
        - [List secrets](https://docs.openetl.io/docs/v0-1/reference/api/secretservice/list-secrets)
        - [Get secret](https://docs.openetl.io/docs/v0-1/reference/api/secretservice/get-secret)
        - [Update secret](https://docs.openetl.io/docs/v0-1/reference/api/secretservice/update-secret): Sets a secret's value or reference. The next sync, check or discover
 that uses it gets the new value.
        - [Delete secret](https://docs.openetl.io/docs/v0-1/reference/api/secretservice/delete-secret): Fails with FAILED_PRECONDITION while a source or destination names it.
      - Source Service: SourceService reads and deletes sources. ApplyService creates and changes
 them.
        - [List sources](https://docs.openetl.io/docs/v0-1/reference/api/sourceservice/list-sources)
        - [Get source](https://docs.openetl.io/docs/v0-1/reference/api/sourceservice/get-source)
        - [Delete source](https://docs.openetl.io/docs/v0-1/reference/api/sourceservice/delete-source)
        - [Check source](https://docs.openetl.io/docs/v0-1/reference/api/sourceservice/check-source): Runs the source's Check on a runner of the data plane, and waits for it.
        - [Discover source](https://docs.openetl.io/docs/v0-1/reference/api/sourceservice/discover-source): Runs the source's Discover on a runner of the data plane, and waits for
 the catalog.
      - Spark Cluster Service: SparkClusterService reads and deletes Spark clusters. ApplyService creates
 and changes them.
        - [List spark clusters](https://docs.openetl.io/docs/v0-1/reference/api/sparkclusterservice/list-spark-clusters)
        - [Get spark cluster](https://docs.openetl.io/docs/v0-1/reference/api/sparkclusterservice/get-spark-cluster)
        - [Delete spark cluster](https://docs.openetl.io/docs/v0-1/reference/api/sparkclusterservice/delete-spark-cluster): Fails with FAILED_PRECONDITION while a pipeline's current version uses
 it. Older versions keep the settings they pin.
      - Webhook Service: WebhookService reads, deletes, disables and enables webhooks, sends them
 test events and lists their deliveries. ApplyService creates and changes
 them.
        - [List webhooks](https://docs.openetl.io/docs/v0-1/reference/api/webhookservice/list-webhooks)
        - [Get webhook](https://docs.openetl.io/docs/v0-1/reference/api/webhookservice/get-webhook)
        - [Delete webhook](https://docs.openetl.io/docs/v0-1/reference/api/webhookservice/delete-webhook): Deletes its deliveries too.
        - [List webhook deliveries](https://docs.openetl.io/docs/v0-1/reference/api/webhookservice/list-webhook-deliveries): Newest first. Deliveries are kept for 7 days.
        - [Disable webhook](https://docs.openetl.io/docs/v0-1/reference/api/webhookservice/disable-webhook): Its pending deliveries fail, and events until it's enabled again aren't
 delivered to it.
        - [Enable webhook](https://docs.openetl.io/docs/v0-1/reference/api/webhookservice/enable-webhook): Delivers events from now on, to a webhook disabled by a call or by
 failing.
        - [Test webhook](https://docs.openetl.io/docs/v0-1/reference/api/webhookservice/test-webhook): Records a webhook.test event for this webhook alone, whatever its
 filters.
      - Apply Service
        - [Apply](https://docs.openetl.io/docs/v0-1/reference/api/applyservice/apply): Stores each definition as an immutable version; unchanged definitions
 keep theirs. A pipeline's version pins the versions of its source and
 destination and its dbt projects' bundles. Either every document applies
 or none does.
  - Internals: How OpenETL works inside, for contributors and the curious
    - [How OpenETL works](https://docs.openetl.io/docs/v0-1/internals): OpenETL's parts, what each one does, and how a sync moves data and commits it, at the level of the code.

# Connectors

- [Amazon Kinesis](https://docs.openetl.io/connectors/kinesis): Read Kinesis data streams shard by shard, and write streams into Kinesis, a record per row, at least once.
- [Amazon S3](https://docs.openetl.io/connectors/s3): Write streams as Parquet files on S3, an S3-compatible store or a local disk, appending each epoch's files exactly once.
- [Apache Iceberg](https://docs.openetl.io/connectors/iceberg): Write streams into Iceberg tables through a REST catalog or AWS Glue, by appending, overwriting or merging, with each epoch committed once.
- [Kafka](https://docs.openetl.io/connectors/kafka): Read Kafka topics as streams, and write streams into topics in one transaction per epoch, in JSON, Avro or Protobuf.
- [PostgreSQL](https://docs.openetl.io/connectors/postgres): Read Postgres tables in full, by a cursor column, or change by change through logical replication, and write into Postgres by appending, overwriting or merging.