Reference

This section links to the resource, command, setting and protocol references for this commit. Each entry identifies its documentation page, command help or source file. Source files define the behavior described by the documentation.

Resource documentsCopy Anchor Link

Everything you apply with openetl apply is a YAML or JSON document with an apiVersion, a kind, metadata and a spec:

apiVersion: openetl.io/v1alpha1
kind: Source
metadata:
  name: faker
spec:
  connector: faker
  config:
    seed: 42

• A file can hold several documents, separated by ---.
• metadata.name is lowercase letters, digits and hyphens, at most 63 characters, starting and ending with a letter or digit.
• metadata.workspace is optional. A document without one goes to the workspace openetl apply --workspace names, default unless it says.
• Fields a kind doesn't have are refused.
• In spec, enums are lowercase (merge), durations are Go durations (15m), and sizes have units (256MB).

Connector and DeclarativeSource also require metadata.version. Use a semantic version such as 1.2.0, without build metadata (+…). Their names can include an owner, such as acme/billing. OpenETL also accepts openetl.io/v1alpha1 connector descriptors as packages with one role.

Pipeline, Source and Destination list every field of those kinds' documents. The fields of each spec are defined by the messages in proto/openetl/api/v1, whose comments describe them, and read from YAML by internal/spec. The examples are working documents of each common kind. A Connector descriptor's fields are in Connector descriptor, and a DeclarativeSource manifest's in Declarative source manifest.

Choose and manage connectors covers package installation and version selection. Connector source lists the fields for adding a public or private package catalog.

Webhook, dbt project, Spark cluster and Flink cluster complete the resource references. Each field table separates required fields, conditional requirements and defaults.

The openetl CLICopy Anchor Link

openetl is the command-line client of the API. Install the CLI shows how to install it. Run openetl for its commands, and openetl COMMAND -h for a command's flags.

openetl connector has commands of its own:

TARGET is an executable, a descriptor, a manifest, or a package version by reference: oci://REGISTRY/REPOSITORY:VERSION or github.com/OWNER/REPO[/PACKAGE]@VERSION.

Commands that call the API use these connection settings:

OPENETL_API_KEY supplies the key directly and takes precedence over the key file. The key is sent in the Authorization: Bearer KEY header.

The commands and their help are in cmd/openetl.

Pipeline CLI commands lists the arguments, flags and defaults for applying, inspecting and operating pipelines. Manage connection secrets covers stored values and cloud-secret references.

openetldCopy Anchor Link

openetld runs OpenETL's server side, in one of these roles:

openetld bootstrap-key gives the admin service account a new owner key, in --bootstrap-key-file, for when every owner key is lost.

Settings have flag and environment variable forms. For example, --database-url corresponds to OPENETL_DATABASE_URL. A flag overrides the environment variable, which overrides the default. Each role accepts all settings but uses only those relevant to that role. openetld ROLE -h lists settings, variables and defaults. The runner token can also use OPENETL_RUNNER_TOKEN, which has no corresponding flag.

openetld settings lists every setting with its flag, variable and default, and the OTEL_* variables that set up traces and metrics.

The APICopy Anchor Link

openetld server serves the API on its listen address over Connect, gRPC, gRPC-Web and REST. Calls carry an API key as Authorization: Bearer KEY; the console signs in with a session instead.

The services, their methods and their messages are defined in proto/openetl/api/v1. The OpenAPI description is generated from them into internal/api/openapi.yaml, and the server serves that file.

RolesCopy Anchor Link

A principal's roles are bound on the org, which covers every workspace, or on one workspace. Each role may do what the roles above it may, and more:

Each API method's role is in internal/api/authz.go.

JSON SchemasCopy Anchor Link

openetl connector schema prints three JSON Schemas, for editors and checks:

Their fields are listed in openetl-connector.yaml, Connector descriptor and Declarative source manifest.

Platform eventsCopy Anchor Link

OpenETL records these events, and webhooks deliver them:

Platform events lists every field of each type's data, from the JSON Schema in internal/events/event.schema.json.

The connector protocolCopy Anchor Link

External connectors implement the protocol in proto/openetl/connector/v1. It defines source and destination operations, host services and error types. The Go SDK implements this protocol. Build a connector explains the contracts and development workflow.

Built-in connectorsCopy Anchor Link

Built-in connectors has every config field of faker, stdout and parquet. Other connectors define their own config references and release them with their code. The official connectors' references are in openetl-connectors.