Source
Every field of a Source document, which names the connector a pipeline reads with and holds its config.
A Source is a system pipelines read from: a connector, and that connector's config. Pipelines name it in source.ref. Pipelines as code shows how the documents fit together.
The Required column applies within the containing object or list entry. Optional objects may be omitted; when you include one, its required children must be set. Conditional requirements are stated in the table. Default describes what omission means.
apiVersion: openetl.io/v1alpha1
kind: Source
metadata:
name: demo-data
spec:
connector: faker
config:
seed: 42
count: 1000The document
| Field | Type | Required | Default | Meaning |
|---|---|---|---|---|
apiVersion | string | Yes | — | openetl.io/v1alpha1 |
kind | string | Yes | — | Source |
metadata.name | name | Yes | — | The source's name: lowercase letters, digits and hyphens, at most 63 characters, starting and ending with a letter or digit |
metadata.workspace | name | No | The workspace openetl apply --workspace names, default unless it says | The workspace it belongs to |
spec | object | Yes | — | The fields below |
A field the document doesn't have is refused.
spec
| Field | Type | Required | Default | Meaning |
|---|---|---|---|---|
connector | connector reference | Yes | — | The connector to read with, and which of its versions. See connector references. |
config | object | Depends on the connector's config schema | None | The connector's config. Its fields are the connector's own: its spec's JSON Schema defines them, which openetl connector spec TARGET prints. A config never holds a secret's value: a field holds { secret: NAME } instead, naming a secret in the same workspace, which must exist when you apply. Where the connector's spec marks a field as secret, the config must name a secret for it. |
aws | object | No | None | An IAM role that the runner assumes for the connector, with the runner's own AWS credentials. The connector then gets that role's credentials and no other AWS credentials from the runner. The runner must allow the role in OPENETL_AWS_ASSUME_ROLES. A built-in connector cannot take a role. Run on AWS explains the setup. |
aws.role | string | Yes, in aws | — | The role's ARN, such as arn:aws:iam::123456789012:role/openetl-lake. The runner tags each session with openetl:workspace, openetl:source (this source's name) and, during a sync, openetl:pipeline. |
aws.externalId | string | No | None | The ExternalId that the role's trust policy requires: 2 to 1224 letters, digits and the characters _+=,.@:/-. |
Connector references
connector is a connector's package name, with a version after @ if you want one. acme/billing below is an example name, not a published connector:
| Reference | What it names |
|---|---|
acme/billing | The highest stable version of acme/billing applied to the control plane, or the highest prerelease if there is no stable version. With no versions applied, the runner looks for acme/source-billing in its connectors directory. |
acme/billing@1, acme/billing@1.4 | The highest stable release with that major, or major and minor, version |
acme/billing@1.2.0 | That version |
acme/billing@sha256:DIGEST | The version with that digest |
billing | Shorthand for openetl/billing. A name without an owner is openetl's. With no versions applied, the runner looks for source-billing. |
Each apply resolves the reference again, so a reference without an exact version or digest may pin a newer version than the last apply did. The built-in source, faker (also openetl/faker), has no versions, so its reference takes none. Built-in connectors has its config.
Connector versions are independent of OpenETL versions. Resolution uses packages already applied to the org, not every release in a remote registry. Choose and manage connectors covers installation and controlled upgrades.