Pipeline CLI commands

These commands call the OpenETL API. The command definitions are in cmd/openetl. The CLI overview lists the other command groups.

Put flags before positional arguments. For a command group, put flags after the subcommand. NAME, PIPELINE, SOURCE and DESTINATION are resource names within a workspace, not full API paths. STREAM is the stream name reported by discovery or the pipeline.

Common flags and credentialsCopy Anchor Link

Every command on this page accepts these optional flags:

OPENETL_API_KEY takes precedence over the key file, even if you supply --api-key-file. The API URL flag takes precedence over OPENETL_API_URL. The key-file flag takes precedence over OPENETL_API_KEY_FILE. There is no --api-key flag.

Help is written to standard error. Successful commands return exit status 0. Errors return exit status 1. Plan warnings alone do not cause an error.

Apply and planCopy Anchor Link

Neither command accepts positional arguments. A document's metadata.workspace takes precedence over --workspace. Organization resources do not belong to a workspace.

After apply changes resources, it asks a runner to check them:

• Each source and destination that the apply created or updated.
• The source and destination of each pipeline that the apply created or updated, on the pipeline's data plane.

apply discovers each source that passes its check, which refreshes the catalog that plans read. Then it prints the problems in the changed pipelines' plans. A failed check or a plan problem prints a warning and does not change the exit status, because the applied resources stay. The checks stop after 2 minutes. --dry-run and --no-check skip them.

plan returns an error if a pipeline that would be created or changed has plan problems. Its output can also show problems on unchanged pipelines. See Plan, then apply for the review procedure.

Pause, resume and syncCopy Anchor Link

Stream pause is unavailable for CDC streams and pipelines with transforms. A paused stream and a paused pipeline have separate states. Resuming the pipeline does not resume streams paused individually. See Pause and resume.

Inspect syncs, versions and logsCopy Anchor Link

syncs shows a shortened error. Use logs to read the full message. A running sync can show failed attempts while recovery continues. A missing lag value is printed as -; it does not mean zero lag. See Troubleshoot a pipeline.

Reset and rewindCopy Anchor Link

Neither operation resumes a paused pipeline. Both operations are unavailable for pipelines with Flink transforms. A reset uses the connector's starting-position behavior. A rewind does not select an arbitrary point in history. See Reading from the start again for destination effects and recovery steps.

RescaleCopy Anchor Link

openetl rescale --size SIZE NAME changes the job size until the pipeline's next version. A running job restarts at a checkpoint.

The pipeline's resource settings define its normal size and limits. The data plane must have capacity for the requested size.

Inspect and remove overridesCopy Anchor Link

These commands have no flags beyond the common flags. Use the FIELD value from the list. Removing an override does not change the pipeline specification. See When heal slows syncs down.

Inspect, release and discard holdsCopy Anchor Link

Release and discard do not change destination data during the CLI request. See Handle schema changes and holds for repair and verification steps.

Check connections and discover streamsCopy Anchor Link

Put flags before source, destination or pipeline for check. check pipeline returns an error if a check fails or the plan has problems. These commands use connections already applied to the API. To test a connector locally with a config file, use the connector CLI.