Run dbt after a sync

Add a dbt step when you want to build or test models after OpenETL loads data. The runner executes your project against the pipeline's Postgres destination. dbt steps support scheduled pipelines only.

The copied data commits before dbt starts. A failed model or test fails the sync, but the copied data remains in the destination.

Prepare the runner and destinationCopy Anchor Link

You need:

• A Postgres destination and the official Postgres connector.
• A destination DSN with one host and an explicit user. Both connection URLs and libpq keyword/value strings are accepted.
• Permission for that user to read the loaded tables and create or update the dbt target schema and models.
• dbt-core and dbt-postgres installed on every runner that can receive the step, including job pods.

For a source-build runner, install dbt in a virtual environment. From the repository root:

python3 -m venv .openetl/dbt-venv
.openetl/dbt-venv/bin/python -m pip install dbt-core==1.12.6 dbt-postgres==1.11.0
.openetl/dbt-venv/bin/dbt --version
export OPENETL_DBT="$PWD/.openetl/dbt-venv/bin/dbt"

These are the versions used by the repository's end-to-end CI. The export sets the executable's absolute path for a runner you start from this shell. The equivalent flag is --dbt. Without either setting, the runner finds dbt on its PATH.

The executable runs inside the runner's environment. A dbt installation on the CLI's computer is insufficient when runners are on other machines.

Prepare the projectCopy Anchor Link

Use examples/dbt as a working project. Its dbt_project.yml names the project and profile:

name: faker
version: "1.0"
config-version: 2
profile: faker
models:
  faker:
    +materialized: table

The profile name must be plain text. OpenETL creates that profile for each step. Your local profiles.yml is not uploaded.

Use OPENETL_NAMESPACE to locate the tables that OpenETL loaded. From the example's sources.yml, with comments omitted:

version: 2
sources:
  - name: faker
    schema: "{{ env_var('OPENETL_NAMESPACE') }}"
    tables:
      - name: users
      - name: purchases

The spend_per_user model reads purchases through source('faker', 'purchases'). It removes repeated purchase IDs before calculating totals.

If your project uses packages, resolve them before applying it. Include the installed dbt_packages/ directory. Runners do not run dbt deps.

The CLI excludes hidden files, root target/, root logs/ and root profiles.yml when packaging the project. Symlinks and other nonregular files are refused. The archive must fit within 16 MiB compressed and 256 MiB unpacked, with at most 20,000 files.

Attach the project to a pipelineCopy Anchor Link

The complete document is examples/faker-to-postgres-dbt.yaml. These parts register the project and select its step:

apiVersion: openetl.io/v1alpha1
kind: DbtProject
metadata:
  name: faker-dbt
spec:
  path: dbt

postSync:
  - dbt:
      project: faker-dbt
      schema: faker_marts

path is relative to the file that contains the DbtProject, so this example packages examples/dbt. A document read from standard input resolves relative paths from your current directory.

The project and pipeline must belong to the same workspace. API clients supply an encoded bundle instead of path. See the dbt project reference.

Check the connection and applyCopy Anchor Link

For this example, first store the destination DSN under local-postgres-dsn. Use Manage connection secrets to create it.

From the repository root:

openetl plan -f examples/faker-to-postgres-dbt.yaml
openetl apply -f examples/faker-to-postgres-dbt.yaml
openetl check destination local-postgres
openetl syncs faker-to-postgres-dbt
openetl logs -f faker-to-postgres-dbt

Applying a new scheduled pipeline starts its first sync. Wait for that sync to finish before requesting another one.

After it succeeds, connect to the destination with your SQL client and check the model:

select user_id, purchases, spend
from faker_marts.spend_per_user
order by user_id
limit 10;

Check the model's values as well as the sync state. The sync log includes dbt output and a dbt finished entry with node counts.

Choose the command and target schemaCopy Anchor Link

The default command is build. Use run to run models or test to run tests. select and exclude pass space-separated selectors to dbt. The pipeline reference lists these fields.

The generated profile has one target, named openetl, with four threads. It uses the destination's host, database, user, password and supported TLS settings.

Keep these schemas separate in your project:

OpenETL supplies connection values through environment variables used by the generated profile. It does not pass arbitrary runner environment variables to dbt. The inherited variables are PATH, HOME, TMPDIR, TZ, LANG, SSL_CERT_FILE and SSL_CERT_DIR.

Update and recoverCopy Anchor Link

Apply the project again after changing its files. A project update creates new dependent pipeline versions. A sync already started keeps its pinned project version.

Steps run in list order. A failed step prevents later steps from running. A dbt command failure reports DBT_FAILED; OpenETL does not retry that command automatically.

After a repair, request a new sync:

openetl sync faker-to-postgres-dbt

This starts a complete sync. There is no CLI operation that reruns only its failed dbt step. Account for the pipeline's write mode when another sync can append the same source rows.

When you finish the example, pause the pipeline:

openetl pause faker-to-postgres-dbt

The copied tables and dbt models remain available for inspection.