Connections

A Connection is a piece of credentialised configuration the control plane encrypts at rest (ADR 0019) and delivers to a running task as AIRFLOW_CONN_<CONN_ID> (ADR 0021). User code (Python psycopg2, requests, your operator of choice) reads the env var and connects.

This page is the index. Each linked entry below is a focused recipe with the URI shape, an example DAG, and how to test it.

Installing a connector’s provider

A Connection only carries credentials. To use a connector — whether through its Airflow hook (PostgresHook) or a raw driver (psycopg2) — the matching Python package has to be in the image / venv. Leoflow gives you two ways to declare that in leoflow.yaml, and you pick whichever fits:

• connectors: (the sugar)
• dependencies: (the escape hatch)

dag_id: my_pipeline
connectors:
  - postgres
  - http

dag_id: my_pipeline
dependencies:
  - apache-airflow-providers-postgres==6.0.0   # pin the provider, or…
  - psycopg2-binary==2.9.10                    # …just the driver, your call

Both lists are additive — the effective install is expand(connectors) + dependencies. A name in connectors: that isn’t in the catalog fails the compile with the offender, the known list, and a pointer to dependencies:, so a typo never slips through to a runtime ModuleNotFoundError in the task pod.

@task
def load():
    from airflow.providers.postgres.hooks.postgres import PostgresHook  # here
    hook = PostgresHook(postgres_conn_id="pg_target")
    ...

Connector → provider package

The catalog is generated from a real Airflow install (scripts/gen_connectors.py → internal/connectors/catalog.json, ADR 0039), so ~86 connector types ship with a dropdown entry, the correct standard-field behavior, and the provider-specific custom fields served to the form (e.g. Snowflake’s account/warehouse, the GCP keyfile) in the exact param-spec shape the Airflow 3.2 FlexibleForm consumes. It is the single source of truth shared by the admin form, the sugar expansion, and compile validation.

The common head:

The package boundary is not a mechanical join of the dotted hook path (amazon.aws → amazon; microsoft.mssql → microsoft-mssql), which is exactly why the mapping is curated rather than derived.

When you press Test on a connection of a known type, the result line also reminds you to declare the provider (connectors: [<type>]) — a Connection in the UI carries credentials, but the DAG still has to install the hook to use them.

Connectors by category

Every documented connector grouped by domain — its conn_type, the provider package the connectors: sugar expands to, and its category. Click a name for the full recipe (URI shape, example DAG, how to test).

Databases

Cloud

Messaging

BI & SaaS

Files

Connector matrix

Every documented connector, its conn_type, the provider package the connectors: sugar expands to (ADR 0038), the auth shape (where the credential lives), the local-testable tier, and a runnable example. Each ships a chain-of-custody delivery test; the tier-1 rows also ship an automated example test (see the contract below). Cloud families share a provider package (e.g. redshift/athena/emr → apache-airflow-providers-amazon; gcpbigquery/gcpcloudsql/gcp_looker → …-google); the boundary is curated, not a mechanical join of the hook path.

Local tier — local (Docker): spin the service up with Docker/Lima and run the example end-to-end; local (tier 1): no external service needed (or already in CI); doc-only: a real run needs the external account/service, so the page is a recipe with a delivery test rather than a runnable example. Example — recipe links back to the connector’s own page (the copy-paste DAG lives there).

The long tail

Every curated connector above is first-class (impl + delivery test + cookbook). Beyond them, all ~86 connector types in the generated catalog (ADR 0039) are usable today via connectors: (dropdown + sugar) — they just don’t each have a dedicated cookbook recipe yet. Any other Airflow provider works via dependencies:. Recipes are added connector-by-connector as they are promoted.

Contract every entry honours

Every entry in this cookbook ships with all three of:

1. A doc page (this dir) covering: URI shape, default port, Lite-vs-Pro caveats, security notes (TLS, auth modes).
2. An example DAG under examples/<type>_*/ with its own README.md showing how to spin up the dependency (Docker / Lima), how to create the Connection in the UI, and the expected end-state of the target system.
3. An automated test that proves the delivery chain — the integration test under internal/storage/ (companion to the example) gated by the integration build tag, runs in CI against a real Postgres.

If a layer is mocked or the e2e test is manual-only, the doc says so and a follow-up issue covers the gap.

Related

• ADR 0019 — secret encryption at rest.
• ADR 0021 — agent secret delivery.
• #67 — connectors umbrella.
• #142 — the cookbook umbrella (this page).