CI/CD & deploy examples

Deploying a Leoflow DAG is the same everywhere because a DAG is an immutable artifact — a dag.json + a container image, versioned together (ADR 0003).

By hand or for a team without a pipeline, one command does it all:

leoflow auth login --server "$LEOFLOW_SERVER"   # once; stores the token
leoflow deploy --yes                            # compile → build → push → register

leoflow deploy is the pipeline-less promotion — it cross-builds for the cluster, pins the image by digest, and registers the artifact. --yes skips the confirmation prompt (use it in automation).

In a pipeline, the same three boundaries are explicit so each can be cached, gated, and audited independently:

flowchart LR
  E[edit dag.py + leoflow.yaml] --> C[leoflow compile --build]
  C --> P[push image → registry]
  P --> R[leoflow push dag.json → control plane]

1. leoflow compile --build — parse dag.py, overlay leoflow.yaml, run the guardrails (unknown task_id, unsupported operator, duplicate keys), and build the DAG image.
2. push the image to your registry, tagged by git SHA (immutable).
3. leoflow push dag.json — register the artifact with the control plane.

Your repo of DAGs (CI is the recommended path)

In Pro you keep your DAGs in a Git repository — one directory per DAG — and CI turns each into an artifact when it changes. This is the recommended path: a push triggers the pipeline, no one builds by hand. Each DAG is just two files — no Dockerfile, no requirements.txt:

my-dags/                      # your Git repo
├── .github/workflows/
│   └── deploy-dag.yml        # the CI below (the recommended path)
└── dags/
    ├── my_pipeline/
    │   ├── dag.py            # the DAG (TaskFlow / operators)
    │   └── leoflow.yaml      # id, python_version, dependencies, registry
    └── another_pipeline/
        ├── dag.py
        └── leoflow.yaml

leoflow compile --build synthesizes the image from leoflow.yaml — FROM the published Leoflow base, your deps/connectors installed, your DAG copied in. No Dockerfile to maintain. (Ship your own Dockerfile only if you want full control; it is then used verbatim. Our examples ship one so you can docker build and inspect them by hand.)

The built image is your artifact — push it wherever you like (Docker Hub, ECR, Artifact Registry, ACR, GHCR, a private registry), via the registry: block in leoflow.yaml or --image. The only image Leoflow owns is the base your DAG layers on.

The mental model: a push that touches dags/my_pipeline/** triggers CI for that DAG only (the paths: filter), which compiles → builds (from yaml) → pushes the image to your registry → registers dag.json. The control plane runs the new version on the next trigger. One DAG per pipeline keeps blast radius small: a broken another_pipeline never blocks my_pipeline.

Prerequisites

• The leoflow CLI on the runner (download the release binary, or go install).
• Python 3.11+ on the runner (leoflow compile invokes the stdlib-only parser shim — ADR 0024 — to turn dag.py into dag.json). See Python on the runner below.
• A container registry your cluster can pull from.
• LEOFLOW_SERVER (control plane URL) and LEOFLOW_TOKEN (a push token) as CI secrets.

Python on the runner

The leoflow compile step needs Python 3.11, 3.12, or 3.13 to parse dag.py. Bring your own Python on the runner — do not rely on leoflow setup to download a managed CPython in CI (that path is designed for first-touch on a developer laptop, not for build pipelines, where it adds ~50 MB to every run and bypasses your runner’s pin/caching).

The recommended path on each runner type:

Older Python (≤3.10) fails the compile cleanly — leoflow compile errors out with the version requirement, not a confusing traceback. Newer Python (3.14+) is accepted by the upper end of the detection range; the range is bumped per release once the parser shim is re-verified against it.

One more step: leoflow setup extracts the parser

After the leoflow binary lands on the runner and Python is in scope, run leoflow setup ONCE per runner. The CLI ships the parser source embedded; setup extracts it under ~/.leoflow/pysrc/parser/ and writes a config file pointing the compile command at the chosen interpreter. Without this step leoflow compile fails with No module named leoflow_parser (the runner’s Python has no idea where the parser lives).

The snippets below all show leoflow setup as the step after the install, before leoflow compile. The follow-up to make this implicit (auto-bootstrap on first compile, or embed the parser execution inside the Go binary) is tracked separately; for now, calling it explicitly is the recommended path because it’s the operation that decides whether managed CPython is downloaded, and that’s a step CI operators should consciously opt into.

Examples

• GitHub Actions
• GitLab CI
• Google Cloud Build + Cloud Run

name: Deploy DAG
on:
  push:
    branches: [main]
    paths: ["dags/my_pipeline/**"]
jobs:
  deploy:
    runs-on: ubuntu-latest
    permissions: { contents: read, packages: write }
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5     # BYO Python — see #python-on-the-runner
        with: { python-version: '3.12' }
      - uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
      - name: Install leoflow
        run: curl -fsSL https://github.com/neochaotic/leoflow/releases/latest/download/leoflow-linux-amd64 -o /usr/local/bin/leoflow && chmod +x /usr/local/bin/leoflow
      - name: Bootstrap the parser (uses the BYO Python from above)
        run: leoflow setup
      - name: Compile + build + push image
        run: |
          IMAGE=ghcr.io/${{ github.repository }}/my_pipeline:${{ github.sha }}
          leoflow compile dags/my_pipeline --image "$IMAGE" --build --push -o dag.json
      - name: Register with the control plane
        env: { LEOFLOW_TOKEN: ${{ secrets.LEOFLOW_TOKEN }} }
        run: leoflow push dag.json --server ${{ secrets.LEOFLOW_SERVER }}

deploy_dag:
  # Default docker:27 is Alpine-based; install python3 before leoflow compile.
  # Alternative: a custom base image that bakes Python+Docker together.
  # See #python-on-the-runner for the rationale.
  image: docker:27
  services: [docker:27-dind]
  before_script:
    - apk add --no-cache python3   # 3.12 on Alpine 3.20+; see #python-on-the-runner
  rules:
    - if: $CI_COMMIT_BRANCH == "main"
      changes: ["dags/my_pipeline/**/*"]
  variables:
    IMAGE: $CI_REGISTRY_IMAGE/my_pipeline:$CI_COMMIT_SHA
  script:
    - echo "$CI_REGISTRY_PASSWORD" | docker login -u "$CI_REGISTRY_USER" --password-stdin "$CI_REGISTRY"
    - wget -qO /usr/local/bin/leoflow https://github.com/neochaotic/leoflow/releases/latest/download/leoflow-linux-amd64 && chmod +x /usr/local/bin/leoflow
    - leoflow setup        # extracts the parser into ~/.leoflow/ using the python3 from before_script
    - leoflow compile dags/my_pipeline --image "$IMAGE" --build --push -o dag.json
    - leoflow push dag.json --server "$LEOFLOW_SERVER"   # LEOFLOW_TOKEN from CI vars

steps:
  - name: gcr.io/cloud-builders/docker
    entrypoint: bash
    args:
      - -c
      - |
        # BYO Python — Cloud Builders' docker image is Debian; install python3.
        # See #python-on-the-runner for the rationale.
        apt-get update -qq && apt-get install -y --no-install-recommends python3
        curl -fsSL https://github.com/neochaotic/leoflow/releases/latest/download/leoflow-linux-amd64 -o /usr/bin/leoflow && chmod +x /usr/bin/leoflow
        leoflow setup    # extracts the parser into ~/.leoflow/ using the python3 just installed
        IMAGE="$_REGION-docker.pkg.dev/$PROJECT_ID/dags/my_pipeline:$SHORT_SHA"
        leoflow compile dags/my_pipeline --image "$$IMAGE" --build --push -o dag.json
        leoflow push dag.json --server "$_LEOFLOW_SERVER"
substitutions:
  _REGION: us-central1
  _LEOFLOW_SERVER: https://leoflow.run.app
options: { logging: CLOUD_LOGGING_ONLY }

• Generic / Makefile

leoflow setup    # one-shot per runner: extracts the parser into ~/.leoflow/
IMAGE="$REGISTRY/my_pipeline:$(git rev-parse --short HEAD)"
leoflow compile dags/my_pipeline --image "$IMAGE" --build --push -o dag.json
leoflow push dag.json --server "$LEOFLOW_SERVER" --token "$LEOFLOW_TOKEN"

Control-plane deployment (Helm chart, in validation)

Deploying the control plane itself (Helm chart, published leoflow-server/ leoflow-migrate images, TLS on the agent channel, keyless cloud auth) is the Pro track. One command installs the chart with auto-generated TLS and no cert-manager — from its published OCI artifact (helm install leoflow oci://ghcr.io/neochaotic/charts/leoflow --version <VERSION>), or from source on main for the bleeding edge (helm install lf ./helm/leoflow …). See Install Pro. The chart is installable today and in validation — see the Helm chart, the reproducible Kubernetes test setup (the deploy/k8s recipe is cloud-portable — it runs unchanged on EKS / GKE / AKS), Operating modes, and the Roadmap. The product proves itself in Lite first.

See also: DAG authoring · Operating modes.