Spark connection

Submit Spark jobs from a task via a managed Leoflow Connection and the Apache Spark provider hooks. The provider exposes a few conn types — all from apache-airflow-providers-apache-spark:

Declare the provider

# leoflow.yaml
dag_id: spark_job
connectors:
  - spark

Fields the UI asks for

The host:port + Extra round-trip is pinned by TestSparkConnectionURIShapeIntegration.

Example DAG (copy-paste)

Docs-only recipe (needs a Spark cluster). The hook is imported inside the task.

# dag.py
from __future__ import annotations

from airflow.sdk import DAG, task


@task
def submit() -> None:
    from airflow.providers.apache.spark.hooks.spark_submit import SparkSubmitHook

    hook = SparkSubmitHook(conn_id="spark_default", application="/opt/jobs/etl.py")
    print("submit: spark-submit via SparkSubmitHook(spark_default)")
    hook.submit()
    print("submit: ok")


with DAG("spark_job", schedule=None, catchup=False, tags=["example"]):
    submit()

# leoflow.yaml
schema_version: "1.0"
dag_id: spark_job
description: Submit a Spark job via SparkSubmitHook.
owner: examples
tags:
  - example
python_version: "3.11"
connectors:
  - spark

Run it

1. Admin → Connections → +, type spark. Set Host + Port (the master), and any Extra tuning.
2. leoflow lite path/to/this/dag → trigger spark_job.

Security notes

• Keytab / principal stay in Extra: for Kerberos-secured clusters keep principal and keytab in Extra (encrypted at rest, ADR 0019) — never inline them in the DAG source.
• Authenticated submit endpoint: submit to a master/Connect endpoint that enforces auth and TLS where the cluster supports it; the Connection only carries where to submit, not the transport policy.
• Secrets in logs: never print() the URI — it may carry credentials. Log the host + port only.
• gRPC channel (agent ↔ control plane): secrets are served only over an authenticated channel (ADR 0021); Pro must run with TLS.

Related

• docs/connections/index.md — Installing a connector’s provider.
• ADR 0019 / 0021 — secret encryption + agent delivery.