Airflow Optimization Specialist (Azure Data Platform)

Role : Airflow Optimization Specialist - Azure Data Platform

Location : London, UK (Hybrid / Onsite preferred)

Type : Contract

Start : Immediate

Rate : Market competitive (Inside IR35 / Outside - depending on experience)

We are seeking an Apache Airflow SME to support optimization of our Azure Data Platform . You will play a key role in fine-tuning orchestration, optimizing DAGs, and building scalable workflow solutions in a production-grade Airflow environment .

Key Responsibilities :

Optimize Apache Airflow performance and orchestration pipelines

Build modular and scalable DAGs

Integrate Airflow with Azure Databricks , Azure Data Factory , and Synapse

Implement CI / CD pipelines using Azure DevOps

Enable observability with Prometheus , Grafana , Azure Monitor

Architect reusable frameworks and automation patterns

Mentor junior engineers and provide hands-on leadership

Must-Have Skills :

5+ years experience with Apache Airflow (production setup, optimization, plugins)

Deep understanding of Airflow scheduler / executors (Celery, Kubernetes)

Azure stack : ADF , Databricks , Azure Functions , Synapse , Storage

Azure DevOps - YAML pipelines, CI / CD, release management

Python (modular / testable coding), Git , Jenkins

Containerization : Docker , Kubernetes , AKS

Monitoring / logging : Grafana , Prometheus , Azure Monitor , Log Analytics

Agile delivery, Jira Service Desk experience