Senior Data Engineer - Airflow Developer

About This Role

The Senior Data Engineer will design, develop, and deploy Apache Airflow DAGs to orchestrate Databricks jobs and build and maintain ETL workflows. This role involves optimizing Airflow DAGs for performance and collaborating with other engineering teams.

Responsibilities

• Design, develop, and deploy Apache Airflow DAGs to orchestrate Databricks jobs
• Build and maintain ETL workflows to extract, transform, and load data
• Optimize Airflow DAGs for performance, scalability, and reliability
• Monitor Airflow and Databricks job runs, and troubleshoot execution failures
• Refactor and maintain reusable workflow components in python
• Collaborate with platform, DevOps, and other data engineering teams

Requirements

• Hands-on experience with Apache Airflow 2.0, Databricks, PySpark, and SparkSQL
• Strong Python programming skills
• Experience designing and developing ETL processes and data pipelines in Databricks
• Solid understanding of scheduling, dependencies, and error handling in Airflow

Skills

* Required skills

Benefits