Databricks Manager

About This Role

This role involves overseeing the design, deployment, and optimization of enterprise-scale data engineering solutions utilizing Databricks across major cloud platforms. You will lead innovation in big data architecture, advise senior stakeholders, and ensure data solutions align with business objectives.

Responsibilities

• Lead the development, implementation, and scaling of advanced data engineering solutions using Databricks across AWS, Azure, or GCP environments
• Establish, document, and promote best-in-class approaches for data architecture, integration, and modelling
• Oversee the design, development, and maintenance of robust data pipelines and data architectures
• Initiate and manage efforts to improve data quality, operational efficiency, and process scalability
• Evaluate, pilot, and integrate new big data and analytics technologies
• Consult on, design, and implement governance, security, and compliance strategies tailored to modern cloud data ecosystems
• Lead, coach, and develop teams of data engineers and architects, fostering technical growth and effective project delivery
• Communicate technical concepts and business value to diverse stakeholders, including executives, business leads, and technology teams
• Oversee the implementation of CI/CD practices with tools such as Azure DevOps, AWS Code Pipeline, Jenkins, TFS, or PowerShell

Requirements

• Bachelor's or Master's degree in Computer Science, Engineering, or a related field
• 6+ years of hands-on experience in data engineering with a strong focus on Databricks, deployed on any major cloud (AWS, Azure, GCP)
• Expertise with cloud-native databases, storage solutions, and distributed compute platforms
• Deep understanding of Lakehouse architecture, Apache Spark, Delta Lake, and related big data technologies
• Advanced skills in data warehousing, 3NF, dimensional modeling, and enterprise-level data lakes
• Experience with Databricks components including Delta Live Tables, Autoloader, Structured Streaming, Databricks Workflows, and orchestration tools (e.g., Apache Airflow)
• Expertise in designing and supporting incremental data loads and building metadata-driven ingestion/data quality frameworks using PySpark
• Hands-on experience with Databricks Unity Catalog and implementing fine-grained security and access control
• Proven track record in deploying code and solutions via automated CI/CD pipelines
• 4+ years leadership experience in managing complex, cross-functional data projects and technical teams
• Experience with performance optimization of Data engineering pipelines, code, compute resources

Qualifications

• Bachelor's or Master's degree in Computer Science, Engineering, or a related field
• 6+ years of hands-on experience in data engineering with a strong focus on Databricks, deployed on any major cloud (AWS, Azure, GCP), and 4+ years in leadership managing complex, cross-functional data projects and technical teams

Nice to Have

• Master's degree preferred
• Comprehensive knowledge of the AWS, Azure, and GCP cloud ecosystems and associated big data stacks
• Demonstrated skill in performance tuning and optimization within Databricks/Apache Spark environments
• Stays current with the latest Databricks feature releases and platform enhancements
• Exceptional communication and stakeholder management abilities, including comfort interfacing with executive leadership
• Experience with Databricks Lakeflow
• Experience in AI/ML

Skills

* Required skills