Databricks Manager

About This Role

This role involves leading the end-to-end design, deployment, and optimization of enterprise-scale data engineering solutions using Databricks on major cloud platforms like AWS, Azure, or GCP. The manager will lead innovation in big data architecture, shape best practices, advise stakeholders, and ensure data solutions align with business objectives.

Responsibilities

• Lead the development, implementation, and scaling of advanced data engineering solutions using Databricks across AWS, Azure, or GCP environments.
• Establish, document, and promote best-in-class approaches for data architecture, integration, and modelling.
• Oversee the design, development, and maintenance of robust data pipelines and data architectures that support large-scale, enterprise data needs.
• Initiate and manage efforts to improve data quality, operational efficiency, and process scalability.
• Evaluate, pilot, and integrate new big data and analytics technologies, ensuring the organization remains at the cutting edge.
• Consult on, design, and implement governance, security, and compliance strategies tailored to modern cloud data ecosystems.
• Lead, coach, and develop teams of data engineers and architects, fostering technical growth and effective project delivery.
• Communicate technical concepts and business value to diverse stakeholders, including executives, business leads, and technology teams.
• Oversee the implementation of CI/CD practices with tools such as Azure DevOps, AWS Code Pipeline, Jenkins, TFS, or PowerShell for streamlined deployments and operations.

Requirements

• 6+ years of hands-on experience in data engineering with a strong focus on Databricks, deployed on any major cloud (AWS, Azure, GCP).
• 6+ years Technical Proficiency with cloud-native databases, storage solutions, and distributed compute platforms.
• Deep understanding of Lakehouse architecture, Apache Spark, Delta Lake, and related big data technologies.
• Advanced skills in data warehousing, 3NF, dimensional modeling, and enterprise-level data lakes.
• Experience with Databricks components including Delta Live Tables, Autoloader, Structured Streaming, Databricks Workflows, and orchestration tools (e.g., Apache Airflow).
• Expertise in designing and supporting incremental data loads and building metadata-driven ingestion/data quality frameworks using PySpark.
• Hands-on experience with Databricks Unity Catalog and implementing fine-grained security and access control.
• Proven track record in deploying code and solutions via automated CI/CD pipelines.
• Minimum of 4 years experience in leadership managing complex, cross-functional data projects and technical teams.
• Experience with performance optimization of Data engineering pipelines, code, compute resources.

Qualifications

• Bachelor's or Master's degree in Computer Science, Engineering, or a related field
• 6+ years of hands-on experience in data engineering with a strong focus on Databricks, deployed on any major cloud (AWS, Azure, GCP), and a minimum of 4 years leadership experience.

Nice to Have

• Comprehensive knowledge of the AWS, Azure, and GCP cloud ecosystems and associated big data stacks.
• Demonstrated skill in performance tuning and optimization within Databricks/Apache Spark environments.
• Stays current with the latest Databricks feature releases and platform enhancements.
• Exceptional communication and stakeholder management abilities, including comfort interfacing with executive leadership.
• Experience with Databricks Lakeflow.
• Experience in AI/ML.

Skills

* Required skills

Benefits