Senior Data Engineer

About This Role

Build and maintain robust data pipelines using Azure Data Factory and Databricks to support internal business analytics. Ensure seamless ingestion and preparation of data from various sources, elevating data quality and influencing critical decision-making processes.

Responsibilities

• Build and maintain Azure Data Factory pipelines to ingest data from multiple sources
• Write Python code in Databricks to clean raw data and move it into the silver layer, handling deduplication, type casting, and validation
• Monitor daily jobs and troubleshoot any failures to ensure pipeline stability
• Implement automated checks to verify data integrity between source systems and the data lake

Requirements

• 5+ years of experience in Data Engineering, with a focus on ETL/ELT pipelines in cloud environments
• Proficiency in Azure Data Factory and Databricks for pipeline development
• Strong proficiency in SQL for data analysis and Python for scripting
• Experience implementing automated data quality checks and troubleshooting pipeline failures
• Familiarity with data schemas and APIs from platforms like Microsoft Dynamics 365, Salesforce, and ServiceNow
• Demonstrated experience using LLMs to enhance data engineering workflows
• Bachelor’s degree in Computer Science, Software Engineering, Data Engineering, or a related field

Qualifications

• Bachelor’s degree in Computer Science, Software Engineering, Data Engineering, or a related field
• 5+ years of experience in Data Engineering, with a focus on ETL/ELT pipelines in cloud environments

Skills

* Required skills

Benefits