Senior Data Engineer

About This Role

Design and develop scalable data pipelines using AWS and Azure services, implement real-time data processing, and collaborate with data scientists to deliver high-quality data solutions.

Responsibilities

• Design and develop scalable data pipelines using AWS services such as S3, Redshift, and Lambda
• Implement real-time data processing solutions using Apache Kafka to handle streaming data efficiently
• Utilize Python and PySpark for data transformation and analysis to ensure high-quality deliverables
• Collaborate with data scientists and analysts to understand data requirements and provide appropriate data solutions
• Conduct data modeling and data warehouse optimization to enhance query performance and support analytics needs
• Leverage dbt for data transformation workflows and ensure version control of analytics code
• Participate in architecture discussions to design modern data infrastructure on Azure that supports future growth and scalability

Requirements

• 8 years of experience in data engineering or a related field
• Strong proficiency in AWS services and architecture for data processing and storage
• Experience with Apache Kafka and its ecosystem for handling real-time data streams
• Proficient in Python and PySpark with hands-on experience in building data processing applications
• Familiarity with dbt for data transformation and analytics workflow management
• Knowledge of Azure cloud services and how they can be integrated into data solutions

Qualifications

• Bachelor's degree in Computer Science, Engineering, or a related field
• 8 years of experience in data engineering or a related field

Skills

* Required skills