Lead Data Engineer

Key Skills Required

• AWS (S3, Redshift, Glue, Lambda, EMR, Athena)
• Data Engineering & Data Modeling (Star Schema, Snowflake, Dimensional Modeling)
• Python, PySpark, SQL
• Big Data Technologies (Hadoop, Spark)
• Infrastructure as Code (Terraform)
• AI/ML integration basics
• Visualization tools (Power BI)

Roles & Responsibilities

• Design, develop, and maintain scalable data pipelines for batch and real-time processing using AWS services
• Build and optimize data lakes and data warehouses using Amazon S3, Redshift, and Glue
• Develop robust ETL/ELT pipelines using Python, PySpark, and SQL
• Implement efficient data modeling techniques such as star schema and dimensional modeling
• Work with large-scale distributed systems using Hadoop and Apache Spark
• Integrate AI/ML models into data pipelines to support advanced analytics
• Automate infrastructure provisioning using Terraform (IaC)
• Ensure data quality, governance, and security across pipelines
• Collaborate with cross-functional teams including data scientists, analysts, and business stakeholders
• Develop dashboards and reports using Power BI for business insights
• Monitor and optimize performance of data pipelines and cloud resources.
• Exposure to AI/ML frameworks (SageMaker, TensorFlow, etc.)