Data Engineering

Job Description

Seeking a Specialist with 7 to 9 years of experience in Apache Spark and Python to design, develop, and optimize scalable data processing solutions within the Python Data domain.

• Develop and maintain scalable and robust data pipelines using Apache Spark and Python.
• Collaborate with data scientists and analysts to enhance and optimize data workflows.
• Ensure data quality, integrity, and consistency across large and complex datasets.
• Implement efficient data transformations and aggregations to support analytics and reporting.
• Integrate data solutions seamlessly into existing systems by working closely with cross-functional teams.
• Monitor and troubleshoot data processing jobs to maintain high reliability and performance.
• Stay updated with emerging trends, technologies, and best practices in big data and Python development.

Roles and Responsibilities

• Design, develop, and optimize high-performance data processing applications leveraging Apache Spark.
• Write clean, reusable, and efficient Python code to support various data engineering initiatives.
• Analyze complex datasets to extract actionable insights and identify trends.
• Collaborate with stakeholders to gather requirements and deliver tailored data solutions that meet business objectives.
• Mentor junior team members and promote knowledge sharing within the team.
• Participate actively in code reviews to ensure adherence to coding standards and best practices.
• Manage deployment of data pipelines and continuously monitor their health and performance in production environments.