Data Engineer

Introduction:

The Data Engineer will be responsible for working with a team to develop, maintain, and optimize data pipelines using Databricks, Python, SQL, and Airflow. They will play a key role in ensuring the smooth flow of data within the organization and will contribute to the overall success of our data-driven initiatives.

Responsibilities:

• Design, build, and maintain scalable data pipelines using Databricks, Python, SQL, and Airflow
• Collaborate with data scientists and analysts to understand data requirements and deliver solutions
• Optimize data pipelines for performance and reliability
• Troubleshoot and resolve data pipeline issues in a timely manner
• Ensure data quality and integrity throughout the data pipeline
• Document data pipelines and processes for knowledge sharing and future reference

Requirements:

Required Skills:

• Strong experience with Databricks and Python
• Proficiency in SQL for data manipulation and querying
• Hands-on experience with Airflow for workflow management
• Proven track record as a Data Engineer

Preferred Skills:

• Experience working with big data technologies such as Hadoop or Spark
• Familiarity with data warehousing concepts and tools
• Knowledge of machine learning and data science principles
• Excellent problem-solving and analytical skills