Sr Data Engineer

About

We are seeking a highly skilled and experienced Senior Data Engineer to join our team. This role will focus heavily on data architecture to enable analytics use cases, which are ultimately served via PowerBI. The ideal candidate will have extensive experience with Databricks, and proficiency in Python. Additionally, the candidate should be adept at following and learning data ingestion guidelines from multiple domains and teams, possess a deep understanding of Databricks tooling to build workflows that support data schemas, and have expertise in using Unity Catalog and its tools. Our cloud environment is based on Azure, and familiarity with it is essential. Some Java knowledge will be required.

Must Have Qualifications

• Bachelor’s or master’s degree in computer science, Engineering, or a related field.
• Minimum of 4 years of experience in data engineering.
• Proficiency in Databricks and experience with its tools (e.g., Delta Lake, Databricks SQL, MLflow, Unity Catalog) and other big data technologies (e.g., Spark, Hadoop).
• Strong programming skills in Python, with experience in building and optimizing data pipelines.
• Some Java knowledge/understanding.
• Solid understanding of database systems, data modeling, and ETL processes.
• Experience testing in a Databricks environment.
• Excellent problem-solving skills and attention to detail.
• Strong communication and collaboration skills, with the ability to work effectively in a team environment.
• Understanding data security best practices and compliance standards.
• Experience with project management or agile methodologies is a plus.

Nice Have Qualifications

• Familiarity with PowerBI is nice to have, as it provides context for the analytics use cases.
• Relevant certifications, such as Microsoft Certified: Azure Data Engineer Associate or Databricks Certification, is a plus.

Key Responsibilities

• Design and implement robust data architecture solutions to support analytics use cases, ensuring data integrity and security.
• Develop, optimize, and maintain scalable data pipelines using Databricks and other big data technologies.
• Utilize Databricks tools such as Delta Lake, and Unity Catalog to build efficient workflows and support complex data schemas.
• Manage data governance and security using Unity Catalog to ensure compliance and data quality across the organization.
• Leverage Azure services, including Azure Data Lake, for data warehousing and storage solutions.
• Implement CI/CD practices for data pipelines to ensure efficient and reliable data operations.
• Collaborate with cross-functional teams to gather and analyze data requirements, translating them into technical specifications.
• Follow and learn data ingestion guidelines from multiple domains and teams to ensure consistency and compliance.
• Utilize Python for building, testing, and deploying data processing scripts and applications (mostly within DBX).
• Monitor and troubleshoot data systems to ensure high availability and performance.
• Implement best practices for data management, including data quality, data governance, and data lifecycle management.
• Ensure data security and compliance with relevant standards and regulations (e.g., GDPR, CCPA).
• Mentor and provide guidance to junior data engineers, fostering a culture of continuous learning and improvement.
• Stay updated with the latest industry trends and technologies to drive innovation within the team.

#J-18808-Ljbffr