Senior Azure Data Engineer – Databricks

Role Overview

Data Architecture & Leadership

• Design, implement, and evolve Medallion Architecture (Bronze, Silver, Gold) using Delta Lake on Azure Databricks.
• Lead data architecture decisions, establish engineering standards, and define best practices.
• Review designs and code from team members, ensuring scalability, performance, and data quality.

Data Engineering & Development

• Build and maintain high‑volume, fault‑tolerant ETL/ELT pipelines using Azure Databricks and Azure Data Factory.
• Develop optimized PySpark transformations for large‑scale batch and streaming data.
• Implement reusable frameworks for ingestion, validation, transformation, and enrichment.

Data Ingestion & Streaming

• Architect and develop ingestion pipelines for structured, semi‑structured, and unstructured data.
• Implement batch and real‑time streaming pipelines using Azure Event Hubs / Kafka / Structured Streaming.
• Support incremental loads and CDC‑based data ingestion strategies.

Performance, Optimization & Reliability

• Optimize Spark workloads using partitioning, caching, Z‑Ordering, and file sizing strategies.
• Tune Delta Lake tables for performance and cost efficiency.
• Monitor pipeline health, troubleshoot failures, and improve data platform reliability.

Cloud, Security & Governance

• Leverage Azure services including ADLS Gen2, Synapse Analytics, Key Vault, Azure Monitor, and networking/security patterns.
• Implement role‑based access control, secrets management, and data protection best practices.
• Align data solutions with governance, compliance, and auditing requirements.

Collaboration & Mentorship

• Work closely with data scientists, analysts, product owners, and business stakeholders.
• Enable advanced analytics and machine learning workloads.
• Mentor junior and mid‑level data engineers and contribute to team skill growth.

Required Skills & Qualifications

• 7+ years of data engineering experience with strong Azure cloud expertise.
• Hands‑on expertise with Azure Databricks, Delta Lake, and Medallion Architecture.
• Advanced proficiency in PySpark and Python.
• Strong experience with Azure Data Factory for orchestration.
• Deep understanding of data modeling, data lakes, and big data processing.
• Proven ability to design scalable, production‑grade data platforms.
• Strong communication skills and experience working in agile environments.

Preferred Qualifications

• Experience implementing CI/CD pipelines for data platforms (Azure DevOps, GitHub Actions).
• Exposure to streaming technologies (Kafka, Event Hubs, Structured Streaming).
• Familiarity with data governance, cataloging, and lineage tools (e.g., Azure Purview).
• Experience supporting ML and advanced analytics workloads.
• Prior experience in financial services, healthcare, retail, or large‑scale enterprise environments.

Disclaimer: AI tools may assist in the recruitment process; however, all hiring decisions are made by the recruitment team based on a comprehensive evaluation of candidates.