Azure Data Engineering Lead with Databricks

About This Role

This role offers the opportunity to lead and build enterprise‑scale Azure data platforms using a modern lake house architecture. Candidates gain deep exposure to Azure, Databricks, advanced data modeling, and governance while working on complex, high‑impact healthcare data. It strengthens long‑term career growth as a senior data engineering or architecture leader in a high‑demand domain.

Core Technical Expertise

• 10–15+ years of experience in data engineering and hands on technical delivery roles, preferably in Healthcare Payer / Medicaid environments
• Extensive, Azure first cloud experience, including solution architecture, implementation, and operations
• Strong hands-on expertise with Azure data platform services, including Azure Data Lake Storage (ADLS Gen2), Azure Databricks, Azure Data Factory (ADF), Azure Synapse Analytics, Azure SQL / SQL Server, Azure Key Vault, Azure DevOps (Repos, Pipelines, Boards.
• Strong proficiency in: SQL (advanced querying, tuning, optimization) , Python and PySpark, ETL / ELT development
• Strong data modeling experience, including Conceptual, logical, and physical modeling , Dimensional modeling (star/snowflake schemas), Experience handling SCDs and historized data
• Deep understanding of: Cloud native Data Lake / Lakehouse architectures , Batch and streaming processing patterns , Incremental processing, and CDC
• Proven experience reviewing and optimizing : Data models , Transformation logic, Schema evolution strategies
• Experience implementing cloud and data governance controls across Dev/Test/Prod, Agentic IDE & AI Assisted SDLC

Experience

• Proven experience architecting and delivering systems using agentic IDEs
• Ability to: Define architectural and data modeling intent that agents can follow
• Break features into agent executable tasks
• Govern AI autonomy (guardrails, permissions, reviews)
• Integrate agentic workflows into CI/CD pipelines
• Strong understanding of Security implications of autonomous code execution
• Compliance, auditability, and traceability
• AI assisted SDLC operating models

Healthcare Domain Expertise

• Strong experience in Healthcare Payer / Medicaid data environments
• Familiarity with:
  • Claims
  • Member
  • Provider
  • Eligibility data
• Understanding of HIPAA, PHI, and healthcare compliance requirements

Education

• Engineering Degree – BE / ME / BTech / M Tech / B.Sc. / M.Sc.
• Technical certification in multiple technologies is desirable.