Sr. Azure Data engineer

Role Summary

We are looking for a hands-on Data Engineer with strong experience in Azure Databricks to support the migration from Hive Metastore to Unity Catalog. The role focuses on remediating Databricks notebooks, validating Unity Catalog–compliant access to Azure Data Lake (ADLS Gen2), and performing end-to-end testing after migration.

The engineer will also work closely with Azure Data Factory (ADF) pipelines used for orchestration, ensuring seamless execution post‑migration and minimal disruption during cutover.

Key Responsibilities

Unity Catalog Migration (Azure Databricks)

• Support migration from Hive Metastore to Unity Catalog in Azure Databricks.
• Validate and work with Unity Catalog objects: Catalogs, schemas, managed & external tables External locations and volumes backed by ADLS Gen2.
• Ensure compliance with Unity Catalog governance and access controls (RBAC, ACLs).

Notebook Remediation & Code Changes (Primary)

• Update Databricks notebooks to be Unity Catalog–compliant:
  • Replace legacy database references with 3‑level namespace.
  • Refactor Spark SQL and PySpark code impacted by UC migration.
• Update storage access patterns:
  • Replace legacy mount-based paths with Unity Catalog external locations or volumes.
  • Ensure secure access to Azure Data Lake (ADLS Gen2) using managed identities / service principals.
• Parameterize hardcoded paths, schemas, and table references where required.
• Resolve permission-related issues caused by UC enforcement.

ADF Orchestration Validation

• Validate and support Azure Data Factory (ADF) pipelines that orchestrate Databricks notebooks and jobs.
• Ensure:
  • Correct notebook paths and parameters post-migration.
  • Service principal permissions align with Unity Catalog policies.
  • No regression in scheduled and triggered executions.
• Troubleshoot end-to-end pipeline failures spanning ADF → Databricks → ADLS.

Testing & Post-Migration Validation (Primary)

• Execute comprehensive testing after migration:
  • Data validation & reconciliation (row counts, aggregates, sample checks).
  • Functional testing of notebooks and Databricks jobs.
  • ADF pipeline execution validation.
  • Security & access testing (users, groups, service principals).
• Identify and resolve performance regressions or permission-related failures.
• Support production cutover and hypercare activities.