Lead the migration from Glue/Redshift to Databricks

Responsibilities

• Lead the migration from Glue/Redshift to Databricks

Qualifications

• 10+ years of professional data engineering experience, with at least 2+ years in a tech-lead or lead engineer capacity.
• 5-6 years of experience in each of the following areas:
  • Deep, hands-on experience with Apache Spark (PySpark and Spark SQL), including performance tuning (partitioning, shuffles, skew, caching, file sizing).
  • Hands-on production experience with Databricks — Delta Lake, Unity Catalog, Workflows, and either Delta Live Tables or a comparable declarative pipeline framework.
  • Strong production experience with AWS Glue (jobs, crawlers, Data Catalog, triggers) and Amazon Redshift(including Redshift Spectrum, workload management, distribution and sort keys, performance tuning, and Redshift SQL / stored procedures).
  • Proven experience leading at least one substantial data platform migration or re-platforming effort end-to-end.
  • Expert SQL skills and strong software-engineering fundamentals: Python, version control, testing, and CI/CD.
  • Deep knowledge of AWS data services (S3, IAM, KMS, VPC, Lambda, Step Functions, MSK/Kinesis, CloudWatch).
  • Strong understanding of data modeling for analytics (dimensional, Data Vault, or medallion patterns) and lakehouse architecture.
  • Excellent communication skills; able to translate between executive stakeholders and engineers, and to write clear design docs and migration plans.