Senior Data Engineer (m/f/d)

About

For our Insurance business unit, we are seeking an experienced Senior Data Engineer (m/f/d) to play a key role in the further development of an analytics platform. The ideal candidate will have a deep understanding of Azure Databricks and experience in building resilient data pipelines and administering Databricks workspaces.

Responsibilities

• Supporting the further development of the platform into a Lakehouse environment.
• Contributing to the migration of the platform to a new Azure tenant.
• Technical integration of other IT systems.
• Implementing development processes for Lakehouse developers.
• Automatically provisioning permission groups and monitoring dashboards.
• Migrating existing assets, data, and data pipelines.
• Integrating GitLab CI/CD with GitHub Actions.
• Decommissioning cloud resources in the old Azure tenant.

Requirements

• Deep understanding of permission structures within Databricks using Unity Catalog.
• Experience in designing and administering Databricks workspaces.
• Development of resilient data pipelines with PySpark, including test automation, packaging, and deployment with Declarative Automation Bundles (DAB).
• Knowledge of data contracts.
• Data partitioning in Delta Lake.
• Experience with GitLab CI/CD and/or GitHub Actions.

Desired Skills

• Experience implementing data ingest and transformations.
• Knowledge of cloud architectures and technologies.
• Familiarity with AI systems and their hosting.

Details

• Start: ASAP
• Duration: 15 months
• Location: Cologne / remote