Data Engineer (Senior): Real-time, cloud-based data platforms

Introduction

We are seeking a hands-on Data Engineer with strong experience in building scalable data pipelines and

analytics solutions on Databricks. You will design, implement, and maintain end-to-end data flows, optimize

performance, and collaborate with data scientists, analytics, and business stakeholders to turn raw data into trusted insights.. In this role, you’ll work on near real-time analytics, building scalable data pipelines and transforming complex data into meaningful insights that power production quality across global operations. If you thrive on solving complex problems, enjoy working with modern cloud technologies like Databricks and Azure, and want to be part of a collaborative team shaping the future of data engineering, this opportunity is for you.

Essential & Advantageuous Skills

ESSENTIAL SKILLS:

Expertise with Apache Spark (PySpark), Databricks notebooks, Delta Lake, and SQL.

Strong programming skills in Python for data processing.

Experience with cloud data platforms (Azure) and their Databricks offerings; familiarity with object storage (ADLS).

Proficient in building and maintaining ETL/ELT pipelines, data modeling, and performance optimization.

Knowledge of data governance, data quality, and data lineage concepts.

Experience with CI/CD for data pipelines, and orchestration tools (GitHub Actions, Asset Bundles or Databricks’ jobs).

Strong problem-solving skills, attention to detail, and ability to work in a collaborative, cross-functional team.

ADVANTAGEOUS SKILLS:

Experience with streaming data (Structured Streaming, Kafka, Delta Live Tables).

Familiarity with materialized views, streaming tables, data catalogs and metadata management.

Knowledge of data visualization and BI tools (Splunk, Power BI, Grafana).

Experience with data security frameworks and compliance standards relevant to the industry.

Certifications in Databricks or cloud provider platforms.

Role & Responsibilities

Design, develop, test, and maintain robust data pipelines and ETL/ELT processes on Databricks (Delta Lake, Spark, SQL, Python/Scala/SQL notebooks).

Architect scalable data models and data vault/ dimensional schemas to support reporting, BI, and advanced analytics.

Implement data quality, lineage, and governance practices; monitor data quality metrics and resolve data issues proactively.

Collaborate with Data Platform Engineers to optimize cluster configuration, performance tuning, and cost

management in cloud environments (Azure Databricks).

Build and maintain data ingestion from multiple sources (RDBMS, SaaS apps, files, streaming queues) using modern data engineering patterns (CDC, event-driven pipelines, change streams, Lakeflow Declarative Pipelines).

Ensure data security and compliance (encryption, access controls) in all data pipelines.

Develop and maintain CI/CD pipelines for data workflows; implement versioning, testing, and automated

deployments.

Partner with data scientists and analysts to provision clean data, notebooks, and reusable data products;

support feature stores and model deployment pipelines where applicable.

Optimize Spark jobs for speed and cost; implement job scheduling, monitoring, and alerting.

Document data lineage, architecture, and operational runbooks; participate in architectural reviews and bestpractice

governance.

Qualification/Experience

Bachelor’s or Master’s degree in Computer Science, Data Engineering, Information Systems, or a related field.

3+ years of hands-on data engineering experience.

What do we offer you?

Cutting edge global IT system landscape and processes

Flexible working of 1960 hours in a 12-month period

High Work-Life balance

Remote / On-site work location flexibility

Affordable BMW Group vehicle promotions (buying or leasing options available, terms and conditions apply)

Highly motivating, energetic, and fast-paced working environment

Modern, state-of-the-art offices

Dynamic Global Team collaboration

Application of the Agile Working Model Methodology