Senior Data Engineer

About Schwarz Digits

Schwarz Digits schafft das technologische Fundament für digitale Entscheidungsfreiheit in Europa. Als IT- und Digitalsparte der Schwarz Gruppe entwickeln und verantworten wir einerseits die IT-Infrastrukturen für die Handelssparten Lidl und Kaufland sowie die Schwarz Produktion und PreZero. Gleichzeitig agieren wir als unabhängiger Anbieter am externen Markt, um Unternehmen in ganz Europa bei ihrer digitalen Transformation zu unterstützen. Unsere Kernleistungen bündeln wir in den Bereichen Cloud, Cyber Security, Data & AI, Communication und Workspace.

Trage auch du zur digitalen Entscheidungsfreiheit in Europa bei. Bei uns arbeitest du an der Schnittstelle zwischen Agilität und Sicherheit: Du profitierst von den schnellen Entscheidungswegen, genießt echte Gestaltungsspielräume in deinen Projekten und baust dabei auf das stabile Fundament der Schwarz Gruppe.

Your Responsibilities

• Data Product Engineering: You will conceptualize, develop, and operate high-performance batch and streaming data flows in our central Google Cloud Data Lakehouse for analytics and machine learning use cases.
• Architecture & Scaling: You will optimize the cloud architecture (GCP), ensure the scalability of our solutions, and advance our standards for real-time and batch data processing.
• Software Excellence: You will apply modern software engineering methods and set new benchmarks for CI/CD, automated testing, and deployment processes within the data organization.
• Framework Co-creation: You will support the further development of our central orchestration tools (based on Airflow and dbt) to increase the efficiency and quality of data delivery across the company.
• Data Quality & Governance: You will implement automated validation processes (e.g., for streaming events) and ensure compliance with security and data protection standards.
• Agile Collaboration: You will work closely with Product Owners, Data Scientists, and business departments in interdisciplinary teams to translate complex business requirements into technical solutions.

Your Qualifications

• Experience: Several years of solid experience in Data Engineering, Big Data environments, or Software Engineering, ideally in a senior role.
• Cloud & Streaming Expertise: You possess in-depth knowledge of cloud infrastructures (Google Cloud Platform) and extensive experience in processing real-time data with Apache Kafka.
• Tech Stack: Very good knowledge of Python and Spark. You are proficient in using containerization (Kubernetes/Docker).
• Orchestration & Transformation: Experience with tools like Apache Airflow and transformations via dbt is a significant advantage.
• Software Best Practices: You are confident in handling the software development lifecycle, version control (Git), as well as DevOps principles and automated tests.
• Conceptual Strength: You are enthusiastic about modern architectural concepts like Data Lakehouses and Data Mesh and enjoy actively shaping technological standards.
• Personality: Analytical thinking, an agile mindset, and a self-directed, solution-oriented approach to work.
• Languages: Very good English skills; German language skills are preferred.