Stage - Data engineer Modernisation plateforme data (Orchestration & Streaming) H/F

About

Integrated into the Voltalis Data Platform (VDP) team, a team of 5 data engineers and analytics engineers, you will work on two structuring projects for the evolution of our data infrastructure.

At Voltalis, we receive tens of millions of lines of data daily from our fleet of connected boxes. To absorb this growth and improve the freshness of our data, the VDP is undertaking two major transformations — and you will be a key contributor (you will be supported by senior data engineers on both projects).

Project 1 — Orchestrator Migration

Our data platform currently relies on Google Workflows to orchestrate its pipelines. As part of this internship, you will participate in the migration to a new orchestrator, to gain observability, developer experience, and automation.

Specifically:

• Implement the tool (we are already considering Prefect, but it is not yet validated)
• Audit existing pipelines and define the migration strategy
• Develop and deploy new workflows
• Ensure the progressive transition to production

Project 2 — Introduction of Streaming

Currently, our platform operates entirely in batch. To reduce latency on certain critical data, you will contribute to the introduction of real-time streams.

Specifically:

• Design the streaming architecture in conjunction with BigQuery
• Implement the first streams and integrate them into the existing stack
• Define the migration strategy and implement it
• Set up associated monitoring

Stack

Python · SQL · dbt core · BigQuery · GCS · Cloud Run · Google Workflows · Prefect · GitLab CI/CD