Senior Data Engineer (Spark and Python Specialist)

Description

An award-winning privacy-preserving data collaboration platform that enables companies to analyze and collaborate on consumer data to gain insights, build predictive models, and monetize data without sharing the raw data or compromising consumer privacy, is seeking a Senior Cloud Data Engineer who will be a lead technical contributor responsible for building and optimizing high-performance data processing engines.

Responsibilities: • Spark Optimization: Act as the internal SME for Spark internals; manage memory, shuffle tuning, and partitioning for cost-effective performance. • Cloud-Agnostic Development: Build pipelines using Python and Delta Lake, decoupling code from specific cloud providers and reducing reliance on GUI tools (e.g., ADF). • Refactoring and Modernization: Migrate complex SQL-based ETL into modular, testable, and maintainable Python libraries. • Lakehouse Engineering: Manage Medallion Architecture (Bronze/Silver/Gold) using Delta Lake, ensuring storage performance via Z-Ordering and Vacuuming. • Code-First Orchestration: Support the transition to code-centric patterns (Airflow, Dagster) to prioritize portability. • Technical Excellence: Lead code reviews, mentor junior engineers, and implement automated testing frameworks (Pytest).

Minimum Requirements: • Education: Bachelor’s degree in Computer Science, Information Systems, Engineering, or a related field. • Spark Mastery: 6+ years of Spark/PySpark experience; expert ability to diagnose bottlenecks via Spark UI and optimize complex DAGs. • Advanced Python: Proficiency in production-grade Python, including building reusable libraries and automated testing. • Azure Ecosystem: Strong experience with Azure Synapse, Dedicated SQL Pools, and Data Factory. • Modern Data Stack: Hands-on experience with Delta Lake, Parquet, and containerization (Docker). • Migration Skills: Solid T-SQL skills to interpret and migrate legacy logic into Python-centric environments. • Security and Governance: Proven ability to implement high levels of security and compliance across data processes.

Benefits: • Competitive salary based on experience (salary can potentially be more based on experience/skills)

IF you meet the above requirements and want to make a career-changing move, apply today by emailing your CV to itcareers@hireresolve.za.com

Package & Remuneration

R840 000 - R960 000 per annum About Us

Hire Resolve is a Recruitment Agency in South Africa that focuses on working the following jobs in South Africa.

Engineering Recruitment Agency in South Africa focusing on the following disciplines:

Civil Engineering jobs in South Africa. Johannesburg, Cape town, Durban, Eastern Cape (Gqeberha and East London) Mechanical engineering jobs in South Africa, Johannesburg, Cape town, Durban, Eastern Cape (Gqeberha and East London) Electrical engineering jobs in South Africa, Johannesburg, Cape town, Durban, Eastern Cape (Gqeberha and East London) Construction jobs in South Africa, Johannesburg, Cape town, Durban, Eastern Cape (Gqeberha and East London) Quantity Surveying jobs in South Africa, Johannesburg, Cape town, Durban, Eastern Cape (Gqeberha and East London)

Finance Recruitment Agency in South Africa focusing on the following disciplines.

Finance jobs in South Africa. Johannesburg, Cape town, Durban, Eastern Cape (Gqeberha and East London) Actuarial jobs in South Africa, Johannesburg, Cape town, Durban, Eastern Cape (Gqeberha and East London) Investment jobs in South Africa, Johannesburg, Cape town, Durban, Eastern Cape (Gqeberha and East London) Financial risk jobs in South Africa, Johannesburg, Cape town, Durban, Eastern Cape (Gqeberha and East London) Tax jobs in South Africa, Johannesburg, Cape town, Durban, Eastern Cape (Gqeberha and East London) Audit Jobs in South Africa, Johannesburg, Cape town, Durban, Eastern Cape (Gqeberha and East London)

Mining Recruitment Agency in South Africa. Mining jobs in Johannesburg, Rustenburg, Limpopo, Northern Cape, Mpumalanga and more.

Manufacturing jobs in South Africa. Johannesburg, Cape Town, Durban, Eastern Cape (Gqeberha and East London) Supply Chain jobs in South Africa. Johannesburg, Cape Town, Durban, Eastern Cape (Gqeberha and East London)

IT jobs in South Africa. Johannesburg, Cape Town, Durban, Eastern Cape (Gqeberha and East London)

View our jobs below.

Need staff. Email us. clients@hireresolve.us