Senior / Lead Data Engineer

About

We are supporting a leading organisation within the Insurance / Financial Services sector in hiring a Senior / Lead Data Engineer with strong AI and PySpark expertise .

This is a high-impact role focused on building and scaling data platforms that support advanced analytics, machine learning, and AI-driven initiatives across large enterprise environments.

Key Responsibilities

• Design, build, and optimise scalable data pipelines using PySpark and modern data engineering frameworks
• Work closely with Data Scientists and AI/ML Engineers to productionise machine learning models
• Lead the development of data architectures supporting AI and analytics use cases (e.g. fraud detection, risk modelling, claims automation)
• Handle large-scale structured and unstructured datasets in distributed environments
• Ensure data quality, governance, and performance across enterprise data platforms
• Collaborate with cross-functional teams including engineering, analytics, and business stakeholders
• Provide technical leadership and guidance to junior engineers (for Lead-level candidates)

Required Skills & Experience

• Proven experience as a Senior or Lead Data Engineer in large enterprise environments
• Strong hands-on expertise with PySpark and distributed data processing
• Experience working on AI / Machine Learning data pipelines
• Advanced Python programming skills
• Experience with big data technologies (e.g. Spark, Hadoop ecosystem)
• Strong experience with cloud platforms (AWS, Azure, or GCP )
• Solid understanding of data modelling, ETL/ELT processes, and data architecture
• Experience working within large Financial Services, Insurance, or FinTech environments
• Strong stakeholder communication skills and ability to work in distributed teams

Nice to Have

• Experience with MLOps tools (e.g. MLflow, Kubeflow)
• Exposure to real-time data processing (Kafka, streaming frameworks)
• Experience with data governance and regulatory environments (e.g. GDPR)
• Prior experience in a Lead or Architect capacity