Databricks Platform Engineer - PySpark

Role Purpose

• Own the optimisation and operational excellence of the Databricks platform, ensuring data pipelines run efficiently, cost-effectively, and at scale across the organisation.

What You'll Be Doing

Databricks Platform Optimisation

• Monitor and optimise:
  • Pipeline performance
  • Cluster usage/DPUs
  • Subscription-level costs
• Support both streaming and batch workloads

Infrastructure & Operations

• Improve resource allocation (scaling up/down)
• Drive execution efficiency
• Ensure platform stability and optimal runtime performance

CI/CD & Release Management

• Implement and maintain CI/CD pipelines
• Manage asset bundles and deployment processes
• Standardise release and governance workflows

Vendor Collaboration

• Work directly with Databricks
• Provide performance evidence
• Raise and track issues
• Drive platform-level improvements

Developer Enablement

• Act as the internal Databricks expert
• Guide engineers on:
  • Best practices
  • Efficient pipeline design
  • Performance and cost optimisation

What We're Looking For

Essential Technical Skills

• Strong Databricks platform expertise
• Advanced PySpark
• Deep knowledge of:
  • Spark execution models
  • Performance tuning
  • Cost optimisation strategies

Engineering Background

• Experience aligned with:
  • Platform Engineer
  • DevOps Engineer
  • Site Reliability Engineer (SRE)
• Strong operations and optimisation focus (not feature development)