Senior Performance Engineer- Hybrid Working

Company Description

ReKnew empowers enterprises to harness the full potential of their data and digital assets. Through its innovative platform of pre‑built accelerators, the company enables rapid delivery of impactful use cases such as risk management, compliance, fraud detection, and complaint resolution. By modernizing data infrastructure, ReKnew ensures secure, reliable AI adoption tailored to business needs. As a growing organization, ReKnew is committed to building a team of top talent to drive this mission forward. Must be able to join immediately.

Role Overview

As Senior Databricks Engineer, you will design, build, and optimise the Lakehouse data platform that underpins AI‑driven automation across the business. You will own the medallion architecture on Delta Lake, build production‑grade data pipelines, enforce governance through Unity Catalog, and ensure the data layer performs at scale for ML models, real‑time analytics, and intelligent agent workflows.

Responsibilities

Lakehouse Architecture & Data Pipelines

• Design and implement a scalable Bronze → Silver → Gold medallion architecture on Delta Lake
• Build and maintain production ETL/ELT pipelines using Databricks Workflows, Delta Live Tables (DLT), and Apache Spark
• Integrate structured and unstructured data from diverse enterprise sources including ERPs, CRMs, APIs, and IoT platforms
• Optimise Databricks SQL endpoints for high‑performance analytics and downstream reporting

Governance, Quality & Security

• Implement Unity Catalog for end‑to‑end data governance, lineage tracking, and fine‑grained access control
• Define and enforce data quality rules, schema evolution policies, and pipeline SLA monitoring
• Ensure compliance with local data residency, privacy, and regulatory requirements
• Manage secrets, credentials, and sensitive data handling in line with enterprise security standards

ML & AI Enablement

• Build and maintain feature stores and curated datasets to support ML model development and inference
• Collaborate with data science and AI engineering teams to ensure low‑latency, high‑quality data access for models and agents

Integration & Operations

• Design and manage data integration pipelines across cloud services (Azure Synapse, Data Factory, or equivalent)
• Implement real‑time and near‑real‑time streaming pipelines for event‑driven data ingestion
• Establish comprehensive monitoring, alerting, and observability for all data infrastructure
• Maintain CI/CD pipelines for data assets using Databricks Asset Bundles, Terraform, or equivalent tooling

Requirements

• 5+ years of hands‑on experience with Databricks (including Delta Lake, Unity Catalog, DLT, and MLflow)
• Strong proficiency in Apache Spark (PySpark / Scala), SQL, and Python
• Demonstrated experience building medallion architectures (Bronze/Silver/Gold) at enterprise scale
• Deep knowledge of Azure cloud services: Azure Data Factory, Synapse Analytics, Azure Storage, and Key Vault
• Experience integrating ERP systems (SAP, Oracle, or similar) and third‑party APIs into a Lakehouse platform
• Solid understanding of data governance, data quality frameworks, and regulatory compliance
• Experience with CI/CD for data pipelines (Databricks Asset Bundles, Terraform, or equivalent)