Lead Data Engineer

Title: Lead Data Engineer

Contract Duration: 12 Months+

Location: Dublin, Ireland(Hybrid)

We are seeking an experienced Lead Data Engineer to join a high-impact team supporting Financial Institutions. This role is central to building resilient, governed, and scalable data platforms that power advanced analytics and detection capabilities.

You will play a key role in designing and evolving a Databricks + AWS lakehouse, enabling investigators, data scientists, and product teams to uncover criminal behaviour and act with confidence. This is a hands-on leadership role combining deep technical expertise with strong ownership, mentoring, and stakeholder collaboration.

Responsibilities: • Own the end-to-end design, build, optimisation, and support of scalable Spark / PySpark pipelines on Databricks (batch & streaming). • Define and enforce Lakehouse & Medallion architecture standards (Bronze/Silver/Gold), including governance, lineage, quality SLAs, and cost controls. • Architect and maintain secure, compliant AWS data infrastructure (S3, IAM, Glue, Lake Formation, KMS, Lambda, Step Functions, EKS/EC2). • Lead data ingestion using Apache NiFi, APIs, SFTP/FTPS, onboarding diverse internal and external datasets. • Implement robust orchestration using Airflow, Databricks Workflows, and Step Functions, with strong observability and reliability patterns. • Champion data quality, reliability, and observability, including expectations, anomaly detection, SLIs/SLOs, alerting, and runbooks. • Embed metadata and lineage (Unity Catalog, Glue, OpenLineage) to support auditability and regulatory transparency. • Drive CI/CD and Infrastructure as Code practices for data assets across environments. • Mentor engineers on Spark performance, Delta Lake optimisation, partitioning strategies, and cost/performance trade-offs. • Collaborate closely with data science, product, security, and compliance teams to deliver trusted, production-grade data solutions. • Lead technical design reviews, code reviews, incident response, and continuous improvement initiatives.

Experience Required: • Expert-level SQL skills with strong hands-on experience in Databricks, Snowflake, Python, and PySpark. • Proven production experience building and optimising large-scale Spark pipelines (Delta Lake, Photon, cluster tuning). • Strong AWS data ecosystem expertise, including security, networking, encryption, and cost optimisation. • Hands-on orchestration experience with Airflow, Databricks Workflows, and Step Functions. • Solid experience with CI/CD, Git workflows, and IaC (Terraform / CloudFormation). • Deep understanding of data governance, lineage, and compliance (PII/PCI, retention, access controls). • Demonstrated ability to lead, mentor, and influence, working effectively with both technical and non-technical stakeholders. • Pragmatic, delivery-focused mindset with experience in incident management and on-call readiness. • Bonus: Financial Crime domain exposure, Python packaging, OpenTelemetry, advanced observability practices.