Senior Data Engineer

THE ROLE

We are recruiting on behalf of a highly secure engineering and technology programme seeking an experienced Senior Data Engineer to support the design, development and delivery of secure, scalable and high-performance data solutions within a regulated environment.

This role will focus heavily on the Elastic Stack and Apache NiFi, supporting enterprise-scale data ingestion, transformation and analytics capabilities across complex secure systems.

The successful candidate will play a key role in architecting resilient data platforms, optimising real-time and batch processing pipelines, and ensuring compliance with strict governance and security standards.

KEY RESPONSIBILITIES

Data Architecture & Engineering

• Design and implement scalable, resilient data pipeline architectures using Elastic Stack and Apache NiFi
• Develop secure frameworks for data ingestion, transformation and integration
• Build and maintain batch and real-time data processing pipelines
• Ensure high standards of data quality, integrity and traceability across all systems

Elastic Stack & Platform Management

• Configure, optimise and maintain Elasticsearch clusters for performance and resilience
• Develop and maintain Logstash pipelines for efficient data processing
• Create dashboards and visualisations within Kibana to support operational and strategic reporting
• Monitor and manage large-scale data flows and platform performance in real time
• Carry out troubleshooting, root cause analysis and performance tuning activities

Security, Governance & Resilience

• Work closely with security and governance teams to ensure compliance with stringent security standards
• Implement secure data handling processes including encryption, RBAC and audit logging
• Support clustering, replication and failover strategies to ensure system availability
• Develop backup, disaster recovery and data retention strategies

Documentation & Collaboration

• Produce technical documentation covering architecture, workflows and security controls
• Support knowledge transfer and operational handovers to internal teams
• Collaborate with engineering, infrastructure and DevOps teams across the programme

WHAT WE'RE LOOKING FOR

• Strong hands-on experience with: • Elasticsearch • Logstash • Kibana • Apache NiFi
• Proven experience designing and implementing enterprise-scale data pipelines
• Background working within secure or regulated environments such as defence, government or finance
• Strong understanding of: • Data ingestion and processing frameworks • Real-time and batch processing architectures • Data modelling for analytics • Distributed system troubleshooting and optimisation
• Knowledge of: • Encryption and secure data handling • RBAC and audit controls • DevOps and CI/CD principles • Git/version control systems
• Active DV Clearance essential
• Sole British National essential

DESIRABLE EXPERIENCE

• AWS, Azure or GCP within secure environments
• Docker and Kubernetes
• Kafka, Spark or additional data engineering technologies

WHY APPLY?

This is an opportunity to work on highly secure, technically advanced programmes within a complex and mission-critical environment. You'll be joining a team delivering enterprise-scale data solutions where security, resilience and performance are central to operational success.

ADDITIONAL INFORMATION

• Contract: Outside IR35
• Rate: £68 per hour
• Working Pattern: 5 Days On Site
• Security Clearance: DV Cleared
• Nationality Requirement: Sole British Nationals Only