Software Engineer - Kafka & NiFi Admin with Ansible & Terraform

Missions

Position Overview

We are seeking an experienced Senior Kafka & NiFi Administrator to design, deploy, manage, and optimize enterprise-scale data streaming and data flow platforms. The ideal candidate will have deep expertise in Apache Kafka, Kafka Connect, Schema Registry, and Apache NiFi, along with strong automation, scripting, CI/CD, and cloud-native experience. This role ensures high availability, secure configurations, performance tuning, and operational excellence for real-time data pipelines.

Key Responsibilities

1. Kafka Administration

• Install, configure, and manage Apache Kafka clusters (on-premise or cloud-native: Azure HDInsights).
• Manage Kafka ecosystem components: Kafka Connect, Schema Registry, Kafka Streams, ZooKeeper, etc.
• Perform cluster scaling, partition rebalancing, topic management, and multi-DC replication (MirrorMaker).
• Implement monitoring, alerting, and logging using tools such as Prometheus, Grafana, ELK/ECE.
• Ensure Kafka security (TLS encryption, SASL, RBAC, Kerberos, OAuth).
• Troubleshoot broker issues, performance bottlenecks, consumer lag, and message serialization errors.

2. NiFi Administration

• Install, manage, and maintain Apache NiFi and NiFi Registry.
• Design, optimize, and troubleshoot complex NiFi data flows.
• Manage NiFi cluster configuration, back-pressure settings, tuning, and provenance repository.
• Integrate NiFi with Kafka, S3, HDFS, RDBMS, REST APIs, and cloud services.
• Implement access control, SSL/TLS security, policies, and NiFi user/group management.

3. Automation & DevOps

• Develop automation using Ansible, Terraform, or Bash.
• Build CI/CD pipelines for Kafka, NiFi, and data flow deployments using Jenkins, Azure DevOps, or GitHub Actions.
• Automate cluster provisioning and configuration using Ansible and Terraform.
• Create reusable templates and automation for topic creation, ACL management, connector deployment, and flow lifecycle.

4. Operations & Support

• Provide L3 support for streaming platforms, including incident analysis and root cause identification.
• Establish and enforce best practices for data governance, data flow reliability, and operational standards.
• Maintain detailed documentation for configurations, architectures, and runbooks.
• Collaborate with platform engineering, data engineering, security, SRE, and cloud teams.

Key Skills Required

Must Have

• 6–10+ years of experience in Kafka & NiFi administration.
• Strong knowledge of Big Data architecture and administrator responsibilities.
• Strong knowledge of Hadoop, Kafka internals, NiFi flow design, performance tuning, various data formats, Schema Registry, Kafka Connect, etc.
• Configuration and performance tuning of Kafka & NiFi clusters.
• Application deployment and disaster recovery.
• Automation of Big Data infrastructure using Ansible and Terraform.

Good to Have

• Java and shell scripting.
• Hadoop administration.
• Excellent communication skills.

Profile

Must Have

• 6–10+ years of experience in Kafka & NiFi administration.
• Strong knowledge of Big Data architecture and administrator responsibilities.
• Strong knowledge of Hadoop, Kafka internals, NiFi flow design, performance tuning, various data formats, Schema Registry, Kafka Connect, etc.
• Configuration and performance tuning of Kafka & NiFi clusters.
• Application deployment and disaster recovery.
• Automation of Big Data infrastructure using Ansible and Terraform.

Good to Have

• Java and shell scripting.
• Hadoop administration.
• Excellent communication skills.

CGI provides a reasonable estimate of the salary range for this position. The calculation of this range depends on various factors, including skill level, geographic market, experience, education, as well as professional licenses and certifications. Compensation decisions are based on the specifics of each case. A reasonable estimate for this salary range is between $60,000 and $115,000. This position is currently open.

#LI-AM1

Your future duties and responsibilities

Required qualifications to be successful in this role

Together, as owners, let’s turn meaningful insights into action.

Life at CGI is rooted in ownership, teamwork, respect and belonging. Here, you’ll reach your full potential because…

You are invited to be an owner from day 1 as we work together to bring our Dream to life. That’s why we call ourselves CGI Partners rather than employees. We benefit from our collective success and actively shape our company’s strategy and direction.

Your work creates value. You’ll develop innovative solutions and build relationships with teammates and clients while accessing global capabilities to scale your ideas, embrace new opportunities, and benefit from expansive industry and technology expertise.

You’ll shape your career by joining a company built to grow and last. You’ll be supported by leaders who care about your health and well-being and provide you with opportunities to deepen your skills and broaden your horizons.

At CGI, we value the strength that diversity brings and are committed to fostering a workplace where everyone belongs. We collaborate with our clients to build more inclusive communities and empower all CGI partners to thrive. As an equal-opportunity employer, being able to perform your best during the recruitment process is important to us. If you require an accommodation, please inform your recruiter.

To learn more about accessibility at CGI, contact us via email. Please note that this email is strictly for accessibility requests and cannot be used for application status inquiries.

Come join our team—one of the largest IT and business consulting services firms in the world.