Senior Kafka Platform Engineer

Senior Software Engineer - Event Streaming PlatformLocations: Chicago or New YorkWe are looking for a skilled software engineer with extensive experience in Apache Kafka to help advance a modern event-streaming ecosystem. This position suits an individual who blends strong development expertise with a deep understanding of distributed systems and messaging platforms.In this role, you will be responsible for building and enhancing platform services, automation, and developer tools that support a scalable streaming infrastructure. You'll contribute both hands-on engineering and operational excellence to deliver a secure, reliable, and easy-to-use platform for internal teams.Key ResponsibilitiesBuild and maintain internal applications, APIs, and automation to streamline Kafka cluster provisioning, access controls, topic management, and operational processes.Design and manage production Kafka environments (self-hosted or managed services such as Confluent or cloud-native equivalents), including upgrades, scaling strategies, disaster recovery planning, and performance optimization.Deploy and operate Kafka workloads in Kubernetes environments using tools like Helm, operators, and GitOps workflows to ensure consistent, repeatable infrastructure delivery.Implement and support components such as Kafka Connect, schema management services, and cross-cluster replication technologies; standardize connector usage and promote reusable patterns.Strengthen platform reliability through service level objectives (SLOs), incident management practices, automated recovery mechanisms, and clear operational documentation.Develop comprehensive observability solutions, including metrics, logging, tracing, lag monitoring, and capacity reporting dashboards.Ensure platform security through encryption, authentication mechanisms, access controls, and compliance-focused automation practices.Promote best practices for event streaming, including topic structure, partitioning strategies, schema evolution, message ordering, and fault-tolerant processing approaches.Collaborate closely with application engineers, data teams, and infrastructure groups to improve adoption and usability of the platform.Contribute to technical leadership by mentoring team members and shaping platform direction, standards, and roadmap.Required QualificationsStrong collaboration and communication skills when working with engineering and platform teams.Significant hands-on experience operating Kafka at scale in production (including cluster internals, replication, partitioning, and recovery processes).Solid software engineering background, with a track record of delivering reliable production systems.Experience running stateful workloads in Kubernetes environments.Proficiency in infrastructure automation using Terraform, Helm, GitOps tools (e.g., Argo CD or Flux), and CI/CD pipelines.Programming experience in at least one of the following: Python, Go, or Java, along with command-line scripting and Linux fundamentals.Experience implementing monitoring, alerting, and performance tuning solutions for distributed systems.Familiarity with security practices including encryption, identity/authentication protocols, and secrets management solutions.Knowledge of Kafka ecosystem tools such as connectors, schema services, and replication frameworks.Experience working in public cloud environments and understanding networking and access control concepts.Proven experience in incident response, operational readiness, and driving improvements through post-incident reviews.Preferred QualificationsBackground in building developer platforms or self-service infrastructure solutions.Experience with stream-processing technologies such as Kafka Streams, Flink, or Spark Streaming.Familiarity with Kubernetes-native Kafka solutions (e.g., operator-based deployments).Knowledge of change data capture (CDC) tools and scalable database integration patterns.Experience designing multi-region architectures and implementing disaster recovery strategies.