Lead Data Platform Engineer

About Us

We're architecting and building a modern, high-scale data platform to power real-time analytics and a next-generation Data Lake / Data Mesh architecture. Our environment handles massive throughput with Kafka processing over 1 million records per second and petabytes of total data, running on AWS EKS (Kubernetes) with a mix of on-prem and cloud components.

Join in a leadership role on a collaborative team where your work directly enables cutting-edge data streaming and processing. We value people who like to get their hands dirty and keep their skills sharp.

Perks

• Free GrubHub food deliveries to keep you fueled
• Free downtown parking
• Frequent team parties every 2 weeks – we celebrate wins and enjoy time together
• Work on high-impact, large-scale systems without after-hours support demands
• Opportunity to contribute to evolving initiatives like Data Lake, Data Mesh, Apache Iceberg, and real-time processing pipelines

What You'll Do

As a lead member of our team, you'll focus on designing, deploying, and maintaining our core streaming and orchestration platforms. This is a hands-on leadership role.

• Lead the setup, configuration, and ongoing management of Kubernetes clusters on AWS EKS, ensuring high availability and performance for data workloads.
• Design Apache Kafka operations at massive scale: create and manage topics, scale nodes/brokers, implement tagging/classification, tune for 1M+ records/second throughput, and handle petabyte-level storage.
• Deploy Kafka using modern tools like Strimzi (Kafka operator for Kubernetes), Confluent components, and Helm charts.
• Design and build integration and orchestration with tools such as Apache Spark, Airflow, Apache Beam, and emerging Flink pipelines for batch and streaming workloads.
• Contribute to building and enhancing our Data Lake and Data Mesh architecture, incorporating open table formats like Apache Iceberg for reliable, versioned data storage.
• Collaborate with data engineers and application teams to provision resources, troubleshoot issues, and implement best practices for observability, security, and cost efficiency.
• Automate infrastructure and deployments using IaC principles (e.g., Helm, Terraform where applicable) to enable self-service for teams.

What We're Looking For

• You're a hands-on architect who can delegate, director, and mentor other talented team members.
• Strong primary expertise in AWS, especially EKS (Kubernetes cluster management).
• Solid production experience with Apache Kafka at scale—topic creation/management, cluster scaling, performance tuning, and related ecosystem tools.
• Familiarity with Kafka deployment on Kubernetes via Strimzi, Confluent, or Helm.
• Working knowledge of related data processing tools: Spark, Airflow, Apache Beam, and ideally exposure to Flink or similar streaming engines.
• Experience or interest in modern data lake technologies like Apache Iceberg, Data Lake / Data Mesh patterns.
• Comfortable in a mixed on-prem and cloud (AWS) environment.
• Strong problem-solving skills, attention to detail, and a collaborative mindset.

If you're excited about high-throughput streaming on AWS EKS, Kubernetes-native platforms, and contributing to a growing Data Mesh/Data Lake initiative—while enjoying great perks and a balanced lifestyle—we'd love to hear from you!

Apply today and help us build the future of our data platform.