Big Data Engineer with Kubernetes Expertise

About the Role

Join our dynamic team as a Big Data Engineer to drive the design and optimization of large-scale data processing on AWS using Spark and Kubernetes. In this role, you will implement containerized workloads on EMR powered by EKS, develop scalable data pipelines, and enhance performance, reliability, and observability. You will work collaboratively with cross-functional teams, leveraging your Spark tuning skills, and managing Kubernetes infrastructure to facilitate data-driven outcomes. Prior experience in the financial industry is a plus!

We offer W2 and corp-to-corp consulting arrangements. Our W2 consultants enjoy an excellent benefits package that includes Medical, Dental, Vision, 401k with company matching, and life insurance.

Responsibilities

• Design, develop, and maintain large-scale data processing pipelines using Hadoop, Spark, Python, and Scala.
• Architect and deploy containerized big data workloads on Amazon EMR on EKS.
• Design and implement Kubernetes-based infrastructure for running Spark applications at scale.
• Implement scalable ingestion, storage, transformation, and analysis solutions.
• Stay current with industry trends and emerging big data technologies to improve architecture.
• Collaborate with cross-functional teams to translate business requirements into technical solutions.
• Optimize and enhance existing data pipelines for performance, scalability, and reliability.
• Develop automated testing frameworks and implement continuous testing for data quality.
• Conduct unit, integration, and system testing for data pipeline robustness and accuracy.
• Support data scientists and analysts with reliable datasets and tooling.
• Monitor and troubleshoot production data pipelines and resolve issues.
• Manage Kubernetes clusters, pods, services, and deployments for big data workloads.

Experience Requirements

• Hands-on experience with AI development tools such as GitHub Copilot, Q Developer, ChatGPT, or Claude.
• Proficiency with Hadoop, Spark, Hive, and Trino.
• Strong experience with Kubernetes, including pods, services, deployments, namespaces, ConfigMaps, and Secrets.
• Experience with EMR on EKS for Spark workloads.
• Expertise in Kubernetes resource management, scheduling, and auto-scaling.
• Knowledge of Helm charts, Kubernetes networking, PVs/PVCs, security best practices, kubectl, and YAML manifests.
• Deep understanding of Spark internals including executors, tasks, stages, and DAGs.
• Proven ability to troubleshoot cluster issues and optimize Spark jobs.
• Strong AWS experience including S3, EMR, EMR on EKS, Glue, and Lambda.
• Excellent programming skills in Python or Scala, writing clean and efficient code.
• Advanced SQL skills and experience with ETL/pipeline management.
• CI/CD experience with tools like Jenkins, GitLab CI, or GitHub Actions.

Education Requirements

• Bachelor's degree in Computer Science, Information Systems, or related field; Master's degree preferred.
• AWS and Kubernetes certifications are a plus.

Join us and be part of our mission to leverage technology for impactful outcomes within the financial services industry!