Senior Big Data Engineer

About

We are looking for an experienced Senior Big Data Engineer with strong expertise in Scala, Apache Spark, and Apache Kafka to design and build large-scale, high-performance data processing systems. The ideal candidate will have hands-on experience in real-time and batch data pipelines and a deep understanding of distributed computing.

Responsibilities

• Provide consulting services on new initiatives (small to large of varying complexity).
• Explore new emerging technologies and how they best suit our applications
• Develop, code, document, and execute unit tests, system, integration, and acceptance tests using different languages and testing tools for functions of high complexity.
• Ensure adequate technical documentation and training.
• Optimize spark jobs and java applications.
• Architect, design, and implement solutions that meet the stakeholder’s needs
• Participate actively in requirements gathering, data modeling, and design sessions
• Prepare high-level and detailed technical specifications for the projects in accordance with PLC, security, and architecture documentation objectives
• Develop detailed plans and accurate estimates for the completion of build, system testing, and implementation phases of a project
• Develop, code, document, and execute unit tests, systems, integration and acceptance tests, and testing tools for functions of high complexity
• Write, test, and maintain detailed programs according to specifications given by computer software engineers and systems analysts

Knowledge and Experience

• Strong Scala programming experience (8+ years)
• Apache Spark (Core, SQL, DataFrames/Datasets, Spark Streaming or Structured Streaming) Apache Kafka (Architecture, Topic design, partitioning, consumer groups, offset management)
• Experience with batch and real-time data processing
• Strong understanding of distributed systems, concurrency, and fault tolerance

What do you need to succeed?

Must Have

• Experience in developing and optimizing Big Data applications using Scala and Spark on Cloudera/HDP.
• Experience in building data pipelines
• Experience in developing/designing micro-service architecture.
• Working knowledge of Jenkins CI, Git, JIRA
• Ability to seek improvements to all aspects of the development process

Nice to have

• Knowledge of Credit Risk Markets.