Cloud Data Engineer

About KDA Consulting Inc.

KDA Consulting Inc. is on the lookout for a talented Cloud Data Engineer to craft, construct, and oversee scalable data pipelines and architectures that support mission-critical programs within the Intelligence Community (IC). This exciting role is centered around facilitating efficient data processing, analytics, and decision-making by delivering high-quality, reliable data solutions in cloud environments.

The ideal applicant will possess significant experience with AWS cloud services, data engineering best practices, and big data technologies, and be adept at thriving in a fast-paced, collaborative atmosphere that supports complex mission systems.

Key Responsibilities

Data Pipeline Development & Engineering

• Design, develop, and maintain scalable ETL/ELT pipelines for data ingestion, transformation, and loading from various sources.
• Build and enhance data workflows for both batch and real-time processing.
• Ensure the high availability and reliability of data pipelines throughout enterprise systems.

AWS Cloud & Data Services

• Utilize AWS services such as S3, Glue, Lambda, Redshift, EMR, and Kinesis for data engineering solutions.
• Create and manage secure, scalable, and cost-effective cloud-based data architectures.
• Implement best practices for data storage, partitioning, and lifecycle management.

Data Optimization & Performance

• Enhance data storage and retrieval for top performance, scalability, and cost efficiency.
• Deploy indexing, partitioning, and query optimization strategies.
• Monitor and diagnose data pipeline performance issues.

Data Quality & Governance

• Guarantee data accuracy, consistency, and integrity across all pipelines and systems.
• Establish validation, monitoring, and error-handling systems.
• Support data governance practices and compliance with security and IC standards.

Collaboration & Integration

• Collaborate closely with data scientists, software engineers, and mission stakeholders to meet analytics and operational requirements.
• Integrate data solutions with enterprise applications, APIs, and downstream analytics platforms.
• Engage in Agile development methods and contribute to continuous improvement initiatives.

Requirements

• Active TS/SCI with Polygraph required.
• Bachelor's degree in Computer Science, Engineering, or a related technical field (or equivalent experience).
• Proven experience in data engineering and pipeline development.
• Extensive experience with AWS data services (e.g., S3, Glue, Redshift, Lambda, EMR).
• Proficiency in SQL and Python for data processing and transformation.
• Experience with large-scale data sets and big data technologies.
• Strong analytical and problem-solving capabilities.