Senior IT Data Engineer

Position Overview

The Senior IT Data Engineer will play a key role in designing, building, and maintaining scalable data pipelines and data platforms. This role involves working with complex data systems to ensure high-quality data processing, integration, and accessibility across multiple sources.

Responsibilities

• Design, develop, and maintain robust data pipelines and data processing systems
• Build and optimize ETL workflows for large-scale structured and unstructured data
• Ensure data quality, integrity, and performance across multiple data sources
• Integrate data from various platforms and formats into centralized systems
• Troubleshoot and resolve data flow and pipeline performance issues
• Collaborate with cross-functional teams to support data-driven initiatives
• Participate in Agile ceremonies (Scrum/Kanban) and contribute to continuous improvement
• Support CI/CD pipelines and automation for data platform deployment
• Maintain documentation for data processes, workflows, and systems

Required Qualifications

• Minimum 10 years of overall IT experience
• 5+ years of experience in data/application development (Python preferred)
• Strong experience building and managing data pipelines on Cloudera Data Platform
• Experience with ETL processes, data transformation, and performance tuning
• Hands-on experience with tools like PySpark, Pandas, or dbt
• Experience with data ingestion/integration tools such as Apache NiFi
• Advanced knowledge of SQL, Java, and Microsoft SQL Server
• Experience with distributed data technologies such as:
  • Hadoop
  • MapReduce
  • Hive
  • HBase
  • Kafka
  • Spark
• Experience working in UNIX/Linux environments, including shell scripting
• Familiarity with CI/CD pipelines and DevOps practices
• Experience working in Agile environments (Scrum/Kanban)

Preferred Qualifications

• Experience supporting government or regulated environments
• Strong understanding of data governance, security, and compliance practices