Data Engineer - Apache

About

This position focuses on Data pipelines & workflows.

Education and Experience

• Bachelor’s degree in computer science, information systems, data engineering, or related field, or equivalent practical experience. May consider an Associates if the candidate has an additional 3-5 years experience than what is being required.
• 2+ years of professional experience in data engineering, ETL development, or related work, or equivalent hands-on experience
• Experience or interest in scientific software, materials science, research environments, or technically complex domains is a plus

Scope of the position

• Embed within a cross-functional Agile team, participating in sprint planning, stand-ups, backlog refinement, and technical discussions.
• Design, build, troubleshoot, and maintain ETL/ELT workflows that support application functionality, analytics, reporting, and scientific workflows.
• Develop and manage data pipelines using Apache Airflow, ensuring reliable orchestration, scheduling, monitoring, and recovery of data processes.
• Work with stakeholders including software developers, scientists, and engineers to understand data sources, workflow requirements, and downstream data needs.
• Extract, transform, validate, and load data across systems, including relational databases such as Postgres SQL and Oracle.
• Write, optimize, and maintain complex SQL queries, scripts, and transformation logic to support operational and analytical use cases.
• Troubleshoot data quality issues, ETL failures, pipeline bottlenecks, and schema inconsistencies; identify root causes and implement durable solutions.
• Support database exploration, data validation, and troubleshooting using tools such as DBeaver and related database utilities.
• Evaluate and help adopt new data tools and technologies, including lightweight analytics and transformation solutions (e.g. DuckDB) where appropriate.
• Collaborate with engineering teams to support reliable integration between data pipelines, applications, APIs, and downstream consumers.
• Assist with schema evolution, data modeling, migration planning, and data consistency across systems.
• Document pipeline logic, data dependencies, transformation rules, and operational procedures to support maintainability and team knowledge sharing.
• Help improve data engineering standards, observability, testing practices, and operational reliability across the team.
• Regularly interacting with scientists and engineers to understand research and technical workflows; experience in scientific or research environments is a strong plus.

Technical Skills – 2+ years (or commensurate experience):

• Experience designing, building, and troubleshooting ETL/ELT pipelines
• Hands-on experience with workflow orchestration tools, preferably Apache Airflow
• Strong experience writing and optimizing SQL
• Experience working with relational databases, especially Postgres SQL and Oracle
• Ability to develop and maintain data transformations, validation steps, and pipeline logic across multiple systems
• Experience with database tools such as DBeaver or similar for query development, exploration, and troubleshooting
• Familiarity with modern data processing and analytical tools such as DuckDB or interest in evaluating emerging data technologies
• Understanding of data modeling, schema design, data integrity, and performance tuning
• Experience troubleshooting pipeline failures, performance issues, and inconsistent or incomplete datasets
• Familiarity with scripting or programming for pipeline development and automation; Python experience is strongly preferred
• Understanding of version control and collaborative development workflows
• Experience supporting production data systems with an emphasis on reliability, maintainability, and clear documentation