Backend Data Engineer

Position Overview

Seeking a skilled Backend Data Engineer to join our Technical Data Management (TDM) team in Richmond. This role is crucial for an ERP modernization project involving the migration of legacy systems to Workday and managing sensitive data movement. You will focus on building, automating, and securing data pipelines, maintaining infrastructure (specifically EMR clusters), and contributing to a patent-pending data management framework built in Scala. This position sits at the intersection of data engineering, software engineering, and production support.

Responsibilities

• Build & Maintain Data Pipelines: Develop and maintain secure, scalable data pipelines using Python and AWS Lambda for orchestration, and a proprietary Scala-based framework for core data processing.
• Automation: Focus heavily on automating all aspects of data movement and self-service data management tools.
• Security & Auditing: Implement stringent data obfuscation techniques according to cyber guidelines. Perform constant updates to ensure security as every keystroke and data movement is audited.
• Infrastructure Management: Monitor and maintain the backend infrastructure, including AWS EMR clusters, managing vulnerabilities, and ensuring continuous operations.
• Collaboration: Work closely with a sister team managing the self-service UI and other stakeholders in a highly collaborative environment.
• Support: Provide production support, including debugging issues in file transfers and pipeline execution.

Qualifications

Top-Notch / Mandatory Skills:

• Scala (Programming Language): Strong programming experience required.
• Python (Programming Language): Essential for writing AWS Lambda functions and pipeline orchestration.
• Spark (Framework): Real-world experience building data frameworks.
• AWS (Cloud Platform): Deep knowledge is crucial, including KMS (Key Management Service), IAM (Identity & Access Management) roles, and S3 bucket policy updates.
• Shell Scripting: Required for day-to-day debugging and automation.

Required Experience & Attributes:

• Large Enterprise Background: Experience working in large-scale enterprise environments.
• Ramp-up Ability: Demonstrated ability to quickly learn new contexts, systems, and teams, with a proven history of ramping up in as little as 1-2 weeks.
• Communication & Collaboration: Strong communication skills and a preference for in-person collaboration to solve complex problems and niche skill-related challenges.