GOVERNMENT REPRESENTATIVE 1 SENIOR DATA PLATFORM ENGINEER

Job Description

The Senior Data Platform Engineer reports to the Assistant Director in the Data Platform Unit of the Office of Data Management and Reporting. The Senior Data Platform Engineer designs, builds, and operates cloud native data pipelines, storage, compute, and serving layers. Applies expertise in distributed processing, orchestration, and cloud engineering to deliver scalable, high quality data products and services. This role is hands-on, code oriented, and production focused, capable of deep-diving into performance, reliability, and cost optimization while ensuring consistency, security, and data integrity across the platform.

Data Engineering

• Design, develop, and optimize data pipelines that extract, process, and deliver data from diverse structured and unstructured sources.
• Apply distributed computing frameworks (e.g., Spark) for large-scale transformation and processing operations, ensuring performance, reliability, and scalability.
• Build and maintain batch, streaming, API, and CDC ingestion patterns.
• Integrate data from databases, data warehouses, APIs, and external systems while ensuring data consistency and integrity.
• Apply data cleansing, enrichment, aggregation, and filtering techniques as part of transformation workflows.
• Tune and optimize data pipelines by identifying performance bottlenecks and applying techniques such as efficient partitioning, indexing, and caching.

Cloud Platform Engineering & Infrastructure Automation

• Contribute to the strategic direction for implementation of InfrastructureasCode (Terraform/Bicep/CloudFormation) to provision, manage, and secure cloud resources.
• Build robust CI/CD pipelines to automate deployment of data pipelines, models, and transformations.
• Recommend cloud security best practices including RBAC/ABAC, key management, secrets management, encryption, network controls, and data access patterns.
• Monitor and tune cloud resource utilization for both cost efficiency and performance.

Observability, Reliability & Production Operations

• Implement platform observability using logs, metrics, traces, dashboards, and endtoend lineage.
• Configure runtime orchestration and workflow management (Airflow, ADF, Databricks Jobs, Step Functions, etc.).
• Provide oncall support for production systems; lead incident response, triage, and root-cause analysis.
• Establish and maintain SLOs for latency, data freshness, reliability, and resource efficiency.

Data Quality, Lineage & Governance Integration

• Implement automated data validation, quality checks, and integrity rules within pipelines.
• Ensure lineage is captured and integrated with enterprise metadata/catalog platforms.
• Work closely with the Data Governance team to apply definitions, steward rules, and classification policies.
• Help enforce data consistency and quality standards across all ingestion and transformation workflows.

Collaboration, Leadership & Reuse

• Collaborate with analytics, data governance, MDM, and data product teams to ensure alignment with organizational standards.
• Mentor junior engineers and contribute to establishing engineering best practices.
• Develop reusable templates, modules, libraries, and patterns to increase engineering productivity and consistency.
• Participate actively in release readiness, code review, and change control processes.

General Responsibilities

• Perform additional duties as assigned.

EXPERIENCE:

Minimum Qualifications

• 5-8+ years in cloud data engineering roles.
• Strong hands-on proficiency in Python, SQL, Spark, orchestration frameworks, and CI/CD.
• Experience integrating and validating data from multiple storage systems, APIs, and external systems.
• Experience with data quality, data validation, and lineage tooling.
• Hands-on experience with Azure or AWS data services (e.g., ADLS/S3, ADF/Glue, Databricks/EMR).

Preferred Qualifications

• Prior experience with Oracle GoldenGate is strongly desired.
• Experience with dbt, Delta Lake, and data observability or monitoring platforms.
• Familiarity with event-driven architectures (Kafka/EventHub/Kinesis).
• Public sector or HHS program experience.
• Expertise in pipeline tuning and performance optimization.
• Experience with BI/analytics tools to support downstream modeling efforts.

EMPLOYEE BENEFITS:

The Department of Children and Families provides many employee benefits, including but not limited to:

• Alternative Workweek Program*
• Deferred Compensation
• Health, Dental and Life Insurance
• Flexible and Health Spending Accounts
• Pension
• Telework*
• Public Service Loan Forgiveness
• Benefit Leave (Vacation, Sick, Administrative Leave)
• 13 Paid Holidays
• Pursuant to Department policy, procedures and/or guidelines