Lead Data Engineer / Architect

About

Palantir Foundry Lead-Architect.

We are seeking a highly experienced Foundry Lead Data Engineer to architect, develop, and optimize scalable data solutions using Palantir Foundry. This senior role is ideal for a data engineering expert who thrives in complex, ontology-aware environments and can lead the design of robust data pipelines and applications that drive actionable insights.

You will play a pivotal role in shaping our data strategy, collaborating with stakeholders across the organization, and ensuring the integrity, performance, and cost-efficiency of our data ecosystem. While familiarity with Generative AI (GenAI) is a plus, your core expertise should lie in data architecture, pipeline development, and Foundry platform mastery.

Key Responsibilities

Data Architecture & Pipeline Engineering

• Design, build, and maintain end-to-end ETL/ELT pipelines using PySpark, Spark SQL, and Python.
• Architect scalable data frameworks that support high-performance analytics and operational workflows.
• Conduct data cleansing, transformation, and validation to ensure data quality and consistency.

Palantir Foundry Platform Expertise

• Configure and manage data connections, Pipeline Builder, and Contour for data exploration and visualization.
• Develop and maintain ontology objects, ensuring semantic consistency and reusability across applications.
• Manage code repositories and version control within Foundry, promoting modular and maintainable engineering practices.

Ontology-Aware Application Development

• Collaborate with data owners and domain experts to ingest, transform, and model data into ontology-driven applications such as Workshop and Quiver.
• Maintain clarity on the end-to-end architecture and data flow, ensuring seamless integration from ingestion to ontology.

Cross-Functional Collaboration & Leadership

• Partner with data scientists, analysts, and business stakeholders to translate requirements into scalable data solutions.
• Engage with senior leadership to maintain a feedback loop for Foundry program improvements.
• Provide mentorship and technical guidance to junior engineers and analysts.

Operational Excellence & Cost Management

• Establish and maintain cost monitoring and reporting for Foundry-related operations.
• Lead efforts in troubleshooting, performance tuning, and platform stability.

Continuous Improvement & Innovation

• Stay current with Foundry platform updates, emerging features, and best practices.
• Identify opportunities for automation, optimization, and innovation in data workflows.
• Explore and evaluate GenAI capabilities for potential integration into data processes.

Required Qualifications

• 8+ years of experience in data engineering or related roles.
• Strong proficiency in PySpark, Spark SQL, and Python.
• Deep hands-on experience with Palantir Foundry, including:
  • Data Connections
  • Pipeline Builder
  • Contour
  • Code Repositories
  • Ontology Management
  • Workshop and Quiver
• Solid understanding of data modeling, data governance, and access control.
• Proven ability to troubleshoot complex data issues and optimize performance.
• Excellent communication and collaboration skills.

Preferred Qualifications

• Experience with cloud platforms (e.g., Azure).
• Familiarity with Snowflake and its integration with Foundry.
• Exposure to Generative AI (GenAI) and its application in data workflows.
• Knowledge of data security, compliance, and metadata management.