Data Engineer / Databricks Developer

Job Overview

We are seeking an experienced Data Engineer / Databricks Developer with a strong background in the Investment Banking domain to join our team in New York. The ideal candidate will be responsible for designing, developing, and optimizing scalable data solutions using Azure-based technologies, with a focus on Databricks and modern data architecture.

Key Responsibilities

• Design, develop, and implement scalable data pipelines using Azure Databricks and Azure Data Factory.
• Build and maintain robust ETL/ELT workflows using Apache Spark and PySpark DataFrames.
• Develop high-performance data transformation processes using Spark and SQL.
• Optimize data pipelines for efficient ingestion, processing, and transformation of large datasets.
• Implement and maintain data governance frameworks, including data access, security, compliance, and lifecycle management.
• Ensure data quality, consistency, and lineage across enterprise data platforms.
• Design and support modern Lakehouse architecture and distributed data processing systems.
• Apply Master Data Management (MDM) principles to maintain data integrity and standardization.
• Collaborate with cross-functional teams to deliver scalable, reliable, enterprise-grade data solutions.

Required Skills & Qualifications

• Strong hands-on experience with Azure Databricks
• Proficiency in Apache Spark / PySpark
• Experience with Azure Data Factory
• Advanced knowledge of SQL
• Expertise in building data pipelines and ETL/ELT processes
• Experience with data governance, data lineage, and data quality frameworks
• Strong understanding of Lakehouse architecture
• Experience working in Investment Banking or Financial Services domain (Mandatory)

Preferred Qualifications

• Experience with large-scale distributed data systems
• Familiarity with cloud-based data platforms and enterprise data architecture
• Strong problem-solving and performance optimization skills

Additional Information

• Only local candidates will be considered.
• In-person interview is mandatory.