Senior Azure Data Engineer – Commodity/Energy trading data

Key Responsibilities

• Design, implement, and maintain scalable data engineering pipelines using Python, Spark, and SQL, applying clean coding principles, and optimizing for performance in Azure and Databricks.
• Use good software engineering principles to enable modularity, reliability, and reusability of code.
• Orchestrate complex data workflows using frameworks like Airflow, Databricks workflows (or) equivalent, adopting a code-first approach.
• Provide data engineering support for trading analysts in NY.
• Use CI/CD tools like GitHub actions or equivalent for deploying data pipeline to higher environments efficiently.
• Implement data quality checks, validation frameworks, and monitoring systems to ensure data integrity and reliability.
• Design effective data models, optimize storage and retrieval processes, and efficiently handle semi-structured and unstructured data.
• Create and manage APIs for data access and integration and develop web scraping solutions using leading frameworks when necessary.
• Build scalable data ingestion pipelines for various external data feeds with different formats, velocity, and patterns.
• Collaborate closely with trading analysts in an agile environment, iterating data products in an efficient manner demonstrating strong understanding of the domain context.
• Apply strong analytical and logical problem-solving skills to address complex data engineering challenges and develop innovative solutions.
• Ensure data security and compliance with relevant regulations, implementing best practices for data protection and access control.
• Continuously optimize pipeline performance, evaluate new technologies, and drive innovation in the data engineering ecosystem.
• Maintain comprehensive documentation and contribute to the team's knowledge base, fostering a culture of continuous learning and improvement.
• Collaborate closely with data engineers in London to implement as per standardized guidelines and guardrails.

Qualifications and Requirements

• Degree educated in either Computer Science, Mathematics, Engineering, or a related field.
• Strong hands-on experience with Python, Spark, SQL and strong knowledge of clean coding principles.
• Strong understanding of Data Governance best practices (DQ handling, Lineage..etc)
• Experience in handling various data ingestion patterns.
• Strong accountability and Entrepreneurialism in day-to-day work.
• Storing working experience with Data Orchestration (Airflow), Containerization (Docker, K8) and CI/CD tools.
• Desirable to have working knowledge in Microsoft Azure and Databricks.
• Advantageous to have experience working with Commodity/Energy trading data.
• Demonstrated ability to work effectively both independently and as part of a collaborative global team following internal Data engineering standards.
• Excellent problem-solving skills and a strong attention to detail.
• Strong stakeholder communication skills.

Compensation and Benefits

A reasonable, good faith estimate of the minimum and maximum base salary for this position is $180 K to $200 K per year with Bonus. Employee benefits will also be available, and details are available like Pension/401K/ Paid Vacation/ Life, Medical & Dental insurance etc.

The pay range listed above reflects the expected starting salary /Pay rate for this role. This range may be adjusted based on market conditions, location, and other relevant factors. The Company will determine the final starting salary/Pay rate in consultation with the selected candidate(s), in full compliance with applicable laws