Azure and Databricks Data Engineer designing, building and supporting data driven energy

Position: Azure and Databricks Data Engineer designing, building and supporting the data driven applications for our energy

Azure and Databricks Data Engineer designing, building and supporting the data driven applications for our energy client.

-

Location:

Oshawa - on site 3 days a week

Duration: 1 year

Work hours:

35hrs/week • * must be eligible for enhanced reliability security clearance**

Job Overview • Will build reliable, supportable & performant data lake & data warehouse products to meet the organization’s need for data to drive reporting analytics, applications, and innovation. • Will employ best practice in development, security, and accessibility and design to achieve the highest quality of service for our customers. • Build and productionize modular and scalable data ELT/ETL pipelines and data infrastructure leveraging the wide range of data sources across the organization. • Build curated common data models designed by the Data Modelers that offer an integrated, business-centric single source of truth for business intelligence, reporting, and downstream system use, in collaboration with Data Architect. • Work closely with infrastructure, and cyber teams and Senior Data Developers to ensure data is secure in transit and at rest. • Clean, prepare and optimize datasets for performance, ensuring lineage and quality controls are applied throughout the data integration cycle. • Support Business Intelligence Analysts in modelling data for visualization and reporting, using dimensional data modeling and aggregation optimization methods. • Troubleshoot issues related to ingestion, data transformation and pipeline performance, data accuracy and integrity. • Collaborate with Business Analysts, data scientists, Senior Data Engineers, data Data Analysts and, solution Architects and Data Modelers to develop data pipelines to feed our data marketplace. • Assist in identifying, designing, and implementing internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc. • Work with tools in the Microsoft Stack; Azure Data Factory, Azure Data Lake, Azure SQL Databases, Azure Data Warehouse, Azure Synapse Analytics Services, Azure Databricks, Microsoft Purview, and Power BI. • Work within the agile SCRUM work management framework in delivery of products and services, including contributing to feature & user story backlog item development, and utilizing related Kanban/SCRUM toolsets. • Assist in building data catalog and maintenance of relevant metadata for datasets published for enterprise use. • Develop optimized, performant data pipelines and models at scale using technologies such as Python, Spark and SQL, consuming data sources in XML, CSV, JSON, REST APIs, or other formats.

Qualifications • Completion of a four-year University education in computer science, computer/software engineering or other relevant programs within data engineering, data analysis, artificial intelligence, or machine learning. • Experience as a Data Engineer designing and building data pipelines. • Fluent in creating data processing frameworks using Python, PySpark, Spark

SOL and SOLExperience with Azure Data Factory, ADLS, Synapse Analytics and Databricks • Experience building data pipelines for Data Lakehouses and Data Warehouses • Good understanding of data structures and data processing frameworks • Knowledge of data governance and data quality principles • Effective communication skills to translate technical details to non-technical stakeholders