Azure Data Engineer

About

We are seeking an Azure Data Engineer with strong, hands-on Microsoft Fabric experience to build and operate a scalable middle-layer data and integration platform. The role focuses on ingesting data from diverse sources, implementing transformation and business rules, and delivering curated, consumption-ready datasets/models for analytics and downstream applications.

Key Responsibilities

Fabric Implementation

• Work on the fabrics platform to design and implement robust data solutions, including One Lake architecture for efficient data storage and processing.

Build & Optimize Data Pipelines

• Design, develop, and maintain scalable ingestion and transformation pipelines using Microsoft Fabric (Data Factory in Fabric / Pipelines), ADF/Synapse Pipelines, One Lake storage patterns, PySpark, Python, and SQL across structured and unstructured data.
• Develop pipelines that ingest data from external APIs on a scheduled basis and initiate end-to-end downstream processing, supporting one or multiple daily runs through to curated and consumption-ready layers.

Data Ingestion & Integration

• Integrate data from cloud and on-prem sources including databases, third-party systems, files, and REST/SOAP APIs (auth, throttling, pagination, retries, and error handling).
• Build curated layers and consumption-ready models; implement incremental and batch processing logic; apply data modelling and transformation best practices aligned to reporting/analytics needs.

SQL Development & Tuning

• Develop and optimize complex queries, stored procedures, views, and datasets for efficient analytics and reporting; partner with analytics teams to meet performance SLAs.

Performance Tuning & Cost Optimization

• Tune Spark jobs, ADF data flows and SQL workloads (partitioning, caching, parallelism, cluster sizing/configs) to improve reliability and reduce runtime/cost.

Business Logic Implementation

• Translate requirements into scalable rules (validation, eligibility, availability calculations), manage exceptions, audit logging, and ensure data consistency across systems.

Data Quality & Validation

• Implement automated data quality checks, validation frameworks, reconciliation, and monitoring to ensure trusted datasets.

Security & Compliance

• Implement secure access via Azure AD, Managed Identities, RBAC, least privilege, and secure connectivity to data lake, Fabric/Synapse, and APIs.

Automation & CI/CD

• Build deployment automation using Azure Dev Ops/Git, promoting code across environments with consistent release practices; support testing and release activities.

Monitoring & Troubleshooting

• Monitor pipelines and jobs using Spark UI and Azure Log Analytics; triage failures, perform root-cause analysis, and improve resiliency/runbooks.
• Work closely with architects, platform/Dev Ops engineers, analysts, and data scientists; participate in design sessions and code reviews; operate within Agile/Scrum delivery.

Tools & Technologies

Fabric

• Microsoft Fabric Work spaces
• One Lake
• Fabric Pipelines / Data Factory in Fabric
• Lakehouse/Warehouse (as applicable)
• ADLS Gen2
• Blob Storage
• Synapse Analytics
• App Service (as needed)

Languages

• Py Spark
• Python
• SQL (T-SQL)

Dev Ops

• Git
• Terraform (preferred)

Monitoring

• Spark UI

Data Governance

• AI Tools

Role Experience

• Experience: 5+ years
• Role Level: Individual Contributor (Senior Engineer)

About Symbiotic Digital

Symbiotic Digital provides the IT Experts you need that solve problems and get things done. See what the top 2% in their field can do for you. (Symbiotic Digital is a division of Symbiotic Group Inc.)

We serve customers in two ways, we provide:

1. Digital Experts: IT Consulting expertise. Bright Minds That Produce Proven Results

2. Recruitment: Find The Right People You Need

Serving IT & Business leaders, Symbiotic Digital provides a business system that enable us to provide the Expert Technical staff needed to solve challenging business and IT problems. IT leaders in the most reputable organizations in Canada have relied on Symbiotic Digital to help them staff difficult assignments with proven Experts uniquely evaluated by our proven QMS - Quality Management System Staff Development Model. We are a 100% Indigenous owned company.