Azure Data Engineering Manager

No sponsorship available for this role; only USC and GC can apply

The Cloud Platform Manager is a senior technical leadership role responsible for designing, building, optimizing, and managing scalable enterprise data platforms within a cloud-based Azure ecosystem. This position plays a critical role in driving enterprise data engineering, analytics, machine learning enablement, and business intelligence initiatives through the development of modern Lakehouse architectures and high-performance data pipelines.

Key responsibilities include leading data engineering teams, architecting scalable Azure Databricks solutions, and implementing secure, high-quality enterprise data platforms that support analytics, reporting, and operational decision-making. The role combines hands-on technical expertise with leadership, governance, and strategic planning responsibilities to support enterprise-wide data initiatives.

Key Responsibilities • Lead, mentor, and support teams of data engineers, analysts, and technical professionals while promoting development standards and best practices. • Design, develop, deploy, and optimize scalable enterprise data platforms using Azure Databricks and Azure cloud technologies. • Build and maintain robust batch and streaming ETL/ELT data pipelines using PySpark, Scala, SQL, and Azure-native tools. • Architect modern Lakehouse environments utilizing Delta Lake and medallion architecture methodologies (Bronze, Silver, Gold layers). • Collaborate with business stakeholders, data scientists, and technical teams to gather requirements and translate them into scalable technical solutions. • Oversee data ingestion, transformation, integration, quality validation, governance, and storage optimization initiatives. • Implement data governance frameworks, security controls, RBAC access models, lineage tracking, and compliance processes using Unity Catalog and related technologies. • Integrate enterprise platforms with Azure Data Factory, Azure Data Lake Storage, Azure Synapse, Kafka, DevOps pipelines, and other cloud-native services. • Optimize Spark workloads, Databricks clusters, query performance, indexing strategies, partitioning, and cost efficiency within Azure cloud environments. • Support machine learning operations and model deployment initiatives utilizing MLflow and related MLOps tools. • Develop and maintain scalable data models, schemas, and enterprise analytics frameworks. • Participate in troubleshooting, production support, incident management, and continuous improvement initiatives for enterprise data environments. • Ensure data platform reliability, scalability, performance, and operational excellence across cloud infrastructure.

Minimum Education & Experience Requirements • Bachelor’s degree in Computer Science, Information Technology, Engineering, Data Science, or a related technical field required. • Minimum of 5–7 years of hands-on experience in data engineering, cloud architecture, or enterprise data platform development. • Minimum of 2–4 years of direct experience working with Azure Databricks and Azure cloud data technologies. • Prior experience leading or mentoring technical teams, including data engineers, analysts, or data scientists. • Strong experience developing enterprise-scale ETL/ELT pipelines and distributed data processing solutions. • Experience with modern cloud-based data platforms, big data technologies, and streaming architectures. • Strong background in SQL, Python/PySpark, Scala, and enterprise data modeling principles.

Special Requirements • Ability to work within a hybrid or remote enterprise technology environment. • Availability to support production incidents or critical deployments outside standard business hours when necessary. • Ability to work collaboratively across technical and non-technical business teams. • Participation in strategic planning, architecture reviews, and enterprise governance initiatives as required. • Relevant cloud and data engineering certifications are preferred.

Knowledge, Skills, and Abilities • Deep expertise in Azure Databricks, Azure Data Factory, Azure Data Lake Storage, Azure Synapse, and Azure cloud infrastructure. • Advanced knowledge of Delta Lake, Lakehouse architecture, and medallion data processing frameworks. • Strong proficiency in Python, PySpark, SQL, Scala, and PowerShell. • Expertise with Apache Spark runtime optimization and distributed data processing frameworks. • Experience integrating real-time and batch data processing systems using Kafka, Spark Streaming, and related technologies. • Knowledge of relational and NoSQL database technologies and data architecture principles. • Familiarity with file formats such as Parquet, ORC, and Avro, including storage optimization techniques. • Understanding of CI/CD processes, DevOps methodologies, and orchestration tools such as Airflow and Azure DevOps. • Strong understanding of data governance, security, lineage, RBAC, compliance, and enterprise data management best practices. • Excellent analytical, troubleshooting, and problem-solving abilities. • Strong leadership, mentoring, organizational, and project coordination skills. • Excellent written and verbal communication skills with the ability to communicate technical concepts to diverse audiences. • Ability to manage multiple priorities in fast-paced enterprise environments.

Additional Desired Characteristics • Microsoft Azure Data Engineer, Azure Solutions Architect, or Databricks certifications preferred. • Experience supporting enterprise analytics, reporting, and machine learning initiatives. • Familiarity with MLflow, MLOps practices, and AI/analytics platforms. • Experience with large-scale enterprise data modernization initiatives. • Exposure to legal, healthcare, financial services, or other highly regulated industries preferred. • Strong understanding of enterprise cloud security, networking, and identity management concepts. • Experience implementing enterprise-wide data governance and data quality initiatives.

Work Environment • Hybrid or remote work environment with collaboration across distributed technical teams. • Primarily standard business hours with flexibility to support deployments, maintenance windows, or production incidents as needed. • Fast-paced enterprise technology environment focused on innovation, scalability, and operational excellence. • Significant collaboration with business stakeholders, engineering teams, analytics professionals, and executive leadership. • May require occasional travel for team meetings, planning sessions, or enterprise initiatives.

Other Duties

This job description is not intended to be an exhaustive list of all duties, responsibilities, or qualifications associated with the position. Additional responsibilities and assignments may be required based on organizational priorities and business needs.