Sr. Data Engineer – Databricks / Azure

Sr. Data Engineer – Databricks / Azure | Remote | Contract | 2–11 PM IST | Till end of 2026 (possibility of extension)

Design, build, and maintain automated testing frameworks and Python SDKs for data integration, parity verification, and user accessibility in the Industrial IoT / Smart Buildings domain. We're looking for a Senior Data Engineer with strong Python, PySpark, and Spark skills, with proven experience in automated Data Quality frameworks and CI/CD pipelines, delivering Data & Analytics products using Agile methodology.

Must-Have Skills:

• 5+ years hands-on data engineering experience in Azure ecosystem • PySpark & DataFrames • Apache Spark internals (Catalyst Optimizer, Logical Plans) • Java • Python (building Libraries/SDKs, Py4J) • Databricks & Delta Lake optimization • Automated Testing frameworks (PyTest, JUnit) for data pipelines • Data Quality tools (Great Expectations, Deequ) • Azure DevOps, GitHub, CI/CD, artifact management • Data modeling & warehousing on Azure Databricks • Data governance & security best practices in Azure Preferred Certifications: Azure Fundamentals, Azure Data Engineer Associate, Databricks Certified Data Engineer Professional Responsibilities:

• Implement scalable data solutions on Spark and Flink • Develop Python SDK wrapping core libraries for use in Jupyter Notebooks • Build an automated Parity Test Suite via CI/CD against Golden Datasets (Spark & Flink) • Implement Data Quality rules (Completeness, Validity thresholds) • Manage and optimize Azure & Databricks resources for performance and cost • Transform, clean, and prepare data using SQL, Python, and Java • Monitor and tune workloads/pipelines • Maintain CI/CD pipelines and documentation • Participate in Agile ceremonies and enforce DE best practices