AI & Big Data Engineer

Job Title: AI & Big Data Engineering Specialist

Location: India

Timings : 6:30 PM to 3:30 PM IST

Department: Analytics Engineering

Duration: Until EOY (project is until 2030).

Role Overview

We are seeking a highly skilled AI & Big Data Engineering Specialist to design and implement scalable data pipelines and advanced analytics solutions leveraging machine learning and agent-based automation . This role will focus on building intelligent systems that integrate with modern big data platforms such as Databricks , Snowflake , and AWS , enabling predictive insights and automation within our enterprise data platform.

Key Responsibilities • Design and develop large-scale data pipelines using Spark and Databricks or Snowflake for ingestion, transformation, and integration of structured and unstructured data. • Build and deploy machine learning models for anomaly detection, predictive analytics, and advanced insights using Databricks MLlib, scikit-learn, or Snowpark ML. • Develop AI agents leveraging Databricks Agent Framework for workflow automation and intelligent decision-making. • Implement feature engineering and model lifecycle management using MLflow and Databricks Feature Store. • Optimize data workflows for performance, scalability, and cost efficiency across AWS, Databricks, and Snowflake environments. • Collaborate with analytics and business teams to translate requirements into technical solutions. • Stay current with emerging AI/ML technologies and integrate them into our data platform solutions.

Required Skills & Qualifications • Strong programming skills in Python and SQL . • Hands-on experience with Databricks ML features (Delta Lake, MLflow, AutoML, Feature Store). • Proficiency in Spark for big data processing. • Experience with machine learning techniques (classification, regression, anomaly detection). • Familiarity with agent-based automation frameworks (Databricks Agent Bricks or similar). • Expertise in AWS services (S3, Glue, Lambda, IAM) and Snowflake for data integration. • Solid understanding of data architecture and distributed systems.

Preferred Qualifications • Experience in pharma or healthcare data analytics and engineering . • Knowledge of Generative AI and LLM integration for data insights. • Familiarity with Snowpark ML for ML in Snowflake. • Exposure to real-time data processing and streaming analytics.

Soft Skills • Excellent problem-solving and analytical thinking. • Ability to translate business requirements into technical solutions . • Strong communication and collaboration skills.