Databricks Engineer- GCP Cloud

Requirements

Wissen Technology is Hiring for Databricks Engineer- GCP Cloud

About Wissen Technology:

At Wissen Technology, we deliver niche, custom-built products that solve complex business challenges across industries worldwide. Founded in 2015, our core philosophy is built around a strong product engineering mindset—ensuring every solution is architected and delivered right the first time. Today, Wissen Technology has a global footprint with 2000+ employees across offices in the US, UK, UAE, India, and Australia. Our commitment to excellence translates into delivering 2X impact compared to traditional service providers. How do we achieve this? Through a combination of deep domain knowledge, cutting-edge technology expertise, and a relentless focus on quality. We don’t just meet expectations—we exceed them by ensuring faster time-to-market, reduced rework, and greater alignment with client objectives. We have a proven track record of building mission-critical systems across industries, including financial services, healthcare, retail, manufacturing, and more. Wissen stands apart through its unique delivery models. Our outcome-based projects ensure predictable costs and timelines, while our agile pods provide clients with the flexibility to adapt to their evolving business needs. Wissen leverages its thought leadership and technology prowess to drive superior business outcomes. Our success is powered by top-tier talent. Our mission is clear: to be the partner of choice for building world-class custom products that deliver exceptional impact—the first time, every time.

Job Summary: We are looking for a bright and dynamic engineer, motivated and able to work independently as well as in partnership with IT and Business teams spread across the globe. The candidate needs to be an exceptionally strong Python and SQL programmer with hands-on experience in GCP-native data technologies including BigQuery, Dataproc, Cloud Composer, and Datastream.

Besides technical skills, we are looking for a candidate with a strong sense of ownership and the ability to work in a diverse, cross-functional team spanning Engineering, Research, DataOps, and Compliance.

Experience: 6- 8 years Location: Mumbai/Pune/Bangalore Mode of Work: Full time

Key Responsibilities:

• Build and maintain scalable, distributed, fault-tolerant data pipelines on GCP, including BigQuery-based lakehouse layers and Dataproc-driven Delta Lake workflows

• Actively participate in meetings with various stakeholders across data engineering, compliance, and business teams globally

• Understand market data processing and transformation needs; build pipelines to acquire, normalise, transform, and release large volumes of financial data through the OMDP data factory

• Design and implement bitemporal data models (valid-time + system-time) on BigQuery to support certified, regulatory-grade time-series datasets

• Build, use, and maintain software testing frameworks (unit / non-regression / user acceptance) for data pipelines and transformation logic

• Take complete ownership of solutions and assigned tasks, including ingestion pipelines, QA workflows, correction management, and audit trail implementation.

• Work in a collaborative manner with other team members and contribute to shared platform services rather than vertical-specific implementations

• Have business acumen to understand financial concepts around reference data related to equities and other asset classes

• Support teams across data and technology in implementing AI solutions and integrating their services with MSCI's data science products and platforms, including AI-assisted ingestion, anomaly detection, and semantic search over the lakehouse using Vertex AI

Requirements:

• 6-8 years of experience in data engineering

• Proficient in Python programming — data pipeline development, transformation logic, and automation scripts

• Proficient in data query and analysis using SQL, with strong hands-on experience in BigQuery — partitioning, clustering, materialised views, and time-series query patterns at scale

• Hands-on experience building and scheduling pipelines using Cloud Composer (Apache Airflow) — DAG authoring, SLA alerting, retry logic, and dependency management

• Working knowledge of Dataproc (Apache Spark) — batch ingestion, Delta Lake merge operations, and incremental data processing

• Proficient in AI-assisted development tools such as GitHub Copilot, Cursor, or others for accelerating code generation and enhancing developer productivity

• Code versioning and collaboration using Git — branching strategies, pull request workflows, and pipeline-as-code practices

• Familiarity with REST APIs — consuming external data vendor APIs and building service-layer integrations

• Familiarity with GCP cloud technologies — Cloud Storage, Pub/Sub, Datastream, Cloud Monitoring, IAM, and VPC Service Controls

Good To Have Skills:

• Basic knowledge of data manipulation and analysis libraries — pandas, PySpark, or equivalent

• Basic knowledge of columnar storage, SQL-based querying, and time-series analytics (ClickHouse or equivalent)

• Familiarity with Dataplex for data discovery, lineage, policy tagging, and data quality rule management

• Understanding of Change Data Capture (CDC) patterns using Datastream for replicating transactional data into BigQuery

• Understanding of bitemporal data modeling concepts (valid-time and system-time) and the challenges of implementing them within BigQuery's append-optimised design

• Understanding of financial reference data — equities, fixed income identifiers, corporate actions, or index composition data

• Familiarity with BigQuery cost management — slot reservations, query cost controls, and workload isolation using reservations and assignments

• Exposure to CI/CD pipelines and infrastructure-as-code using Terraform for data platform deployments on GCP

• Prior experience or projects involving LLMs and Agentic AI — particularly using Vertex AI for AI-assisted data quality, anomaly detection, semantic search, or natural language querying over structured datasets — is a strong plus

Wissen Sites:

Website: www.wissen.com

LinkedIn: