Data Engineer - Experimentation

Must Have Technical/Functional Skills

• Core Data Engineering Competencies
• Data Concepts & Data Modelling
• Digital : Big Data Platforms
• data pipelines design
• Microsoft fabric data agents
• Azure AI Services
• AI and ML Integration
• Analytical and problem solving skills
• Performance tuning and monitoring
• Digital : PySpark
• ecommerce domain knowledge
• Digital : Adobe Analytics
• Digital : Customer Analytics
• Adobe customer journey analytics
• clickstream data

Roles & Responsibilities

• Experimentation data enablement (Silver layer ownership)

  • Own the design, build, and maintenance of curated Silver-layer datasets in Microsoft Fabric to support experimentation reporting and analysis.
  • Partner with the Data Reporting/BI team to identify required dimensions, metrics, and joins (visitor/session, variant, campaign/flight, geo, device, channel, funnel steps, conversion events) and ensure these are available in Silver.
  • Translate experimentation team needs into standardized, reusable data products (tables/views) that can be consumed consistently for scorecards, dashboards, and ad hoc analysis.
  • Ensure Silver-layer outputs are analysis-ready (cleaned, conformed, deduplicated, and aligned to agreed definitions).

• Data gap analysis and assessment

  • Conduct Regular Gap Assessments Between experimentation requirements (scorecards/KPIs), existing Silver layer availability, and upstream telemetry/source systems.
  • Identify missing/incorrect fields, inconsistent definitions, data latency issues, or join-key problems; document business impact, severity/priority, remediation approach, timelines and dependencies.
  • Provide recommendations on data model improvements (facts/dimensions, grain, surrogate keys, conformance rules) to reduce recurring data quality issues.

• Gold layer requirements and stakeholder requirement gathering

  • Lead requirement workshops with stakeholders (experimentation, measurement, BI/reporting, engineering) to define Gold layer outputs: KPI definitions and calculation logic, experiment attribution rules, scorecard structure, segmentation needs and slicing dimensions, governance and refresh SLAs.
  • Produce clear functional + technical specifications: source-to-target mappings, data dictionary, metric definitions, validation rules, and acceptance criteria.
  • Drive alignment on single source of truth definitions to avoid mismatch across CJA/Power BI/scorecards.

• Data pipeline engineering (1DS + Fabric pipelines / ADF)

  • Build and operate robust pipelines using Microsoft Fabric Pipelines and/or ADF to ingest and transform data into Silver and Gold layers.
  • Understand and work with 1DS (telemetry) pipelines (or equivalent) to ensure required events and attributes flow correctly into Fabric.
  • Implement reliable orchestration, incremental loads, error handling, and monitoring to meet experimentation reporting timelines.

• Data validation and reconciliation (CJA included)

  • Perform data validation and reconciliation between Silver/Gold datasets and Customer Journey Analytics (CJA): event counts, session/user logic, conversions, experiment/variant attribution consistency, time window alignment and filtering rules.
  • Create Validation Checks And Automated Routines For missing data detection, duplicate events, schema drift, metric anomalies (sudden drops/spikes), SRM-supporting signals (where applicable from data).
  • Document issues and coordinate fixes with upstream owners (telemetry, tagging, product engineering, reporting teams).

• Experimentation lifecycle and scorecard readiness

  • Support the experimentation lifecycle by ensuring datasets are ready for: pre-launch readiness checks, launch measurement, scorecard generation, ongoing health checks, post-test learnings/archives.
  • Enable Consistent Scorecard Outputs By Curating experiment metadata (test IDs, start/end dates, allocations), KPI metrics (primary/secondary), and slicing dimensions required by experimentation stakeholders.

• AI agent design & build for experimentation team

  • Design and build AI-powered agents (Fabric Data Agents / Copilot / Azure OpenAI) to accelerate experimentation workflows, such as: automated scorecard creation and narrative summaries, self-serve Q&A over experimentation datasets, anomaly explanations and investigation guidance, metric definition assistant / data dictionary lookup, pipeline health and data quality assistant.
  • Define The Agent’s scope, personas, and usage scenarios, grounding data sources (Silver/Gold tables, metadata, documentation), security model (RBAC, data access boundaries), evaluation metrics (accuracy, timeliness, adoption).
  • Partner with experimentation and reporting teams to iterate through pilot → feedback → rollout.

• Documentation, governance, and operational excellence

  • Maintain Documentation For dataset definitions (Silver/Gold), transformation logic, metric calculation rules, pipeline design and dependencies, validation checklists and runbooks.
  • Establish Best Practices For naming conventions, semantic consistency, versioning and backward compatibility, cost/performance optimization in Fabric.
  • Provide operational support: monitoring, troubleshooting, incident triage, and continuous improvement.

TCS Employee Benefits Summary

• Discretionary Annual Incentive.
• Comprehensive Medical Coverage: Medical & Health, Dental & Vision, Disability Planning & Insurance, Pet Insurance Plans.
• Family Support: Maternal & Parental Leaves.
• Insurance Options: Auto & Home Insurance, Identity Theft Protection.
• Convenience & Professional Growth: Commuter Benefits & Certification & Training Reimbursement.
• Time Off: Vacation, Time Off, Sick Leave & Holidays.
• Legal & Financial Assistance: Legal Assistance, 401K Plan, Performance Bonus, College Fund, Student Loan Refinancing.
• Salary Range: $64,000 - $100,000 a year