Lead Data Scientist for LLMs

About

Lead Data Scientist for LLMs at Dynatrace, driving the end-to-end LLM stack from data and ingestion to prompt design, tooling, safety, and optimization. You’ll own architecture, evaluation, and LLMOps governance to enable enterprise-grade AI capabilities. This role combines hands-on DS work with technical leadership to scale generative AI across complex, multicloud environments. You’ll mentor teams, shape RAG standards, and partner with cross-functional peers to deliver impactful AI solutions.

Responsibilities

• Own the LLM system architecture including retrieval pipelines, prompt and tool design, routing, safety layers, and telemetry with a focus on quality, latency, and cost.
• Define and standardize RAG content ingestion, chunking, hybrid retrieval, reranking, query understanding, and output contracts.
• Develop and maintain evaluation strategies spanning accuracy, grounding, safety, privacy, determinism, latency, and cost.
• Formalize LLMOps practices: versioning for prompts/datasets/models, experiment governance, registries, and promotion criteria (dev→staging→prod).
• Lead tool/agent design: API schema for function calls, error handling, recovery, self-correction, and guardrails.
• Assess build-vs-buy options (managed vs open-source/self-hosted) considering performance, IP, privacy, and compliance.
• Provide mentorship on prompting, retrieval design, evaluations, and safe deployment; lead reviews of prompts, pipelines, and evaluation reports.
• Hands-on end-to-end RAG implementation: ingestion, chunking, embeddings, hybrid search, rerank, prompt assembly, tool calls, post-processing.
• Engineer robust prompts/tools with templates, multi-turn strategies, and structured outputs (JSON Schema/Pydantic).
• Select and tune models and embeddings; apply LoRA/PEFT or distillation when warranted.
• Create eval corpora and KPIs for accuracy, grounding, and tool success; implement guardrails for PII/PHI and safety scoring.
• Ship resilient services with analytics, alerts, SLOs, and drift/cost monitoring; optimize for scale via caching, batching, and routing by intent.

Requirements

• Advanced CS/AI/ML degree or equivalent
• 7+ years DS/ML experience
• 3+ years NLP/LLMs
• Shipped production systems
• 5+ years professional Python
• 3+ years data engineering for unstructured data (text processing, embedding-friendly preprocessing)
• 1+ years proven RAG expertise (embeddings, retrieval, reranking, chunking)
• 1+ years evaluation depth (offline/online for accuracy, grounding, safety)
• 1+ years safety/privacy (PII/PHI redaction, policy enforcement)
• 1+ years LLMOps (prompt/version management, experiment tracking, monitoring)
• Excellent communication and ability to explain trade-offs

Benefits

• remote options
• hybrid work
• international team
• career development program
• visa and relocation support
• flexible work arrangements