Monitor and optimize LLM applications effectively.
LLM Observability Engineer
OpenTelemetry, LangChain, LlamaIndex
Best for
- ▸Building OpenTelemetry instrumentation for LangChain/LlamaIndex applications with custom spans and token tracking
- ▸Setting up cost attribution dashboards that track spend per feature, user, and model across OpenAI/Anthropic APIs
- ▸Implementing automated quality monitoring with online evaluation pipelines for output drift detection
- ▸Creating latency SLAs and alerting for multi-step RAG pipelines with p95/p99 performance tracking
What you'll get
- ▸Step-by-step OpenTelemetry instrumentation code with GenAI semantic conventions for capturing model, tokens, and latency
- ▸Cost attribution dashboard configuration with per-feature token spend calculations and budget alerting thresholds
- ▸Quality monitoring pipeline architecture with online evaluation classifiers and drift detection algorithms
Details about your LLM tech stack (models, frameworks, deployment), current monitoring tools, and specific observability gaps you need to address.
Implementation guides with code examples for instrumentation, dashboard configurations, alerting rules, and cost tracking setups tailored to your stack.
What's inside
“You are an LLM Observability Engineer. You design production-grade monitoring systems for LLM applications, specializing in cost tracking, quality evaluation, distributed tracing, and compliance. - **Cost as a First-Class Signal**: Treat token consumption and per-request costs ($0.001-$1+) as primar...”
Covers
Not designed for ↓
- ×General application performance monitoring without LLM-specific metrics
- ×Building machine learning training pipelines or model development infrastructure
- ×Basic logging setup or traditional web application monitoring
- ×LLM model fine-tuning or inference optimization
SupaScore
88.65▼
Evidence Policy
Standard: no explicit evidence policy.
Research Foundation: 7 sources (3 official docs, 2 books, 1 web, 1 industry frameworks)
This skill was developed through independent research and synthesis. SupaSkills is not affiliated with or endorsed by any cited author or organisation.
Version History
v5.5 final distill
Pipeline v4: rebuilt with 3 helper skills
Initial release
Works well with
Need more depth?
Specialist skills that go deeper in areas this skill touches.
Common Workflows
LLM Production Readiness Pipeline
Complete production setup: instrumentation → quality evaluation → safety controls
© 2026 Kill The Dragon GmbH. This skill and its system prompt are protected by copyright. Unauthorised redistribution is prohibited. Terms of Service · Legal Notice