A comprehensive Dynatrace App for monitoring, analyzing, and optimizing Generative AI workloads. Built with React, TypeScript, and the Dynatrace SDK, this app provides real-time visibility into GenAI usage, costs, and performance using Grail data.
- Real-time GenAI metrics: Total requests, tokens, cost estimates, and unique models
- Top models by usage: Quick overview of most-used AI models
- Navigation cards: Quick access to all analytics pages
- Historical cost trends: 30-day visualization of token usage and estimated costs
- Weekly growth rate: Automatic calculation of usage growth trends
- Budget alerts: Set budget limits and get warnings when projected costs exceed thresholds
- Cost by model breakdown: See which models are driving costs
- Optimization suggestions: AI-powered recommendations for cost reduction
- Side-by-side model analysis: Compare costs, performance, and usage across different AI models
- Cost per 1K tokens: Standardized pricing comparison
- Performance metrics: Latency, error rates, and token efficiency
- Migration recommendations: Identify opportunities to switch to cheaper alternatives
- Security Detection: AI-powered analysis for PII, prompt injection, and sensitive data
- 🔐 PII Detection: SSN, emails, phone numbers, credit cards, DOB, medical records (HIPAA/PHI)
⚠️ Injection Attempts: Jailbreak patterns, instruction override, role-playing attacks- 🔒 Sensitive Data: Passwords, API keys, tokens, internal company data
- ⚖️ Bias Risk: Protected characteristics in HR/hiring/decision contexts
- 🎭 Hallucination Risk: Real-time data queries without grounding or factual verification
- 💰 Cost Analysis: High token/cost pattern identification
- 🔄 Cache Opportunities: Repetitive patterns eligible for semantic caching (15+ occurrences)
- Compliance & Governance: Comprehensive security scoring and risk assessment
- Actionable Insights: AI-generated recommendations for remediation
- Filter & Search: Quickly find problematic patterns by issue type
- Token efficiency analysis: Input/output ratio and average token usage
- Optimization recommendations: Suggestions for prompt engineering improvements
- Tool usage visualization: Heatmap of AI agent tool calls
- Agent flow analysis: Understand which tools are called together
- Loop detection: Identify potential infinite loops in agent workflows
- Error rate tracking: Monitor tool reliability
- Policy simulation: Test guardrail policies against historical data
- Violation detection: Identify requests that would be blocked
- Impact analysis: Understand the effect of policies before deployment
- Custom threshold testing: Experiment with different limits
- Cost optimization scenarios: Simulate routing strategies across models
- Performance trade-off analysis: Balance cost vs quality
- Savings projections: Estimate potential cost reductions
- Model routing recommendations: Optimize model selection
genai-ops-manager/
├── ui/
│ ├── main.tsx # App entry point
│ └── app/
│ ├── App.tsx # Main app with routing
│ ├── components/
│ │ ├── Card.tsx # Reusable card component
│ │ └── Header.tsx # App header
│ └── pages/
│ ├── Home.tsx # Dashboard
│ ├── CostForecast.tsx # FinOps analytics
│ ├── ModelCostComparison.tsx # Model comparison
│ ├── PromptAnalyzer.tsx # Prompt analysis
│ ├── AgentToolHeatmap.tsx # Agent tools
│ ├── GuardrailBacktester.tsx # Policy testing
│ └── ModelArbitrageSimulator.tsx # Cost optimization
├── app.config.json # Dynatrace app configuration
├── package.json # Dependencies and scripts
└── vitest.config.ts # Test configuration
- Node.js 16.13.0 or higher (Node.js 22 recommended)
- Dynatrace environment with GenAI observability data
- Required OAuth scopes:
storage:spans:read- For GenAI trace datastorage:logs:read- For log analysisstorage:buckets:read- For Grail bucket accessstorage:events:read- For event datastorage:metrics:read- For metric data
# Clone the repository
git clone https://github.com/pushpendrasinghbaghel-ai/genai-ops-manager.git
cd genai-ops-manager
# Install dependencies
npm install# Start development server
npm run start
# Or specify environment URL
npx dt-app dev --environment-url https://your-tenant.apps.dynatrace.com --open# Build for production
npm run build
# Deploy to Dynatrace
npm run deploy# Run tests
npm run test
# Run tests once
npm run test:run
# Run with coverage
npm run test:coverageThe app uses Dynatrace Query Language (DQL) to fetch GenAI observability data from Grail. Key query patterns:
fetch spans, from: now()-24h
| filter isNotNull(gen_ai.provider.name) OR isNotNull(gen_ai.request.model)
| summarize
total_input = sum(coalesce(gen_ai.usage.input_tokens, gen_ai.usage.prompt_tokens, 0)),
total_output = sum(coalesce(gen_ai.usage.output_tokens, gen_ai.usage.completion_tokens, 0))
fetch spans, from: now()-24h
| filter traceloop.span.kind == "tool"
| summarize call_count = count(), by: { tool_name = span.name }
fetch spans, from: now()-24h
| filter isNotNull(gen_ai.provider.name) OR isNotNull(gen_ai.request.model)
| fieldsAdd prompt = coalesce(gen_ai.prompt.1.content, gen_ai.prompt.0.content)
| filter isNotNull(prompt)
| fieldsAdd prompt_preview = substring(prompt, from:0, to:200)
| summarize
count = count(),
avg_input_tokens = avg(coalesce(gen_ai.usage.input_tokens, 0)),
avg_output_tokens = avg(coalesce(gen_ai.usage.output_tokens, 0)),
by: { prompt_preview, model = gen_ai.request.model, provider = gen_ai.provider.name }
| sort count desc
The Prompt Analyzer includes comprehensive security detection patterns:
| Detection Type | Description | Severity Levels |
|---|---|---|
| 🔐 PII | SSN, email, phone, credit cards, DOB, medical records (HIPAA/PHI) | Critical, High, Medium |
| Jailbreak attempts, instruction override, role-playing attacks | Critical | |
| 🔒 Sensitive | Passwords, API keys, tokens, internal company data | High |
| ⚖️ Bias | Protected characteristics in HR/hiring/decision contexts | High |
| 🎭 Hallucination | Real-time data queries without grounding/verification | Medium, Low |
| 💰 Cost | High token/cost patterns requiring optimization | Critical to Low |
| 🔄 Repetitive | Patterns repeated 15+ times (cache candidates) | Low |
Example Use Cases:
- Compliance: Detect HIPAA/PHI violations before data reaches LLMs
- Security: Identify prompt injection and jailbreak attempts
- Cost Optimization: Find expensive patterns and cache opportunities
- Bias Prevention: Flag protected characteristics in decision-making prompts
- Quality: Detect hallucination risks from real-time data queries
The app includes comprehensive unit tests:
- DQL Query Validation: Tests for correct query syntax and patterns
- Cost Estimation: Tests for pricing calculations across models
- App Configuration: Tests for required OAuth scopes
npm run test:run
# Output:
# ✓ ui/app/tests/dql-queries.test.ts (15 tests)
# ✓ ui/app/tests/app-config.test.ts (9 tests)
# ✓ ui/app/tests/cost-estimation.test.ts (21 tests)
# Test Files: 3 passed (3)
# Tests: 45 passed (45)| Script | Description |
|---|---|
npm run start |
Start development server |
npm run build |
Build for production |
npm run deploy |
Deploy to Dynatrace |
npm run uninstall |
Uninstall from Dynatrace |
npm run test |
Run tests in watch mode |
npm run test:run |
Run tests once |
npm run test:coverage |
Run tests with coverage |
npm run lint |
Run ESLint |
{
"environmentUrl": "https://your-tenant.apps.dynatrace.com/",
"app": {
"name": "GenAI Ops Manager",
"version": "0.0.1",
"description": "Monitor and optimize GenAI workloads",
"id": "my.genai.ops.manager",
"scopes": [
{ "name": "storage:spans:read" },
{ "name": "storage:logs:read" },
{ "name": "storage:buckets:read" },
{ "name": "storage:events:read" },
{ "name": "storage:metrics:read" }
]
}
}- Fork the repository
- Create a feature branch (
git checkout -b feature/amazing-feature) - Commit your changes (
git commit -m 'Add amazing feature') - Push to the branch (
git push origin feature/amazing-feature) - Open a Pull Request
This project is licensed under the ISC License.
Built with ❤️ for the Dynatrace Platform