Skip to content

docs: audit and fix dt-eval-lib, dt-eval-cli, and root README#146

Merged
yassine-dt merged 13 commits into
mainfrom
fix/AI-301-fix-dt-evals-Readme-docs
Jul 24, 2026
Merged

docs: audit and fix dt-eval-lib, dt-eval-cli, and root README#146
yassine-dt merged 13 commits into
mainfrom
fix/AI-301-fix-dt-evals-Readme-docs

Conversation

@yassine-dt

@yassine-dt yassine-dt commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Summary

Systematic audit and fix of dt-eval-lib, dt-eval-cli, and root README against source code as ground truth. Every change is backed by a code reference.

dt-eval-lib

  • Rebuild metrics table from `catalog-data.ts`: add missing `fluency` (14th metric), fix all scoring types (`binary` → `continuous [0.0–1.0]`), fix required fields for 6 metrics, fix count 13→14
  • Clarify threshold semantics: pass when `value >= 0.5` (default), not binary extremes
  • Add `azure-openai` and `bedrock` providers with correct env vars and setup docs
  • Fix Vertex AI setup: document ADC-by-default, remove wrong `GOOGLE_API_KEY` instruction
  • Document `EvalInput` type in Configuration section, fix provider list, remove stale model defaults
  • Add peer dependency install table

dt-eval-cli

  • Remove fictional `dt-evals deploy` CLI command
  • Fix metric count 13→14, fix provider claim to list all 6 providers
  • Remove credentials from YAML examples
  • Rebuild Built-in Evaluators table from `catalog-data.ts`
  • Replace TypeScript Library section (~100 lines) with 2 sentences + link
  • Fix Data Expectations to match actual `dql.ts` defaults
  • Fix `defaults.ts`: default sampling percent `100` → `5`
  • Cut redundant inline comments

Root README

  • Remove fictional `dt-evals deploy` section
  • Fix all default models from `defaults.ts`, fix evaluator count, add missing `--store-evaluated-prompt` flag
  • Remove credentials from YAML examples

🤖 Generated with Claude Code

@yassine-dt yassine-dt self-assigned this Jul 23, 2026
@yassine-dt
yassine-dt marked this pull request as ready for review July 23, 2026 09:23
@yassine-dt
yassine-dt requested review from a team as code owners July 23, 2026 09:23
krismuhi
krismuhi previously requested changes Jul 23, 2026
View reviewed changes
Comment thread dt-eval-cli/README.md Outdated
Comment thread dt-eval-cli/README.md Outdated
Comment thread dt-eval-cli/README.md Outdated
Comment thread dt-eval-cli/README.md
Comment thread dt-eval-cli/README.md Outdated
Comment thread dt-eval-lib/src/engine/types.ts
@yassine-dt
yassine-dt requested a review from krismuhi July 23, 2026 12:49
yassine-dt and others added 12 commits July 23, 2026 15:38
@yassine-dt @claude
docs(dt-eval-lib): fix metrics table — chunk A
1439811 
- Rebuild Available Metrics table from catalog-data.ts as source of truth
- Add missing fluency metric (14th built-in metric, fixes count 13→14)
- Correct all scoring types: binary→continuous [0.0–1.0] for all metrics
- Fix required fields: toxicity (output only), context-relevance (input only),
  prompt-injection (input only), factual-accuracy/faithfulness/hallucination
  (remove context/expectedOutput that catalog does not use)
- Clarify threshold semantics: pass when value >= 0.5 (default), not binary extremes
- Add source-of-truth link to catalog-data.ts
- Add note that EvalInput.output is always required at type level

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@yassine-dt @claude
docs(dt-eval-lib): add Azure OpenAI and Bedrock providers, fix env va…
b6e606d 
…rs — chunk B

- Add azure-openai and bedrock to provider list and setup sections
- Fix env vars: remove incorrect GOOGLE_API_KEY-for-vertex instruction,
  add GOOGLE_CLOUD_PROJECT/LOCATION for ADC, add Azure and Bedrock vars
- Vertex AI: document both auth paths (ADC default, explicit apiKey)
  and clarify GOOGLE_API_KEY is intentionally ignored for vertex
- Azure OpenAI: document all required fields (apiKey, baseUrl, apiVersion, model)
  and note there is no default model (deployment names are user-defined)
- Bedrock: document AWS SDK credential chain, static creds are optional
- Consolidate duplicate .env block into single env vars section

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@yassine-dt @claude
docs(dt-eval-lib): trim provider setup sections — chunk B cleanup
30f8556 
- Drop Gemini setup section (GOOGLE_API_KEY already in env vars block)
- Collapse Vertex to one sentence (ADC-by-default is the key insight)
- Collapse Bedrock to one sentence (credential chain is the key insight)
- Keep Azure OpenAI example (4-required-fields + deployment-name gotcha)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@yassine-dt @claude
docs(dt-eval-lib): document EvalInput, fix Configuration section — ch…
9d11d0d 
…unk C

- Add EvalInput example to Configuration section (documents all 4 fields)
- Fix provider list in config comment to include all 6 providers
- Remove hardcoded stale model defaults, point to providers/index.ts instead
- Fold Threshold Override standalone section into config block inline comment
- Fix stale JSDoc in types.ts with same pointer to DEFAULT_MODELS

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@yassine-dt @claude
docs(dt-eval-lib): add peer dependency install guidance — chunk E
47b631b 
Provider SDKs are optional peer deps (not auto-installed, no npm warning).
Add install table so users know which package to add per provider before
hitting a runtime module-not-found error.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@yassine-dt @claude
docs(root): fix README — remove fictional deploy command, fix models/…
9206c0e 
…count/flags

- Delete fictional `dt-evals deploy` CLI section (command does not exist)
- Fix all default models in Supported Providers table from defaults.ts
- Fix evaluator count 13→14
- Add missing --store-evaluated-prompt flag to run flags table
- Remove credentials from YAML examples (apiToken, Bedrock apiKey/secretKey)
  replaced with env var comments

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@yassine-dt @claude
docs(dt-eval-cli): fix README factual errors and trim verbosity
7a17233 
- Remove fictional deploy command from intro code block and Why Teams Use It
- Fix metric count 13→14 (two places)
- Fix provider claim: OpenAI/Anthropic only → all 6 providers (two places)
- Remove apiToken from YAML examples, replace with env var comments
- Rebuild Built-in Evaluators table from catalog-data.ts: 14 rows,
  alphabetical, corrected required fields, added fluency
- Replace TypeScript Library section (~100 lines) with 2 sentences + link
- Fix Data Expectations: use actual default fields from dql.ts
  (gen_ai.input.messages primary, OpenLLMetry as fallback)
- Remove fictional span.input/output/context fallback claim
- Use consistent span.* naming in spanFields example
- fix(dt-eval-cli): change default sampling percent from 100 to 5

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@yassine-dt @claude
docs(dt-eval-cli): cut redundant comments and fix span field examples
f4355d2 
- Remove obvious/redundant inline comments throughout YAML examples
- Use custom.* naming in spanFields example for clarity
- Fix per-metric input routing example comments
- Remove storeEvaluatedPrompt comment block (covered by flags table)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@yassine-dt @claude
fix: markdownlint double blank line and update sampling default test
945768e 
- Remove extra blank line in root README (MD012)
- Update config.test.ts to expect default sampling percent 5 (was 100)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@yassine-dt
docs: restore deploy docs and unify Google provider naming
a76a311 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@yassine-dt
docs: clarify OpenInference span mapping
8beb5f2 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@yassine-dt
docs: refine span mapping examples
d0f7610 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@yassine-dt
yassine-dt force-pushed the fix/AI-301-fix-dt-evals-Readme-docs branch from 4ad041c to d0f7610 Compare July 23, 2026 13:39
thisthat
thisthat reviewed Jul 24, 2026
View reviewed changes
Comment thread dt-eval-cli/README.md Outdated
@yassine-dt @thisthat
Update dt-eval-cli/README.md
bae4829 
Co-authored-by: Giovanni Liva <giovanni.liva@dynatrace.com>
@thisthat
thisthat dismissed krismuhi’s stale review July 24, 2026 07:24

changed applied

@yassine-dt
yassine-dt merged commit d78c66c into main Jul 24, 2026
10 checks passed
@yassine-dt
yassine-dt deleted the fix/AI-301-fix-dt-evals-Readme-docs branch July 24, 2026 07:25
This was referenced Jul 24, 2026
Merged
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@thisthat thisthat thisthat approved these changes
@krismuhi krismuhi Awaiting requested review from krismuhi

Assignees

@yassine-dt yassine-dt

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@yassine-dt @thisthat @krismuhi