KairaxisKairaxis
Freestructured workflowmedium risk

Capability · rank 13

rag-evaluation-checklist

Evaluate retrieval quality, answer grounding, and regression risk with explicit checks.

Structured RAG evaluation checklist covering retrieval hit quality, grounded answers, failure modes, and measurable next tests.

Listed owner

skillforge/ai-engineering

Package source

Kairaxis manifest

Category

AI Engineering

Package validatedBenchmark testedCreator reviewedAI Engineering
$npx skills add skillforge/ai-engineering --skill rag-evaluation-checklist
OverviewSandboxUsageExamplesLimitationsVersionsRelated Skills

Overview

Structured RAG evaluation checklist covering retrieval hit quality, grounded answers, failure modes, and measurable next tests. Review the sandbox comparison, limitations, permissions, and pinned version before install.

Outcome

Evaluate retrieval quality, answer grounding, and regression risk with explicit checks.

Who it is for

AI engineers shipping retrieval-augmented products.

What it is not for

Training foundation models or legal advice about model outputs.

Runtime export

codex_skill · markdown_export

Sandbox comparison
Generic baseline vs rag-evaluation-checklist on the same sample input · pinned v1.0.0

Sample input

Support bot cites outdated pricing docs 1 in 5 times.

Model: gpt-4.1-mini · temp 0.2

Generic baseline

43/100
Add more documents and improve the prompt to only use retrieved context.

Skill-enhanced

91/100
Checklist findings:
- Retrieval: stale doc versions ranked high
- Grounding: answers mix memory + retrieved snippets
- Metrics: no faithfulness or citation precision gate
Next tests: version-aware corpus, top-k ablation, faithfulness threshold before deploy.

Rubric scores

+48 lift
Failure mode coverage2 → 5
Measurable metrics1 → 5
Actionable next tests2 → 5
Grounding discipline2 → 4

Assumptions

  • Sample input is representative of a real production task
  • Same model and temperature for baseline and skill-enhanced runs

Comparison runs through the shared sandbox engine on registry sample cases (deterministic v0 path).

Usage

  1. 1. Inspect trust badges, limitations, and permissions on this page.
  2. 2. Run the sandbox comparison to see baseline vs skill-enhanced output.
  3. 3. Install the pinned version with the command below for your agent runtime.
  4. 4. Agents can also fetch /api/v1/skills/rag-evaluation-checklist/manifest.
$npx skills add skillforge/ai-engineering --skill rag-evaluation-checklist

Example prompts & use cases

1 / Prompt

Evaluate this RAG answer for grounding failures

2 / Prompt

Build a regression suite for our support retrieval pipeline

Limitations

  • Checklist quality depends on available traces and labeled cases.
  • Does not execute retrieval against private indexes by itself.

Audience

AI engineers shipping retrieval-augmented products.

Not for

Training foundation models or legal advice about model outputs.

Version history

v1.0.0

Jul 2, 2026current

Launch checklist + sample evals