Provider-Side Grading Areas
Dieser Inhalt ist noch nicht in deiner Sprache verfügbar.
This chapter is the normative source for the provider-side grading Areas — the Areas that grade one schema inside one namespace without group context. Each Area is a self-contained grading rubric attached to the primitive it evaluates, written to a _gradings/ folder next to that primitive (see 19-folder-layout.md). The provider side produces the base unit of the FlowMCP corpus — one namespace with one or more schemas, namespace skills, and an About Resource — while higher-level grouping is defined separately in 05-phases-selection.md. A schema graded only on the provider side has gradingTier = autonomous, whose maximum attainable grade is B (06-determinism-and-tier.md); grade A requires a group-bound contribution from the selection side.
The Provider-Side Areas
Section titled “The Provider-Side Areas”The grading system defines eleven Areas in total (see 05-phases-selection.md and 19-folder-layout.md). Of these, the following six are provider-side (everything except the selection Areas):
| # | Area | Evaluates | _gradings/ location | Persona | Det/Non-Det |
|---|---|---|---|---|---|
| 1 | single-test | one tool | providers/<ns>/<schema>/tools/<tool>/_gradings/ | no | deterministic gate + non-deterministic |
| 2 | tools-aggregate-schema | the tools collection of one schema | providers/<ns>/<schema>/_gradings/ | no | both |
| 3 | tools-aggregate-namespace | tools across the namespace | providers/<ns>/_gradings/ | no | both |
| 4 | namespace-description | namespace metadata | providers/<ns>/_gradings/ | no | non-deterministic |
| 5 | namespace-skills | one namespace skill (per skill) | providers/<ns>/<schema>/skills/<skill>/_gradings/ | yes | non-deterministic |
| 6 | about-namespace | the About Resource (declared in one schema) | providers/<ns>/<schema>/resources/about/_gradings/ | yes | deterministic (route-exists) + non-deterministic |
The remaining five Areas (about-selection, selection-skills-L1, selection-skills-L2, selection-skills-L3, selection-aggregate) are selection-side and live in 05-phases-selection.md.
Each Area is graded independently. There is no fixed linear order between Areas; the only ordering obligations are the cascade and veto procedures (see Area Procedures) and the deterministic-first rule of 06-determinism-and-tier.md.
Area Procedures
Section titled “Area Procedures”The Area model retains four procedures that previously lived inside the phase model. They are now expressed as rules that apply across the provider-side Areas.
Description Cascade (within single-test and tools-aggregate-*)
Section titled “Description Cascade (within single-test and tools-aggregate-*)”The description cascade is a mandatory ordered procedure for validating tool descriptions. It MUST be executed in the following order; skipping or reordering steps is a finding.
- Run tests against the endpoint. MUST meet the working-test pass bar defined in
06-determinism-and-tier.md: 2 working tests per tool (status true and non-empty data) for a parameterised tool, 1 for a parameterless tool; 3 is the SHOULD target (thedata-analyzablerung) covering the breadth of the parameter space. A tool below its pass bar is a repairableblocked/not-green state recorded with a status reason — never a terminal rejection. ch06 is the single authoritative bar definition; this step references it and does not set a divergent threshold. - Check the responses and validate the tool description against the actual responses.
- Normalise / update the tool description to match the validated responses.
- All tools, resources, and prompts MUST have descriptions — and each description MUST be individually checked.
- Descriptions MUST be neutral — see Description Neutrality.
The cascade is a contract: outputs of step n are inputs of step n+1. A failure in any step halts the cascade for the affected tool and is recorded as a finding. single-test carries the per-tool cascade result; tools-aggregate-schema and tools-aggregate-namespace aggregate the per-tool cascade outcomes.
Description Neutrality (cross-cutting)
Section titled “Description Neutrality (cross-cutting)”The neutrality rule (cascade step 5) is normative and worth restating:
- A tool description states what the tool does (capabilities, parameters, return shape).
- A tool description MUST NOT state what for it should be used (application scenarios, persona use cases, “good for X”).
- Application scenarios and persona use cases belong in the About Resource (
11-about-convention.md) — not in the tool description.
This separation is essential for LLM-grader reproducibility: neutral descriptions can be deterministically compared to the observed API behaviour; mixed descriptions cannot.
Cascade Stop / Veto (cross-cutting)
Section titled “Cascade Stop / Veto (cross-cutting)”A failed gate MUST halt the dependent grading for the affected schema. A categorical veto raised in any Area MUST stop further grading for that schema. Examples:
api-key-domain-mismatchveto — when the API key declared in the schema metadata does not match the API root domain, the veto is raised and thesingle-testlive tests for the affected tools MUST NOT be treated as pass.- HTTP 4xx — when a tool returns HTTP 4xx (including 401/403), the response MUST NOT be treated as “auth-pass” (see
06-determinism-and-tier.md). The description cascade for that tool cannot be completed and is recorded as a finding. - Eligibility violation — when an endpoint fails an exclusion criterion under
02-eligibility.mdand the schema author insists on including it, the schema is rejected and dependent Areas do not run for it.
Cascade-stop events are recorded in the grading entry. They do not lower the grade silently — a categorical veto replaces the aggregate grade with REJECTED, which the index derivation maps to the terminal status rejected (see 06-determinism-and-tier.md and 19-folder-layout.md).
Base Unit (cross-cutting)
Section titled “Base Unit (cross-cutting)”When the provider-side Areas are complete, the artefact set is the base unit of the corpus:
- one namespace,
- one or more schemas under that namespace,
- one or more namespace skills, and
- an About Resource declared in one schema.
The provider-side grade is closed at this point. Selection-side grading (see 05-phases-selection.md) operates on aggregations of base units and never re-grades a base unit’s schemas.
About as a Schema Resource
Section titled “About as a Schema Resource”The About Resource is graded by the about-namespace Area. It is a markdown Resource declared in one schema of the namespace (main.resources), stored under providers/<ns>/<schema>/resources/about/, not a namespace route. The full content contract and the deterministic / non-deterministic split are defined in 11-about-convention.md.
A Resource technically never lives at namespace level — there is no namespace object to attach it to, only schemas. About is therefore inserted into one schema, and the detector searches for it namespace-wide.
The provider-side Areas produce gradingTier = autonomous. Per 06-determinism-and-tier.md, the maximum attainable grade on autonomous is B. A schema that should be eligible for grade A must additionally be graded on the selection side (group-bound, see 05-phases-selection.md).
Related
Section titled “Related”- ./00-overview.md — see chapter 00.
- ./01-default-journey.md — see chapter 01.
- ./02-eligibility.md — see chapter 02.
- ./03-tos.md — see chapter 03.
- ./05-phases-selection.md — see chapter 05.
- ./06-determinism-and-tier.md — see chapter 06.
- ./11-about-convention.md — see chapter 11.
- ./12-personas-contract.md — see chapter 12.
- ./13-skills.md — see chapter 13.
- ./19-folder-layout.md — see chapter 19.