Provider-Side Grading Areas
Dieser Inhalt ist noch nicht in deiner Sprache verfügbar.
Conformance language (MUST/SHOULD/MAY) follows BCP 14 [RFC2119]/[RFC8174] as defined in
00-overview.md. The binding source is the FlowMCP Schemas Specification v4.2.0.
Introduction
Section titled “Introduction”This chapter is the normative source for the provider-side grading Areas — the Areas that grade one schema inside one namespace without group context. It replaces the linear phase model of earlier spec versions with an Area model: each Area is a self-contained grading rubric attached to the primitive it evaluates, written to a _gradings/ folder next to that primitive (see 19-folder-layout.md).
The provider side produces the base unit of the FlowMCP corpus: one namespace with one or more schemas, namespace skills, and an About Resource. Higher-level grouping (selection side) is defined separately in 05-phases-selection.md.
A schema graded only on the provider side has gradingTier = autonomous. Per 06-determinism-and-tier.md, the maximum attainable grade on this tier is B. Grade A requires a group-bound contribution from the selection side.
The Provider-Side Areas
Section titled “The Provider-Side Areas”The grading system defines eleven Areas in total (see 05-phases-selection.md and 19-folder-layout.md). Of these, the following six are provider-side (everything except the selection Areas):
| # | Area | Evaluates | _gradings/ location | Persona | Det/Non-Det |
|---|---|---|---|---|---|
| 1 | single-test | one tool | providers/<ns>/<schema>/tools/<tool>/_gradings/ | no | deterministic gate + non-deterministic |
| 2 | tools-aggregate-schema | the tools collection of one schema | providers/<ns>/<schema>/_gradings/ | no | both |
| 3 | tools-aggregate-namespace | tools across the namespace | providers/<ns>/_gradings/ | no | both |
| 4 | namespace-description | namespace metadata | providers/<ns>/_gradings/ | no | non-deterministic |
| 5 | namespace-skills | one namespace skill (per skill) | providers/<ns>/<schema>/skills/<skill>/_gradings/ | yes | non-deterministic |
| 6 | about-namespace | the About Resource (declared in one schema) | providers/<ns>/<schema>/resources/about/_gradings/ | yes | deterministic (route-exists) + non-deterministic |
The remaining five Areas (about-selection, selection-skills-L1, selection-skills-L2, selection-skills-L3, selection-aggregate) are selection-side and live in 05-phases-selection.md.
Each Area is graded independently. There is no fixed linear order between Areas; the only ordering obligations are the cascade and veto procedures (see Area Procedures) and the deterministic-first rule of 06-determinism-and-tier.md.
Area Procedures
Section titled “Area Procedures”The Area model retains four procedures that previously lived inside the phase model. They are now expressed as rules that apply across the provider-side Areas.
Description Cascade (within single-test and tools-aggregate-*)
Section titled “Description Cascade (within single-test and tools-aggregate-*)”The description cascade is a mandatory ordered procedure for validating tool descriptions. It MUST be executed in the following order; skipping or reordering steps is a finding.
- Run tests against the endpoint. SHOULD: at least 3 working tests per tool (status true and non-empty data), covering the breadth of the parameter space. Fewer than 3 working tests blocks the tool from full grading and is recorded with a status reason (see
06-determinism-and-tier.md). - Check the responses and validate the tool description against the actual responses.
- Normalise / update the tool description to match the validated responses.
- All tools, resources, and prompts MUST have descriptions — and each description MUST be individually checked.
- Descriptions MUST be neutral — see Description Neutrality.
The cascade is a contract: outputs of step n are inputs of step n+1. A failure in any step halts the cascade for the affected tool and is recorded as a finding. single-test carries the per-tool cascade result; tools-aggregate-schema and tools-aggregate-namespace aggregate the per-tool cascade outcomes.
Description Neutrality (cross-cutting)
Section titled “Description Neutrality (cross-cutting)”The neutrality rule (cascade step 5) is normative and worth restating:
- A tool description states what the tool does (capabilities, parameters, return shape).
- A tool description MUST NOT state what for it should be used (application scenarios, persona use cases, “good for X”).
- Application scenarios and persona use cases belong in the About Resource (
11-about-convention.md) — not in the tool description.
This separation is essential for LLM-grader reproducibility: neutral descriptions can be deterministically compared to the observed API behaviour; mixed descriptions cannot.
Cascade Stop / Veto (cross-cutting)
Section titled “Cascade Stop / Veto (cross-cutting)”A failed gate MUST halt the dependent grading for the affected schema. A categorical veto raised in any Area MUST stop further grading for that schema. Examples:
api-key-domain-mismatchveto — when the API key declared in the schema metadata does not match the API root domain, the veto is raised and thesingle-testlive tests for the affected tools MUST NOT be treated as pass.- HTTP 4xx — when a tool returns HTTP 4xx (including 401/403), the response MUST NOT be treated as “auth-pass” (see
06-determinism-and-tier.md). The description cascade for that tool cannot be completed and is recorded as a finding. - Eligibility violation — when an endpoint fails an exclusion criterion under
02-eligibility.mdand the schema author insists on including it, the schema is rejected and dependent Areas do not run for it.
Cascade-stop events are recorded in the grading entry. They do not lower the grade silently — a categorical veto replaces the aggregate grade with REJECTED, which the index derivation maps to the terminal status rejected (see 06-determinism-and-tier.md and 19-folder-layout.md).
Base Unit (cross-cutting)
Section titled “Base Unit (cross-cutting)”When the provider-side Areas are complete, the artefact set is the base unit of the corpus:
- one namespace,
- one or more schemas under that namespace,
- one or more namespace skills, and
- an About Resource declared in one schema.
The provider-side grade is closed at this point. Selection-side grading (see 05-phases-selection.md) operates on aggregations of base units and never re-grades a base unit’s schemas.
About as a Schema Resource
Section titled “About as a Schema Resource”The About Resource is graded by the about-namespace Area. It is a markdown Resource declared in one schema of the namespace (main.resources), stored under providers/<ns>/<schema>/resources/about/, not a namespace route. The full content contract and the deterministic / non-deterministic split are defined in 11-about-convention.md.
A Resource technically never lives at namespace level — there is no namespace object to attach it to, only schemas. About is therefore inserted into one schema, and the detector searches for it namespace-wide.
The provider-side Areas produce gradingTier = autonomous. Per 06-determinism-and-tier.md, the maximum attainable grade on autonomous is B. A schema that should be eligible for grade A must additionally be graded on the selection side (group-bound, see 05-phases-selection.md).
Cross-References
Section titled “Cross-References”01-default-journey.md— maximalism and the completeness contribution tosingle-test/tools-aggregate-schema.02-eligibility.md— endpoint eligibility (input to schema authoring).03-tos.md— ToS check.05-phases-selection.md— selection-side Areas (consume provider-side base units).06-determinism-and-tier.md— tier and determinism rules (max-grade-B onautonomous).11-about-convention.md— About Resource content contract.12-personas-contract.md— personas referenced by persona-bearing Areas.13-skills.md— namespace skills and selection skills.19-folder-layout.md—_gradings/placement and theindex.jsonrollup.