Rubric v1.6

Every scan is scored against the published rubric below. Rubric changes require two approvals — EAC engineering (measurement correctness) and Evangent leadership (grading integrity) — and every change bumps the version string. Historical scans retain the rubric version that was active when they ran.

The two-tier composite

v1.3 introduced the two-tier Hygiene + Frontier composite, and v1.6 (September 2026) is the first release to re-cut what goes inside it. Hygiene measures whether agents can read and operate the site at all: a robots policy that lets in the crawlers that decide citations, an edge that lets an honest crawler through, content that survives a plain HTTP fetch, a page an agent can afford in tokens with nothing hidden from readers, semantic hooks and machine-readable dates, and controls a browser-driving agent can work. Frontier measures the agent-native surface, weighted differently for content, interactive, and commerce sites: Accept: text/markdown negotiation, an AI catalog at /.well-known/ai-catalog.json, a machine-readable usage policy, advertised markdown alternates, a valid llms.txt, WebMCP tool declarations, A2A agent cards, and for stores the UCP and ACP commerce protocols. Most well-built 2026 sites land in the B range on hygiene alone; Frontier is what lifts a site to A and A+.

What changed in v1.6

Two checks were measuring the wrong thing and are retired. content-signal-header read an HTTP header that mostly exists as a CDN default; ai-usage-policy reads the policy where it lives, a Content-Signal or Content-Usage line in robots.txt or an RSL licence, and scores explicitness rather than permissiveness. mcp-server-discovery probed /.well-known/mcp.json, a path no MCP proposal ever specified; agent-manifest validates the AI catalog and the MCP server cards, A2A cards, and agent skills it links.
Crawlers are scored by class. Search-index crawlers decide whether a site can be cited, user-triggered fetchers produce the referrals, and training crawlers are most of the volume with almost none of the referrals, so robots-ai-policy weights access 45 / 35 / 20 across those classes instead of counting tokens equally. The taxonomy drops two tokens Anthropic never documented and adds the search and user-fetch tokens every major vendor now publishes.
Hygiene has a fourth sub-category, agent operability, plus new checks for edge reachability, machine-readable dates, and hidden text. Accept: text/markdown negotiation moves to Frontier, where it belongs, and token efficiency measures the absolute cost of the page.
The v1.4 hidden-content allowance is reversed (below).
A third site type, commerce, with its own Frontier column: the UCP business profile, the ACP discovery document, storefront MCP, and product structured data.
Frontier checks are redefined around validity and discoverability: llms.txt is scored on the Lighthouse validity rules rather than on presence, markdown discoverability scores the advertised rel=alternate link, the documentation-surface check is rebalanced away from llms-full.txt, and the A2A and WebMCP checks read the current specs.
The blend changes so that good hygiene alone reaches B (below). Cut-points and the human-only ceiling are unchanged.

The evidence behind each change, with sources, is in docs/rubric-v1.6.md in the open-source repository.

Hidden text

v1.4 allowed an “AI-native content layer”: prose present in the DOM for agents but hidden from readers with a screen-reader-only class. v1.6 withdraws that allowance. Google’s spam policies on hidden text and cloaking now apply to AI Overviews and AI Mode, Bing discourages bot-only content, and hidden DOM text is the carrier of the prompt injection security researchers find in the wild. Prose hidden from readers earns no content credit; more than 150 words of it is flagged on the hidden-text-risk check; and instruction-like text hidden anywhere in the page (hidden elements, HTML comments, JSON-LD strings) scores zero, with the excerpt quoted so the finding can be verified. Ordinary accessibility work is unaffected: skip links, icon labels, and other short hidden text are ignored, and JSON-LD strings never count toward the word threshold, so FAQPage markup is fine. Agent-facing prose belongs in a visible section or a markdown alternate, which the Frontier tier rewards.

Composite formula

// Content site
composite = min(100, hygiene * 0.90 + frontier * 0.15)

// Interactive and commerce sites
composite = min(100, hygiene * 0.85 + frontier * 0.20)

The weights sum to more than one on purpose. Perfect hygiene alone reaches the top of B for every site type (a zero-frontier content site’s raw 90 is clamped to 85 by the ceiling below), and frontier work lifts a site into A and A+ instead of merely rescuing it from C. Interactive and commerce sites carry more frontier weight because the tool surface and the transaction surface are where agents get value from them.

Site-type classification

Sites are classified automatically as Content (blog, news, docs, nonprofit), Interactive (SaaS, booking, dashboards), or Commerce (stores), with commerce taking precedence over interactive and interactive over content. Commerce needs a decisive signal: Product structured data with offer details (availability, a SKU or GTIN) or several products, a checkout or cart form, a store platform fingerprint such as Shopify or WooCommerce assets, or a valid UCP or ACP discovery document; a bare Offer, a lone Product with only a price, or a products menu is only a hint. The classifier can be overridden from the scan form, and the scorecard records which type applied.

Hygiene checks

Hygiene is a weighted average of four sub-categories. The sub-category weights roll up into the Hygiene composite; within each sub-category the check weights sum to 100%.

Sub-category	Weight in Hygiene
Crawler accessibility	28%
Content readability	32%
Semantic structure	25%
Agent operability	15%

Check	Sub-category	Within-sub-category weight
robots.txt AI-agent policy	crawler	40%
JavaScript dependency	crawler	35%
Edge reachability	crawler	15%
Login / paywall wall	crawler	10%
Content-to-chrome ratio	readability	45%
Page cost in tokens	readability	35%
Hidden text and hidden instructions	readability	20%
Heading hierarchy	semantic	25%
JSON-LD / schema.org	semantic	20%
sitemap.xml	semantic	20%
Machine-readable dates	semantic	15%
Language declaration	semantic	10%
Open Graph & Twitter Cards	semantic	10%
Accessible controls and landmarks	operability	60%
First-load friction	operability	40%

Frontier checks

Frontier is weighted per site type and each column sums to 100%; a check with no weight for a site type (shown as a dash) is neither run nor counted for it. Content sites are scored on what answer engines and coding agents read, interactive sites on the tool surface, and commerce sites on the protocols shopping agents transact through. The top scores (95+) are reserved for sites that have genuinely built comprehensive agentic-web support.

Check	Content	Interactive	Commerce
Accept: text/markdown negotiation	25%	20%	3%
Documentation surface	20%	18%	5%
Machine-readable AI usage policy	12%	10%	5%
Rich structured data	12%	—	—
Markdown discoverability	11%	7%	—
Agent capability manifest	10%	25%	10%
llms.txt validity	10%	5%	2%
WebMCP tool declarations	—	10%	8%
A2A agent card	—	5%	2%
UCP business profile	—	—	25%
ACP discovery document	—	—	15%
Product structured data	—	—	15%
Storefront MCP	—	—	10%

Letter grades

Composite	Grade	Meaning
95–100	A+	Agentic-web leadership. Genuinely exceptional.
88–94	A	Ahead of the curve on AI readability.
78–87	B	Fundamentals solid; frontier features would push to A.
65–77	C	Readable, but real gaps. Clear next steps.
50–64	D	Serious issues; agents struggle here.
0–49	F	Unreadable to most agents.

Human-only ceiling

A site that scores below 15 on Frontier — meaning no deliberate agent-native step yet — has its composite capped at 85 (top of B) regardless of how strong its Hygiene score is. One step clears it: markdown negotiation, an agent catalog, a usage-policy line in robots.txt, or a valid llms.txt. Rich structured data alone does not, because a perfect score on it is worth 12 points of the content column, below the floor, and the 2026 evidence says structured data is not an agent-native surface. The cap is disclosed on every affected scorecard. It is aspirational, not punishing: B is a respectable grade that says “you’re doing the basics right”, and the cap preserves a meaningful difference between sites that have invested in the agentic web and sites that have only invested in the traditional web.

Where we're opinionated

The rubric tracks the surfaces agents actually use in 2026 and weights them by evidence, not by how often they are recommended. Cloudflare’s April 2026 scan of the top 200,000 domains found 78% with a robots.txt, 4% with a Content Signals line, 3.9% answering Accept: text/markdown, and MCP server cards or API catalogs on fewer than fifteen sites. That is why negotiation, policy, and discoverability carry the most Frontier weight for content sites, and why the agent catalog is weighted for interactive sites, where a tool surface matters. llms.txt stays a minor signal, scored on validity rather than presence: Ahrefs found that 97% of the files are never fetched and no answer engine reads them, but Chrome Lighthouse validates the format and coding agents follow the links. Where we measure the same things as Chrome’s Agentic Browsing category or Cloudflare’s Agent Readiness report we say so, so a site can compare graders. And we never impersonate another vendor’s crawler to see how an edge treats it: reachability is scored from CanAISeeBot’s own honestly identified fetch, and the evidence states that limitation.

Governance

Rubric changes are tracked in the lib/rubric/ directory of the open-source repo, one file per version. v1.2 through v1.5 stay in-tree alongside v1.6 so historical scans continue to read their own version's scoring logic, and docs/ carries the human-readable rationale for each version. The rubric is reviewed quarterly against the current agentic-web landscape; the next review re-checks Cloudflare’s 15 September crawler defaults, the outcome of the WebMCP origin trial, the IETF aipref and Web Bot Auth drafts, and the final server-card path.