🤖 Module 90: Agentic-Web Monetization: Pay-Per-Crawl, Robots.txt & Feed-First Content
High Priority
Date: 2026-09-20 | Category: Agentic Web / Technical SEO | Module ID: 90
🚨 The Signal
The crawl economy is flipping from "block all bots" to "price your content per agent visit." In the week of Sep 14–19, 2026: Google piloted AI-agent content payments (with Cloudflare and Microsoft running competing models), a publisher charged AI agents a penny per page. And watched Claude pay it, Cloudflare began auto-writing robots.txt for AI crawler governance, and Google updated Mediapartners-Google crawler guidance for the agent era.
Sources: Search Engine Journal, "Google AI Payment Pilot, Search Profiles At 10,000" (Sep 19, 2026) · "Google's AI Payment Pilot Vs. Cloudflare & Microsoft Models" (Sep 18) · "I Made My Website Charge AI Agents A Penny Per Page, Then I Watched Claude Pay It" (@suganthan, Sep 18) · "Cloudflare Will Write Your Robots.txt" (@slobodanmanic, Sep 18) · "Google Updates Mediapartners-Google Crawler Guidance" (@martinibuster, Sep 18). Radar:
2026-09-20 REX Radar.
1. What changed (plain English)
- Three payment models, one direction: Google, Cloudflare, and Microsoft each propose a way for publishers to get paid when AI agents read their pages. Details differ; the direction is identical, machine access becomes a metered product.
- Proof it works: a per-page micro-charge (one cent) did not stop a frontier agent, Claude completed the transaction and read the content. Friction is low; enforcement is code, not lawsuits.
- Robots.txt becomes a price list: Cloudflare auto-generating robots.txt means many sites will soon carry AI-crawler directives they never hand-wrote. Defaults matter, review them.
- Mediapartners-Google split: updated guidance separates agent-era crawling from legacy AdSense crawling. Log them separately or fly blind.
2. Why it matters for Footprint clients
- Lead-gen SMB and insurance content (city pages, FAQ hubs, plan guides) is exactly what shopping/comparison agents will read first. Clean schema + feeds = tollable, citable assets.
- GBP-first still rules local. But the website now earns twice: human conversions AND agent reads (paid or cited).
- Sites with no machine-readable layer (no feeds, thin schema, JS-only content) will be skipped by agents, same as being de-indexed for the agentic channel.
3. The Footprint playbook
✅ Do this (in order)
- Robots.txt audit (this week): confirm every client site explicitly addresses AI crawlers (GPTBot, ClaudeBot/CCBot, PerplexityBot, Google-Extended, Mediapartners-Google). Decide allow vs. meter per client, never leave it to a CDN default unreviewed.
- Feed-first content: every money page gets Offer and/or FAQPage schema; publish a clean content/product feed path an agent can consume without rendering JS.
- Separate the logs: Mediapartners-Google and known AI user-agents get their own log segment; report agent hits alongside human traffic monthly.
- Canonical hygiene (Mueller warning, Sep 18): every money page self-canonicals; no cross-domain canonicals except deliberate migrations (prefer 301s); verify Google-selected canonical in GSC after any syndication.
- Price later, structure now: don't paywall agents yet on lead-gen sites, citations are worth more than pennies today. Build the metering capability; switch it on when the market matures.
⛔ Don't do this
- Don't blanket-block all AI crawlers. You trade pennies of control for invisibility in AI answers (see Modules 10/16/87).
- Don't accept Cloudflare's auto-robots.txt blindly, read the diff.
- Don't use cross-domain canonicals for syndication convenience. A misfire can de-index your own pages.
- Don't rent parasite placements. The Aug 28 Site Reputation Policy update keeps that door under enforcement.
4. Starter robots.txt pattern (adapt per client)
# Human search: allow
User-agent: Googlebot
Allow: /
# AI training crawls: client decides (example = metered stance)
User-agent: GPTBot
User-agent: CCBot
User-agent: PerplexityBot
# Allow agent shopping/comparison reads; revisit pricing Q4 2026
Allow: /
# Google agent + ads surfaces: keep distinct from AdSense
User-agent: Google-Extended
User-agent: Mediapartners-Google
Allow: /
5. GBP-first tie-in (Diamante framework)
Nothing in this module demotes GBP: reviews velocity, photos, Q&A, and services still feed every SERP variant, especially now that Sterling Sky confirms post-May-2026 SERPs differ per user (Module 90 appendix, Signal 5). The agentic layer is additive: GBP wins the human local pack; feeds + schema win the agent channel.
6. WITNESS checklist (verify before marking done)
- robots.txt fetched live and AI-crawler directives documented per client domain
- Offer/FAQ schema validated (Rich Results Test) on top 5 money pages
- GSC canonical check passes on all money pages post-audit
- Agent user-agents segmented in analytics/logs
- Site Reputation Policy risk: zero rented placements confirmed
← Back to All Modules
Educational content only, no carrier, eligibility, or legal advice. Insurance pages keep licensed-author bylines and compliant disclaimers. Review through compliance-filter before client delivery.