🤖 Module 90: Agentic-Web Monetization: Pay-Per-Crawl, Robots.txt & Feed-First Content

High Priority   Date: 2026-09-20 | Category: Agentic Web / Technical SEO | Module ID: 90

🚨 The Signal

The crawl economy is flipping from "block all bots" to "price your content per agent visit." In the week of Sep 14–19, 2026: Google piloted AI-agent content payments (with Cloudflare and Microsoft running competing models), a publisher charged AI agents a penny per page. And watched Claude pay it, Cloudflare began auto-writing robots.txt for AI crawler governance, and Google updated Mediapartners-Google crawler guidance for the agent era.

Sources: Search Engine Journal, "Google AI Payment Pilot, Search Profiles At 10,000" (Sep 19, 2026) · "Google's AI Payment Pilot Vs. Cloudflare & Microsoft Models" (Sep 18) · "I Made My Website Charge AI Agents A Penny Per Page, Then I Watched Claude Pay It" (@suganthan, Sep 18) · "Cloudflare Will Write Your Robots.txt" (@slobodanmanic, Sep 18) · "Google Updates Mediapartners-Google Crawler Guidance" (@martinibuster, Sep 18). Radar: 2026-09-20 REX Radar.

1. What changed (plain English)

2. Why it matters for Footprint clients

3. The Footprint playbook

✅ Do this (in order)

  1. Robots.txt audit (this week): confirm every client site explicitly addresses AI crawlers (GPTBot, ClaudeBot/CCBot, PerplexityBot, Google-Extended, Mediapartners-Google). Decide allow vs. meter per client, never leave it to a CDN default unreviewed.
  2. Feed-first content: every money page gets Offer and/or FAQPage schema; publish a clean content/product feed path an agent can consume without rendering JS.
  3. Separate the logs: Mediapartners-Google and known AI user-agents get their own log segment; report agent hits alongside human traffic monthly.
  4. Canonical hygiene (Mueller warning, Sep 18): every money page self-canonicals; no cross-domain canonicals except deliberate migrations (prefer 301s); verify Google-selected canonical in GSC after any syndication.
  5. Price later, structure now: don't paywall agents yet on lead-gen sites, citations are worth more than pennies today. Build the metering capability; switch it on when the market matures.

⛔ Don't do this

4. Starter robots.txt pattern (adapt per client)

# Human search: allow
User-agent: Googlebot
Allow: /

# AI training crawls: client decides (example = metered stance)
User-agent: GPTBot
User-agent: CCBot
User-agent: PerplexityBot
# Allow agent shopping/comparison reads; revisit pricing Q4 2026
Allow: /

# Google agent + ads surfaces: keep distinct from AdSense
User-agent: Google-Extended
User-agent: Mediapartners-Google
Allow: /

5. GBP-first tie-in (Diamante framework)

Nothing in this module demotes GBP: reviews velocity, photos, Q&A, and services still feed every SERP variant, especially now that Sterling Sky confirms post-May-2026 SERPs differ per user (Module 90 appendix, Signal 5). The agentic layer is additive: GBP wins the human local pack; feeds + schema win the agent channel.

6. WITNESS checklist (verify before marking done)

← Back to All Modules
Educational content only, no carrier, eligibility, or legal advice. Insurance pages keep licensed-author bylines and compliant disclaimers. Review through compliance-filter before client delivery.