Crawler access rules, answer-first HTML chunking, connected Schema.org @graph entities, /llms.txt setup, and the 30-point monthly audit we run on cr8vstacks.com.
Instant full-chapter reader + downloadable PDF emailed directly · Zero spam · Privacy Policy
✓ Playbook Unlocked
The PB-01 playbook is yours.
Full chapters and checklists are open below, and your styled PDF copy is ready for instant download.
Which AI Search Crawlers to Allow (OAI-SearchBot vs. Training Bots)
Key Focus
Distinguish live search bots like OAI-SearchBot and PerplexityBot from pure model-training bots, and fix CDN WAF rules returning 403.
Modern AI visibility starts before a model reads a single paragraph. Separate search-index bots (OAI-SearchBot, ChatGPT-User, PerplexityBot, Googlebot) from optional training bots (GPTBot, ClaudeBot, Google-Extended) inside /robots.txt, then test live curl headers with each User-Agent to verify Cloudflare or host firewalls are not challenging automated fetchers. Keep an XML sitemap explicitly declared inside robots.txt alongside a self-referencing rel="canonical" tag.
STEP 01
The Buyer’s Question
Asked in ChatGPT, Perplexity, Claude, or Google AI Overviews.
STEP 02
Live Retrieval Search
OAI-SearchBot & PerplexityBot fetch top-ranking, readable SSR HTML pages.
Answer lifts concise sentences and cites canonical source links directly.
Google Classic Search (SEO)
Ranks full pages via domain backlinks and keyword density across 10 blue links.
ChatGPT & Perplexity (AEO)
Extracts specific HTML chunks, pricing tables, and verified entity citations.
Chapter 02
Writing & Structuring /llms.txt So Agents Map Your Site in One Request
Key Focus
Exact Markdown hierarchy for company summary, flagship services, case studies, calculators, and canonical links.
Publish /llms.txt at the domain root as lightweight UTF-8 Markdown: start with an H1 brand title, add a blockquote defining what your company builds and where you operate, then group clean Markdown links under Core Services, Flagship Case Studies, Free Tools, and Guides. Keep descriptions factual and verify HTTP 200 status with clean text headers.
Crawler
What It Does
If You Block It
OAI-SearchBot
Finds and indexes pages for ChatGPT’s live search features
Your pages aren’t shown or cited in ChatGPT search answers
GPTBot
Crawls public content that may be used for model training
Your content is opted out of training; live search is separate
ChatGPT-User
Visits a page in real time when a user asks ChatGPT to read a URL
Interactive prompt fetches fail to inspect your live page
PerplexityBot
Builds Perplexity’s live answer and citation index
Excluded from Perplexity source cards and inline footnotes
cr8vstacks.com/robots.txt (recommended production rules)
✓OpenAI and Perplexity update crawler state within ~24 hours of a robots.txt or WAF firewall allowlist change.
Chapter 03
Server-Side HTML Delivery: Zero Blank Shells for Stateless Fetchers
Key Focus
Why AI retrieval pipelines truncate slow bundles or skip client-hydrated React apps—and how WordPress SSR stays extractable.
Unlike full desktop browsers, most retrieval agents fetch raw HTTP HTML with tight byte and timeout budgets. Ensure primary H1/H2 copy, service pricing ranges, FAQs, and case study metrics render in initial HTML inside semantic <main>, <header>, <nav>, and <section> landmarks without waiting for hydration scripts.
An Answer-First Architecture, Part by Part
1
2
3
4
5
1
Buyer Question in H1Direct search or prompt intent matched clearly in the top heading.
2
20–40 Word Entity LeadSelf-contained definition sentence models can lift verbatim.
Semantic Tables & SpecsHTML <table>, <dl> & <ol> data blocks instead of vague marketing prose.
5
Named Author, Date & @graphVerified provenance + connected Organization & Service JSON-LD.
Chapter 04
Connecting Your JSON-LD @graph Across Organization, Services & FAQs
Key Focus
Unify @id anchors, authoritative sameAs profiles, BreadcrumbList trails, and Service schema without plugin conflicts.
Maintain one canonical Organization @id (#organization) and Person/Founder @id across every template. Link WebSite, WebPage, Service, Article, Case Study CreativeWork, FAQPage, and BreadcrumbList via matching @id references so crawlers resolve who provides what service, for which markets, and at what verified address.
Unified Organization & Service JSON-LD @graph (cr8vstacks.com)
Key Focus
Turn vague agency paragraphs into chunkable extraction anchors models can quote verbatim.
Lead core sections with a crisp 20–40 word entity definition sentence. Convert deliverables into semantic <ol> or <ul> lists, represent key metadata in <dl><dt><dd> blocks, and include clean HTML comparison <table> elements where buyers evaluate stacks, timelines, or agency models.
Key Focus
Why missing footer legal routes, thin Contact pages, and empty alt attributes quietly drop provenance scores.
AI systems downrank anonymous domains. Anchor physical or registered office details, direct email/phone contact routes, visible author attribution with dateModified timestamps, /privacy-policy/ and /terms/ governance disclosures in the global footer, and descriptive alt attributes on product screens and diagrams.
STEP 01
Prompt Audit
Run your top 20 buyer prompts across ChatGPT, Perplexity, Claude & Google AI.
STEP 02
Log Citations
Record whether your domain is named, which URLs are cited, and who ranks instead.
STEP 03
Compare Delta
Track month-over-month citation share and newly indexed comparison or service pages.
STEP 04
Verify UTM Traffic
Check analytics for referrals tagged utm_source=chatgpt.com and perplexity.ai.
STEP 05
30-Point AEO Scan
Run modified templates through the Cr8v Stacks Free AEO Checker after deploys.
STEP 06
Ship Next Fix
Prioritize 0-point blockers first: crawler blocks, SSR gaps, and missing schema edges.
Live Cr8v Stacks Diagnostic Scoreboard
cr8vstacks.com · Verified WordPress SSR
92 / 100 · GRADE A
8 / 8 AllowedAI Search Crawlers
< 1.1s SSRZero JS Hydration Wait
Unified @graphOrganization + Services + FAQ
Chapter 07
The 30-Point Monthly AEO Audit Loop in Cr8v Stacks Free AEO Checker
Key Focus
Run a repeatable monthly scorecard in 5 minutes and prioritize high-point remediation fixes first.
Audit your homepage, primary service landing page, and top guide once per month using /toolkits/free-aeo-checker/. Track Bot Discovery, SSR Word Count, Schema Graph edges, Extractable Answers, Authority signals, and Agent Actions—fixing 0-point failures first.