# ======================================================================== # robots.txt for American Commercial Glass (acglass.com) # Updated: 2026-09-16 # Explicit AI crawler permissions for answer engine optimization (GEO) # ======================================================================== # Content Signals Policy (https://contentsignals.org/) # search=yes: allow indexing for traditional search results # ai-input=yes: allow content as real-time AI input (RAG, AI overviews) # ai-train=yes: allow content for AI model training (lets ACG be named from model memory) # ------------------------------------------------------------------------ # One group governs every crawler. # # Google's specification: "Only one applicable group is used for a crawler... # User-agent-specific groups and the global * group are not combined." # https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt # # This file previously carried 32 named crawler groups that each contained # nothing but "Allow: /". Because groups are never combined, every one of those # crawlers -- GPTBot, ClaudeBot, PerplexityBot, Bingbot, CCBot, Google-Extended, # Applebot and the rest -- matched its own group and was therefore exempt from # all three Disallow rules below. They could crawl the dealer admin view and the # JSON-LD fragment. The groups looked permissive and were actually a hole. # # Deleting them changes nothing about what is permitted, because "Allow: /" was # already the default, and it restores the three Disallow rules for everyone. # # The allow-everything-that-cites posture is deliberate and is preserved: AI and # search crawlers are welcome. Only Bytespider is refused, and that is a # deliberate exception kept below. # # Content-Signal is [SPECULATIVE]. It is not a supported robots.txt field and no # crawler is known to honour it. It is retained as a declaration of intent. Do # not describe it as a control. # # Kept as a comment since 2026-10-01: as a live line it is an unknown directive, # which robots.txt validators (Lighthouse "robots.txt is not valid") flag as an # error on every page. The declaration of intent is unchanged. # ------------------------------------------------------------------------ User-agent: * # Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / # Internal admin & dealer-only views - no value to crawlers, contain no public content. Disallow: /dealer/admin.html Disallow: /dealer/thanks.html # JSON-LD include fragment, not a page. Kept out of sitemaps. The file also # carries noindex,follow + a self-canonical. Do not wrap it in a full HTML # document; that would trip tn-claim-guard. Disallow: /services-schema-block.html # Drafts and internal paths must be blocked for all crawlers including Googlebot. # Groups do not combine, so these Disallows belong in User-agent: *, not after Bytespider. Disallow: /drafts/ Disallow: /_internal/ # ------------------------------------------------------------------------ # The one refusal. TikTok/Doubao's crawler takes content without citing it. # ------------------------------------------------------------------------ User-agent: Bytespider Disallow: / # ------------------------------------------------------------------------ # Sitemaps # One current apex sitemap only. Child urlsets remain on disk because # crawl-check.py asserts master == union of those urlsets. They are not # advertised here or in the sitemap index. GSC was still crawling the # old index (2,516) and a stale www sitemap (342, last read 2026-04-24). # The public pages child is a retired empty stub (do not resubmit). The # former 418-URL child is a CI fixture only and is not published. # ------------------------------------------------------------------------ Sitemap: https://acglass.com/sitemap.xml # ------------------------------------------------------------------------ # AI / GEO signals # ------------------------------------------------------------------------ # The llms.txt file describes site structure for AI crawlers: # https://acglass.com/llms.txt # A comprehensive knowledge base for AI answer engines: # https://acglass.com/llms-full.txt # Explicit AI crawler welcome/permission policy: # https://acglass.com/ai.txt