# AI training crawlers: blocked User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: CCBot User-agent: Applebot-Extended Disallow: / # Everyone else, including search and answer crawlers User-agent: * User-agent: AhrefsSiteAudit User-agent: AhrefsBot Disallow: / Allow: /$ Allow: /about/$ Allow: /about$ Allow: /pricing/$ Allow: /pricing$ Allow: /contact/$ Allow: /contact$ Allow: /supported-counties/$ Allow: /supported-counties$ Allow: /refer-a-friend/$ Allow: /refer-a-friend$ Allow: /terms-of-service/$ Allow: /terms-of-service$ Allow: /privacy-policy/$ Allow: /privacy-policy$ Allow: /refund-policy/$ Allow: /refund-policy$ Allow: /custom-comps-report-request/$ Allow: /custom-comps-report-request$ # 301 to the blog (page removed); allowed so crawlers see the redirect Allow: /lower-my-property-taxes/$ Allow: /lower-my-property-taxes$ Allow: /assets/ Allow: /assets/* Allow: /static/ Allow: /static/* Allow: /_astro/ Allow: /_astro/* Sitemap: https://squaredeal.tax/sitemap-index.xml