User-agent: * Allow: / # Sitemaps Sitemap: https://www.intratio.com/sitemap.xml # Guide for AI assistants and agents (llmstxt.org) # https://www.intratio.com/llms.txt # Administrative and System Paths Disallow: /admin/ Disallow: /django-admin/ # User Account Areas # blocking /accounts/ prevents bots from wasting budget on login/signup/reset pages Disallow: /accounts/ # Referrals Private Areas Disallow: /referrals/dashboard/ Disallow: /referrals/admin/ Disallow: /referrals/api/ Disallow: /referrals/ajax/ # API and Backend endpoints # Blocking these prevents duplicate content or exposing raw data that shouldn't be indexed Disallow: /api/ Disallow: /API/ Disallow: /webhooks/ Disallow: /ajax_calls/ Disallow: /stock_price/ Disallow: /fundamentals_iframe/ Disallow: /signals-iframe/ Disallow: /portfolio/ # Static Data Files (unless needed for rendering) Disallow: /*.json # Actions Disallow: /subscribe/ Disallow: /unsubscribe/ # GPTBot (OpenAI training/index crawler) did 148k renders in the week of # 2026-07-21, mostly locale-duplicated ticker pages (13k tickers x 14 locales). # Restrict it to English pages: keeps ChatGPT presence at ~7% of the crawl cost. # ChatGPT-User (live fetches for users) and OAI-SearchBot keep full access. # ClaudeBot (Anthropic's index crawler) did 110k renders / 5.1 GB on # 2026-10-14 alone (4k the day before), same locale-duplicated ticker pages: # same English-only restriction. Claude-User and Claude-SearchBot keep full access. # PerplexityBot (Perplexity's index crawler) fetched ~26k full pages / 1.6 GB # a day once the 2026-10-15 scraper block was live, mostly locale ticker # pages: same English-only restriction. Perplexity-User keeps full access. User-agent: GPTBot User-agent: ClaudeBot User-agent: PerplexityBot Allow: /en/ Allow: /$ Allow: /sitemap.xml Allow: /llms.txt Disallow: /en/subscribe/ Disallow: /en/unsubscribe/ Disallow: / # SEO/backlink harvesters and AI-training-only scrapers: no business value, # significant AWS cost (Ahrefs+Semrush alone: ~200k renders/week measured # Jul 2026). They are also blocked server-side with 403. User-agent: AhrefsBot User-agent: SemrushBot User-agent: MJ12bot User-agent: DotBot User-agent: BLEXBot User-agent: serpstatbot User-agent: DataForSeoBot User-agent: PetalBot User-agent: ZoominfoBot User-agent: MegaIndex User-agent: SeekportBot User-agent: AwarioBot User-agent: Barkrowler User-agent: SEOkicks User-agent: SISTRIX User-agent: linkdexbot User-agent: rogerbot User-agent: Bytespider User-agent: CCBot User-agent: omgili User-agent: Diffbot User-agent: Timpibot User-agent: ImagesiftBot User-agent: meta-externalagent User-agent: SleepBot Disallow: /