# ========================== # robots.txt — Bloquear solo bots indeseados # ========================== # --- REGLA GENERAL: todo permitido --- User-agent: * Disallow: # ========================== # BLOQUEAR BOTS SEO / SCRAPERS / CRAWLERS AGRESIVOS # (La mayoría ignoran robots.txt, pero al menos quedan "marcados") # ========================== User-agent: AhrefsBot Disallow: / User-agent: SemrushBot Disallow: / User-agent: dotbot Disallow: / User-agent: MJ12bot Disallow: / User-agent: BLEXBot Disallow: / User-agent: DataForSeoBot Disallow: / User-agent: Barkrowler Disallow: / User-agent: CCBot Disallow: / User-agent: SEOkicks-Robot Disallow: / User-agent: SISTRIX Crawler Disallow: / User-agent: Sistrix Disallow: / User-agent: SerpstatBot Disallow: / User-agent: SERankingBot Disallow: / User-agent: SiteAuditBot Disallow: / User-agent: Screaming Frog SEO Spider Disallow: / User-agent: Sitebulb Disallow: / User-agent: DeepCrawl Disallow: / User-agent: LumarBot Disallow: / User-agent: RyteBot Disallow: / User-agent: OnCrawl Disallow: / User-agent: Linkdexbot Disallow: / User-agent: seoscanners.net Disallow: / User-agent: woorank Disallow: / User-agent: seobilitybot Disallow: / User-agent: RankActiveLinkBot Disallow: / User-agent: NetEstate NE Crawler Disallow: / User-agent: Mail.RU_Bot Disallow: / # --- Bloqueo de agentes genéricos de scraping (cortesía, muchos lo ignoran) --- User-agent: crawler Disallow: / User-agent: python-requests Disallow: / User-agent: python-httpx Disallow: / User-agent: Go-http-client Disallow: / User-agent: Java Disallow: / User-agent: libwww-perl Disallow: / User-agent: curl Disallow: / User-agent: wget Disallow: /