--- title: "Block Meta Crawler in robots.txt and Keep Link Previews" description: "Keep Facebook link previews working and refuse Meta AI training. Copy-paste robots.txt lines for each Meta crawler, plus WordPress, Wix and Squarespace steps." url: "https://webleveling.com/blog/allow-or-block-meta-crawlers/" lang: "en" published: "2026-10-17" modified: "2026-10-17" author: "Cal Hewitt" tags: ["seo","website-maintenance","small-business"] image: "https://webleveling.com/og/blog-allow-or-block-meta-crawlers.jpg" alternates: es: "https://webleveling.com/es/blog/bloquear-bots-de-meta-en-robots-txt.md" --- # Block Meta Crawler in robots.txt and Keep Link Previews Keep Facebook link previews working and refuse Meta AI training. Copy-paste robots.txt lines for each Meta crawler, plus WordPress, Wix and Squarespace steps. ![A brass key and a closed padlock on a plain wooden desk beside a blank paper tag.](https://webleveling.com/blog/allow-or-block-meta-crawlers/meta-crawlers-featured.jpg) You share your pages on Facebook and Instagram, and you want the preview card to keep showing the right title, image and description. You also do not want Meta using your site to train its AI. Those two goals do not conflict, because Meta runs separate crawlers for each job. The short version: leave the link-preview crawler allowed and give the AI crawler its own block in robots.txt. This matters this month because Meta's Commercial Terms preview, as of October 17, 2026, says the new terms take effect October 30, 2026, and they authorize Meta, or entities acting for Meta, to use automated software to crawl domains and pages tied to a business's products or services. The wording of that clause is covered in [what Meta's new terms say about crawling your site](https://webleveling.com/blog/meta-terms-crawl-business-websites/). What you control is your robots.txt file: the exact lines to paste and where they go. **Key Takeaways** **Keep previews, refuse training** Leave \`facebookexternalhit\` allowed and add a separate group that disallows \`Meta-ExternalAgent\`, the crawler Meta documents for training foundation AI models. **Five crawlers, five decisions** Meta documents \`facebookexternalhit\`, \`Meta-WebIndexer\`, \`Meta-ExternalAds\`, \`Meta-ExternalAgent\` and \`Meta-ExternalFetcher\`, and blocking one does not block the others. **A rule is a request** Meta says two of its crawlers may bypass robots.txt, and a new rule does not remove content that was already collected, so check the live file and your logs after the change. ## Leave facebookexternalhit allowed and give Meta-ExternalAgent its own block The rule set for an owner who wants previews and does not want AI training is short. Add a group that allows \`facebookexternalhit\`, and a group that disallows \`Meta-ExternalAgent\`. Each crawler reads the group that names it, so the two decisions stay separate. \`\`\`text User-agent: facebookexternalhit Allow: / User-agent: Meta-ExternalAgent Disallow: / \`\`\` Meta describes \`facebookexternalhit\` as the crawler that fetches a link when someone shares it on Facebook, Instagram or Messenger, which builds the preview card. Meta describes \`Meta-ExternalAgent\` as crawling the web for uses such as training foundation AI models or improving products by indexing content directly. Meta's [crawler documentation](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/) says to use robots.txt to set these preferences, and says it prefers that over non-standard formats like NoAI tags. ![A single brass key on a wooden desk beside a closed padlock and a blank paper tag.](https://webleveling.com/blog/allow-or-block-meta-crawlers/link-preview-allowed-training-blocked.jpg) One crawler gets the key for previews; another gets a closed lock. The named \`facebookexternalhit\` group matters most when your file already has a blanket rule. Crawlers pick the most specific matching group, according to [Google's robots.txt specification](https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec), so a named allow group is how you keep previews working under a \`User-agent: \*\` block. The standard itself, [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309), treats the user-agent token without regard to case, so the capital letters in Meta's names are for readability. ## Meta runs five crawlers, and each one has a different job A single search for "Meta bot" turns up several names, and owners often treat them as one thing. Meta's documentation lists five, each with its own purpose and its own robots.txt behavior. Read the table before you add a line, because the cost of blocking differs for each row. Meta's five documented crawlers | Crawler | What Meta says it does | Honors robots.txt? | If you block it | | --- | --- | --- | --- | | \`facebookexternalhit\` | Fetches a link when it is shared on Facebook, Instagram or Messenger, to build the preview | May bypass for security or integrity checks | Preview cards can break | | \`Meta-WebIndexer\` | Improves Meta AI search results; allowing it helps Meta cite and link to your content | Yes | Meta AI is less likely to cite or link to you | | \`Meta-ExternalAds\` | Crawls pages for advertising and other business products | Yes | Ad-related checks of your landing pages can be affected | | \`Meta-ExternalAgent\` | Crawls for training foundation AI models or improving products by indexing content | Yes | Narrowest way to refuse the training crawler | | \`Meta-ExternalFetcher\` | Fetches an individual link at a user's request | May bypass | Little effect, since it can bypass robots.txt | Two details change what you can expect. Meta says \`Meta-ExternalFetcher\` fetches links when a person asks, which is why it may bypass robots.txt. Meta also says \`facebookexternalhit\` might bypass robots.txt for security or integrity checks, such as checking for malware. So, the table has two crawlers a robots.txt file may not stop, and three it is built to stop. Meta also notes that \`Meta-ExternalAgent\` crawls for "improving products by indexing content directly," not only for training. Blocking it is the choice that matches the training goal, and it may also remove your content from that product-indexing use. ## Each robots.txt choice changes something different You have three realistic setups. Pick the one that matches what you want, copy it, and change nothing else in your file unless you know why. The first setup is the one above: previews stay, training crawler blocked. \`Meta-WebIndexer\` is left alone, so Meta AI search can still cite and link to you. The second setup also blocks Meta AI search indexing. Choose it if you want Meta AI to leave your pages out entirely, and accept that Meta AI will be less likely to cite or link to you. \`\`\`text User-agent: facebookexternalhit Allow: / User-agent: Meta-ExternalAgent Disallow: / User-agent: Meta-WebIndexer Disallow: / \`\`\` The third setup allows everything on purpose. This is a fine choice if you want Meta AI to be able to cite your pages and you are comfortable with the training crawler. Writing it out gives you a record of the decision. \`\`\`text User-agent: Meta-ExternalAgent Allow: / User-agent: Meta-WebIndexer Allow: / User-agent: facebookexternalhit Allow: / \`\`\` Skip a blanket "block all Meta" file. It would disallow \`facebookexternalhit\` too, and your Facebook and Instagram shares can lose their preview cards. If you run Meta ads, a blanket block can also interfere with \`Meta-ExternalAds\` checks of your landing pages. Narrow rules protect the things that bring customers; wide rules cost you some of them. ![Five blank index cards fanned out on a desk beside a pencil.](https://webleveling.com/blog/allow-or-block-meta-crawlers/five-meta-crawlers-five-cards.jpg) Five crawlers, five separate decisions, each on its own card. ## Where the rules go on WordPress, Wix, Squarespace and a custom site The lines are the same everywhere. What changes is how your platform lets you edit robots.txt, and some platforms do not let you edit it at all. Check which situation you are in before you spend time on the syntax. ### WordPress WordPress can generate a virtual robots.txt file instead of storing a real one. You can change the output with a suitable SEO plugin, with a physical \`robots.txt\` file in the site root, or with the WordPress [\`robots\_txt\` filter](https://developer.wordpress.org/reference/hooks/robots_txt/). After you edit, open \`https://yourdomain.com/robots.txt\` and confirm only one version is being served, because a plugin and a physical file can disagree. ### Wix Wix has an official Robots.txt Editor under SEO & GEO, then Tools and settings. Wix's [help page for editing robots.txt](https://support.wix.com/en/article/editing-your-sites-robotstxt-file) walks through it. Add the crawler-specific groups, save, and then read the live file. ### Squarespace Squarespace does not let you edit robots.txt directly. Its Crawlers settings have a "Block known artificial intelligence crawlers" option, and the preset list it adds includes \`Meta-ExternalAgent\`, as described in Squarespace's [AI crawler help page](https://support.squarespace.com/hc/en-us/articles/360022347072-Request-that-AI-models-exclude-your-site). It does not give you the same per-crawler control over the rest of Meta's list. If the toggle covers your goal, turn it on and stop there. ### A custom site Put a UTF-8 plain-text file named \`robots.txt\` at the root of each host, for example \`https://www.example.com/robots.txt\`. Rules apply to the host, protocol and port where the file is served, so \`example.com\` and \`www.example.com\` each need their own file. Google's guide to [creating a robots.txt file](https://developers.google.com/search/docs/crawling-indexing/robots/create-robots-txt) covers the basics. Then confirm the response is a plain text file and not an HTML error page. Our post on [whether your website needs a sitemap and robots.txt](https://webleveling.com/blog/does-your-website-need-a-sitemap-and-robots-txt/) covers the basics of both files. ![A closed laptop on a desk beside a plain manila folder with a blank tab.](https://webleveling.com/blog/allow-or-block-meta-crawlers/robots-txt-edit-at-the-site-root.jpg) The file lives at the root of the host, and the live copy is the one that counts. ## A robots.txt rule is a request, so check the live file and your logs A rule that sits in your file does nothing for you until you confirm three things: the live file says what you think, Meta's crawlers behave, and your previews still work. Start with the live file. Open \`https://yourdomain.com/robots.txt\` in a browser. Find each Meta name you added and read the lines under it. A blank \`Disallow:\` allows everything; \`Disallow: /\` asks that crawler to stay off the whole site. If no named group exists for a crawler, it falls back to the \`User-agent: \*\` group. Then test a share. Post a link to a page in a private Facebook post or message and confirm the title, image and description still appear. If the card is wrong, check \`facebookexternalhit\` first. After that, look at your server logs. Filter for the five names, and record the time, requested URL, status code, IP address and user-agent string. A user-agent is self-declared, so a log line shows what a request claims to be, not proof of who sent it. If you use Cloudflare, [Bot analytics](https://developers.cloudflare.com/bots/bot-analytics/) shows bot traffic and detection sources, and Enterprise Bot Management customers can compare crawler activity in [Business Insights](https://developers.cloudflare.com/bots/business-insights/). Our post on [seeing which AI agents visit your website](https://webleveling.com/blog/see-which-ai-agents-visit-your-website/) covers the log side in more detail, and [Cloudflare's Disallow AI Training setting](https://webleveling.com/blog/cloudflare-disallow-ai-training-googlebot/) is the edge-level counterpart if your site sits behind Cloudflare. ![A magnifying glass resting on a stack of blank paper with faint pencil ticks along one edge.](https://webleveling.com/blog/allow-or-block-meta-crawlers/check-the-log-after-the-change.jpg) The log tells you which crawlers showed up after the rule went live. Keep the limits in mind. Google notes that robots.txt controls crawling, not guaranteed removal from search results, and that a disallowed URL can still appear without a normal snippet; for removal from Google Search you need \`noindex\` or password protection, as Google explains in its [robots.txt introduction](https://developers.google.com/search/docs/crawling-indexing/robots/intro). A new rule is also not retroactive. It asks Meta's compliant crawlers to stay away from now on; it does not pull back content collected earlier. And a crawler visit in your logs does not prove that a model trained on that page. ## Meta's October 30 terms make this a good week to check your file On October 15 to 17, 2026, coverage of Meta's pending Commercial Terms spread through the news. The preview page on Meta's site says the terms are scheduled to take effect October 30, 2026, and it describes Meta, or entities acting for Meta, using automated software to access, index, cache, analyze, collect, retrieve or crawl domains, pages and URLs associated with a business's products or services. The stated purpose is Meta's commercial products, services and features. You can read the preview yourself at [Meta's Commercial Terms preview](https://www.facebook.com/legal/commercial_terms_preview), and the changes to its wording are tracked in [our separate post on the terms](https://webleveling.com/blog/meta-terms-crawl-business-websites/). The terms describe permissions and uses. They do not show that Meta trained a model on a particular business's site. What you can do today is the part in your hands: decide which of the five crawlers you want, and write that decision into robots.txt before October 30. If you want a second pair of eyes, [website maintenance](https://webleveling.com/services/website-maintenance/) is where a file like this gets reviewed over time, since a platform update or plugin change can regenerate robots.txt and quietly undo your lines. ## A one-line rule is a job you can do yourself Adding a \`Meta-ExternalAgent\` group on a platform that exposes robots.txt takes a few minutes and costs nothing. If that describes your site, do it today and skip any package that sells "AI crawler protection" for a rule this small. Paying for help starts to make sense in a few cases: your robots.txt is generated by a plugin and a CDN at once, you run several hosts, your previews break and you cannot tell why, or you want the logs read and the change monitored with a way to roll back. The value is in not breaking previews, ads or search while you make one small change, and in confirming the live result afterward. If you run a hand-coded site, the file is one you write directly and there is no plugin generating a second copy, which removes one of the common ways a rule goes missing. **Quiz: Check your Meta crawler rules** 1. You want Facebook link previews to keep working and you want to refuse Meta's AI training crawler. Which setup fits? - Disallow `facebookexternalhit` and allow `Meta-ExternalAgent` - Allow `facebookexternalhit` and disallow `Meta-ExternalAgent` **(correct answer)** - Disallow every crawler that starts with Meta 2. Which Meta crawler does Meta document as one that may bypass robots.txt because it fetches a link at a user's request? - `Meta-ExternalAgent` - `Meta-ExternalAds` - `Meta-ExternalFetcher` **(correct answer)** 3. You add a `Meta-ExternalAgent` block today. What does it do to content Meta already collected? - It removes that content - Nothing; a new rule is not retroactive **(correct answer)** - It blocks your Facebook previews ## Frequently Asked Questions About how to block meta crawler robots.txt **Which Meta crawler makes link previews?** \`facebookexternalhit\`. Meta describes it as the crawler that fetches a link when it is shared on Facebook, Instagram or Messenger. Blocking it can break the preview card. **Which Meta crawler is the AI training crawler?** Meta documents \`Meta-ExternalAgent\` as crawling for uses such as training foundation AI models or improving products by indexing content directly. Disallowing it is the narrowest way to opt out of that crawler. **Can I block Meta AI training and keep Facebook link previews?** Yes, in the usual setup. Add a \`Meta-ExternalAgent\` group with \`Disallow: /\` and leave \`facebookexternalhit\` allowed. Test a share afterward to confirm the card still appears. **Can robots.txt be ignored, and is it legally enforceable?** Yes, it can be ignored. It is a published request that compliant crawlers follow under RFC 9309, not a lock, and Meta says \`Meta-ExternalFetcher\` may bypass it and \`facebookexternalhit\` might bypass it for security or integrity checks. Anyone can also send a request that claims to be a Meta crawler. Whether ignoring a file creates legal liability is a legal question that depends on the facts, so ask a lawyer if it matters for your site. **Does a new rule remove content Meta already collected?** No. A robots.txt rule applies to future crawling by compliant crawlers. It does not pull back earlier collection. **Can I edit robots.txt on Squarespace?** Not directly. Squarespace offers a "Block known artificial intelligence crawlers" setting whose preset list includes \`Meta-ExternalAgent\`. It does not give you individual control over every Meta crawler. ## What This Means for You You can keep your Facebook and Instagram previews and still refuse Meta's documented AI training crawler. Leave \`facebookexternalhit\` allowed, add a \`Meta-ExternalAgent\` group with \`Disallow: /\`, and decide on \`Meta-WebIndexer\` based on whether you want Meta AI to cite and link to your pages. Then open the live file, test a share, and read your logs. Doing this now settles the question before Meta's terms take effect on October 30, 2026, and the written rule stays in place until you change it. A small, specific file protects your previews and your search visibility while it states your choice. If you would rather hand this off, we at [Web Leveling](https://webleveling.com/) set up and check crawler rules as part of [generative engine optimization](https://webleveling.com/services/generative-engine-optimization/), including live-file checks and preview tests after every change. We work with small and medium businesses across the country and overseas, and you can reach us through our [contact page](https://webleveling.com/contact/) to ask us to review your robots.txt. ## Terms Terms Tap a term to see what it means. **robots.txt.** A plain-text file at the root of a website that tells compliant crawlers which paths they may or may not request. **User-agent.** The name a crawler uses to identify itself, which robots.txt groups match against. **facebookexternalhit.** Meta's crawler that fetches a shared link to build a preview on Facebook, Instagram and Messenger. **Meta-ExternalAgent.** Meta's crawler documented for training foundation AI models and improving products by indexing content. **Meta-WebIndexer.** Meta's crawler that improves Meta AI search results and helps Meta cite and link to content. **Crawler.** Software that requests pages from a website automatically. **Link preview.** The card with a title, image and description that appears when a link is shared on a social platform. - [Rank report shows a drop? Check Search Console first](https://webleveling.com/blog/rank-tracker-numbers-look-wrong/): Keywords missing from your weekly rank report? See what tracker vendors reported on dates from September 24, how to check your own data, and what to ask. - [Search Console Performance Report Delayed: What to Check First](https://webleveling.com/blog/search-console-performance-report-delayed/): Search Console Performance report days behind? Check Google organic sessions and leads in Analytics first, and leave your site alone while Google catches up. - [Does My Website Need a Sitemap? What to Check First](https://webleveling.com/blog/does-your-website-need-a-sitemap-and-robots-txt/): Google says a sitemap helps most on large, new, or media-rich sites. Check yours and robots.txt, then submit a sitemap in Search Console if you have one.