A Search Console message mentions your sitemap. Or an SEO quote lists "robots.txt review" as a line item, and you are not sure whether to pay for it, ignore it or panic. It is a fair thing to wonder at night, with a laptop open and nobody to ask. The question underneath is simple: does my website need a sitemap, and is anything on my site blocking Google? You can answer both yourself in about two minutes, without logging into your hosting or touching a single setting. Google has a clear position on which sites benefit most from a sitemap, and your platform may have already made one for you. What follows tells you what to check first, what each file does, and when a file is not yours to edit.
Key Takeaways
Google says a sitemap is most useful for large sites, new sites with few links pointing to them, sites with rich media and news sites, while a small, well-linked site may work without one.
Open yoursite.com/robots.txt and yoursite.com/sitemap.xml in a browser; both should show readable text, not an error or a login page.
A sitemap lists pages to help Google find them, and robots.txt tells crawlers which URLs they may request.
To keep a page out of search results, use noindex, a password or removal, because a URL blocked in robots.txt can still appear without a description.
WordPress, Wix, Squarespace and Shopify each create a sitemap automatically, so check your platform's settings before editing anything by hand.
Google says a sitemap helps most on large, new and media-heavy sites
Google's sitemap overview describes a sitemap as a file that gives information about the pages, videos, images and other files on your site and how they relate. It helps search engines crawl a site more efficiently. Google says a sitemap is especially useful if your site is large, if it is new and has few external links pointing to it, if it has rich media, or if it appears in Google News.
For smaller sites, Google defines "small" as about 500 pages or fewer, counting only the pages you want showing up in search. A site that size, with pages that link to each other clearly, may not need a sitemap. Google also says that in most cases your site will benefit from having one. So, the useful question is not whether a sitemap is required. It is whether yours exists, loads and lists the pages you care about.
Two limits are worth keeping in mind as you read on. A sitemap is a hint that helps Google find your URLs, and Google decides which ones to crawl and index. The same goes for the idea that a sitemap raises your rankings; it supplies information such as update dates, and the ranking decisions stay with Google. A brand-new site with a handful of links pointing to it gets the most from a sitemap, because Google has fewer other paths to reach its pages.
You can check both files tonight in about two minutes
Start with the free route, because it answers most of the worry without costing anything. You only need a browser, and nothing here changes your site.
- Open your robots.txt: Type your domain followed by `/robots.txt`, for example `https://yourdomain.com/robots.txt`. You should see plain text, not a 404 page, a login screen or a server error.
- Look for a Sitemap line: Many robots.txt files include a line starting with `Sitemap:` that gives the full address of the sitemap.
- Open the sitemap: Try that address, or `https://yourdomain.com/sitemap.xml`. WordPress itself uses `/wp-sitemap.xml`, and some platforms use a sitemap index that points to several smaller sitemaps.
- Confirm the domain: The addresses inside should show your real, live domain, not a staging or test address.
- Scan the rules: In robots.txt, a line that reads `Disallow: /` under a general rule for all crawlers tells them to stay off the whole site. If you see that on a live site, flag it to whoever manages your site right away.
Google lists several mistakes that block more than intended in its guidance on troubleshooting crawling errors. They include a sitewide `Disallow: /`, blocking the CSS or JavaScript a page needs to display, blocking the sitemap itself, and blocking a page that carries a noindex tag. Write down what you see rather than fixing anything yet. A file that loads and lists the right pages needs no change.
An empty robots.txt, or no robots.txt at all, is fine for a site that may be crawled entirely. Google's robots.txt introduction says so directly, so a missing file alone is no reason to hire anyone.
A sitemap lists pages, and robots.txt sets which URLs crawlers may request
The two files get mentioned together, which makes them sound like one thing. They have separate jobs, and the fix for a problem depends on which file it lives in.
| Sitemap | robots.txt | |
|---|---|---|
| What it is | A file listing pages and related files you want Google to find | A file telling crawlers which URLs they may request |
| Typical address | /sitemap.xml, or /wp-sitemap.xml on WordPress | /robots.txt |
| Main job | Help Google find your pages and see update dates | Manage crawler traffic and skip unimportant or similar resources |
| Effect on indexing | A hint; Google decides what to crawl and index | Controls crawling; it does not reliably remove a page from results |
| If missing | A well-linked small site may still be found | A site that may be crawled entirely can use an empty file or none |
Google's robots.txt guidance says the file exists mainly to manage crawler traffic and to avoid requests for unimportant or similar resources. It is a set of requests that well-behaved crawlers follow, standardized as the Robots Exclusion Protocol in RFC 9309. It is not a lock or a security feature, so a private page needs a password, not a robots.txt rule.
Google also suggests leaving robots.txt out of routine decisions about crawl budget. For an ordinary small business site, the platform default covers what you need.
Your platform probably already made both files
If your site sits on WordPress, Wix, Squarespace or Shopify, the platform builds the sitemap for you and updates it when pages change. That is why the check above often ends with both files loading. Each platform's own help pages give the right address and say who is meant to edit what.
WordPress
WordPress core creates XML sitemaps by default at `/wp-sitemap.xml`, as the WordPress core team announced, and serves a robots.txt response that points to the sitemap index. A site that uses an SEO plugin may have a different sitemap address, so look at the plugin's settings before assuming something is missing. WordPress.com sites get an automatically generated sitemap as well, described on the WordPress.com sitemaps page, commonly at `/sitemap.xml`.
Wix
Wix automatically creates and updates a sitemap index at `/sitemap.xml`, as its sitemap help article explains, and completing the SEO Setup Checklist can submit it to Google. Wix also serves a robots.txt file and updates it after relevant site changes. Its robots.txt article treats editing the file as an advanced feature, which is a good signal to leave it alone unless you have a specific reason.
Squarespace
Squarespace creates and updates `/sitemap.xml` on its own, and you can view it through the steps in its site map help article. The sitemap itself cannot be edited by hand, and page visibility settings control what appears in it. For anything involving robots.txt, check your current Squarespace settings or ask the person who built your site before assuming you can change it.
Shopify
Shopify creates `/sitemap.xml` automatically and updates it when products, pages, collections or posts change, as its site map help page describes. It serves a preconfigured robots.txt file, and editing it through the `robots.txt.liquid` template is a customization Shopify's robots.txt guide says belongs with someone with the relevant technical skill.
Submit your sitemap in Search Console once you have confirmed it loads
If the sitemap loads and lists the right pages, tell Google where it is. Submitting is a one-time step for most sites, and it gives you a report to watch afterward. The Search Console steps below follow Google's Sitemaps report help page.
- Pick the right property: In Google Search Console, select the property for your live domain. If you have not set one up, the post on Google Search Console setup for a small business walks through it.
- Open Sitemaps: Choose Indexing, then Sitemaps.
- Add the path: In the Add a new sitemap box, enter the path, such as `sitemap.xml`.
- Submit: Select Submit.
- Read the report: Check the status, the last read date, the number of discovered URLs and any parsing errors.
The report lists only sitemaps submitted there. Google may find others on its own, so a sitemap missing from the list is not proof that Google has never seen it. If a sitemap shows an error, the first question is whether the affected URLs matter to your business, whether they load, and whether you want them in search at all. A redirected page, a private page or a duplicate in the list is a cleanup item, not an emergency.
For the other side of the file question, Search Console includes a robots.txt report where it is available for your property, and it shows whether Google can process the public file. To test one page, use URL Inspection and look at whether the page is blocked by robots.txt. Tutorials that show a separate robots.txt Tester tool describe an older screen, so follow the menus you actually see.
Keep a page out of results with noindex, a password or removal
This is where the two files get confused most often, and where a wrong move costs the most. People reach for robots.txt when they want a page gone from Google. It tells crawlers not to request the URL, and Google says a blocked URL can still appear in results if it learns about the page through links or other references, usually without a description.
The tools that work for exclusion are different. Google's guide to blocking indexing points to a `noindex` instruction on the page, password protection, or removing the page. Your website editor or SEO plugin normally has a setting for noindex on each page.
One detail changes how you apply it. Google has to crawl a page to see the noindex instruction on it, so a robots.txt rule that blocks the page stops noindex from working. Google also does not support a noindex rule written inside robots.txt itself. If a page must stay out of results, allow crawling, add the noindex, and wait for Google to read the page again.
These setup problems are the ones worth fixing first
When something is wrong, it is more often a small setup issue than a missing file. The order below starts with the mistakes that can cost you the most.
| Problem | What to do | Typical job size |
|---|---|---|
| Checking the wrong domain, a staging address or the HTTP version | Select the live property and resubmit the right sitemap | Small administrative task |
| A valid sitemap that was never submitted | Submit it in Search Console and watch the status | A few minutes |
| Sitemap lists redirected, broken, private or duplicate URLs | Remove them through your platform or SEO settings | Small to medium |
| Sitewide `Disallow: /` after a redesign | Find the rule, remove or narrow it, then inspect key pages | Small if a setting caused it, medium if code is involved |
| Robots.txt used when the goal was noindex | Allow crawling and add noindex in the page settings | Small per page |
| A plugin or custom edit changed the platform default | Restore the default and test the public file | Medium |
The fixes in the table are file and settings work. A page can load fine, appear in the sitemap and still not show up in search because of weak internal links, a different canonical URL or content that does not help the searcher. That is a different problem, and the post on why a page is not indexed in Search Console covers it. If you just published something new, getting a new page found by Google covers the next steps.
A redesign or a move to a new platform deserves its own care. Old addresses, new addresses and the sitemap all change at once, and moving your website without losing Google rankings covers what to carry over. If you want a wider view of where your site stands, the website check you can do in an afternoon puts these two files next to everything else worth a look.
Bring in help when the files are healthy but pages still are not showing
You can check whether the files exist in a few minutes, and for many sites that is the end of the job. Help earns its fee when the answer is not obvious: a redesign or migration is involved, an important page stays blocked, redirected or absent from the index, or a platform setting is beyond what you can safely change.
A good technical reviewer shows you the exact problem, names which pages matter, says whether your platform already manages the files, and separates a file fix from broader SEO work. What technical SEO includes lays out where file checks sit among the rest. If the files turn out healthy, the useful move may be to leave them alone and put the effort into your content, your internal links and the pages your customers read.
Check what you know about sitemaps and robots.txt
Pick an answer to begin.
1. What does robots.txt control?
2. You want a page kept out of search results. What does Google point to?
3. Your site runs on Shopify, Wix or Squarespace. What is the sensible first step for the sitemap?
Frequently Asked Questions About does my website need a sitemap
Does a small website need a sitemap?
Google says a well-linked site of about 500 pages or fewer, counting pages meant for search, may not need one. It also says that in most cases a site benefits from having one, and your platform may create it for you.
Does submitting a sitemap guarantee my pages get indexed?
No. A sitemap helps Google find your URLs, and Google decides which ones to crawl and index.
What is robots.txt?
A text file at yourdomain.com/robots.txt that tells crawlers which URLs they may request. Google describes its main uses as managing crawler traffic and skipping unimportant or similar resources.
How do I submit my sitemap to Google?
In Search Console, choose your property, open Indexing, then Sitemaps, enter the path such as sitemap.xml in Add a new sitemap, and select Submit.
Can I put noindex in robots.txt?
No. Google does not support a noindex rule in robots.txt. Put the noindex on the page itself and let Google crawl the page so it can read it.
Should I edit the robots.txt my platform created?
Only for a specific reason you understand. Platform defaults cover ordinary sites, and Wix and Shopify both treat editing the file as an advanced task.
Wrapping Up
A small, well-linked site may work without a sitemap, and Google still says that in most cases a site benefits from one. The check is quick: open yourdomain.com/robots.txt and yourdomain.com/sitemap.xml, confirm both load from your live domain, and look for a sitewide Disallow rule. A sitemap helps Google find pages, robots.txt sets which URLs crawlers may request, and neither one is the tool for keeping a page out of results.
With those two addresses checked and your sitemap submitted in Search Console, you will know where you stand before the next SEO quote arrives. You can read a Search Console message calmly, tell a normal platform setting from a real block, and spend your budget on the pages that bring in customers.
If the files look healthy and your pages still are not getting found, or you want a redesign or migration handled without losing your place in search, Web Leveling can look at it with you. Our search engine optimization work covers the technical checks alongside the content and links around them, and the free checks above come first. We work with small and medium businesses across the country and overseas. Contact us about your sitemap and robots.txt questions, and we will help you read what your site is telling Google.
Terms
Sitemap and robots.txt words in this post
Tap a term to see what it means.
Sitemap. A file that lists the pages, images, videos and other files on a site so search engines can find them more easily.
robots.txt. A text file at the root of a site that tells crawlers which URLs they may request.
Crawling. A search engine requesting and reading the pages of a site.
Indexing. A search engine storing a page so it can show it in results.
noindex. An instruction on a page that asks search engines to keep it out of their results.
Search Console. Google's free tool for seeing how your site appears in Google Search and for submitting a sitemap.
Sitemap index. A sitemap that points to several smaller sitemaps, which some platforms use.
