Shopify robots.txt tester

What does robots.txt do on Shopify?

robots.txt tells crawlers which URLs they may fetch. It does not decide what Google indexes. Shopify generates the file for every store at /robots.txt. The default blocks checkout, cart, account and admin pages, plus sorted and filtered collection URLs that would waste crawl budget, and leaves products, collections, pages and blog posts open. For most stores the default is right and needs no changes.

A page blocked in robots.txt can still appear in Google if other sites link to it, just without a description. Google says robots.txt is not a mechanism for keeping a web page out of Google. To keep a page out of results, leave it crawlable and add a noindex robots meta tag. A noindex line inside robots.txt does nothing: Google stopped supporting it on September 1, 2019.

How crawlers read the rules

This tester follows RFC 9309, the robots.txt standard, and Google's robots.txt documentation:

• One group per crawler. A crawler uses the group whose User-agent line names it most specifically, matched without case. Groups that name the same crawler are merged. If none names it, it uses the User-agent: * group, and the two are never combined.
• The longest matching rule wins. Length is counted on the rule's path, wildcards included. If an Allow and a Disallow are the same length, Allow wins.
• Wildcards. * matches any characters and $ marks the end of the URL. /*.php$ blocks /index.php but not /index.php?id=1.
• Paths are case-sensitive and compared after percent-encoding, so /Shoes and /shoes are different URLs.
• An empty Disallow: blocks nothing, and /robots.txt itself is always allowed.
• Ignored lines. Comments after #, unknown fields, and Crawl-delay (Google ignores it, though Bing uses it). Sitemap: lines apply to every crawler, wherever they appear.

How to edit or reset robots.txt in Shopify

Shopify builds robots.txt from the robots.txt.liquid theme template. To change it:

1. In your Shopify admin, go to Online Store and then Themes.
2. Next to your live theme, open the menu and click Edit code.
3. Click Add a new template, choose robots, then Create template.
4. Add your rules around the default groups, save, and test the result here before you rely on it.

To go back to Shopify's default, delete robots.txt.liquid from the Templates folder. Shopify's guide Editing robots.txt.liquid warns that a wrong edit can cost you all your traffic, and the robots.txt.liquid developer docs show how to keep Shopify's default groups so new rules Shopify adds still reach your store. Keep the defaults and add to them rather than pasting a plain text file.

Mistakes this tester catches

• Disallow: / for every crawler or for Googlebot, often left over from a store that was password protected or in development.
• Blocking /collections, /products or /blogs, which stops Google from crawling the pages that sell.
• Blocking /cdn/ or .css and .js files, so Google can't render the page the way shoppers see it.
• Blocking AI search crawlers like OAI-SearchBot, PerplexityBot and Claude-SearchBot, which send shoppers from ChatGPT, Perplexity and Claude answers. Blocking training crawlers like GPTBot or CCBot is a separate choice and doesn't remove you from those answers.
• A missing Sitemap: line, and rules Google ignores, like noindex and Crawl-delay.

Related guides

For what belongs in the file, read Shopify robots.txt: what to include. To make sure Google finds every product, see how to submit your Shopify sitemap. If the store still isn't in search results after the file checks out, work through why your Shopify store isn't showing on Google, and check duplicate URLs with the Shopify canonical tags guide. To check how a page will look in results once it is crawled, use the meta title and description checker.