Shopify Robots.txt: What to Include

Robots.txt is one of those files most Shopify merchants never touch, and for good reason: the robots.txt Shopify generates is well built. But knowing what's in it, what you can change, and what you must not change is worth fifteen minutes.

The short version: the file is at yourstore.com/robots.txt, you edit it through a robots.txt.liquid template, you reset it by deleting that template, and the one edit that can deindex your whole store is Disallow: /. Everything below expands on those four facts.

What Is robots.txt?

Robots.txt is a plain text file at the root of your domain that tells search engine crawlers which paths they may and may not crawl. It's part of the Robots Exclusion Protocol, followed by Google, Bing, and most reputable crawlers. It is advisory: well-behaved bots obey it, scrapers often don't.

The key distinction: robots.txt controls crawling, not indexing. Google's robots.txt introduction says plainly that it is not a mechanism for keeping a page out of Google, and that a disallowed URL can still be indexed if it is linked from elsewhere. To keep a page out of results you need a noindex tag, and the page must stay crawlable so Google can read it.

The format is simple. Each group starts with a User-agent line naming the crawler it applies to (* means all), followed by Allow and Disallow lines for paths. Most files end with a Sitemap line pointing to the XML sitemap.

Five minutes from Google's own team on what the file does and doesn't do:

YouTube · Google Search CentralHow Robots.txt WorksWatch on YouTube (opens in a new tab)

Shopify's Default Robots.txt

Your store's file lives at https://yourstore.com/robots.txt. Open it in any browser; it's public. Shopify's default output is long (it has grown over the years as Shopify adds rules), but the important part looks like this:

User-agent: *
Disallow: /admin
Disallow: /cart
Disallow: /checkout
Disallow: /account
Disallow: /collections/*sort_by*
Disallow: /collections/*+*
Disallow: */collections/*filter*&*filter*
Disallow: /search

Sitemap: https://yourstore.com/sitemap.xml

This default is good. It blocks functional pages (admin, cart, checkout, account) that have no ranking value, blocks sorted and multi-filter collection URLs that create near-duplicate pages, blocks theme previews and internal search, and leaves every product, collection, blog and page crawlable. Shopify's robots.txt help page describes the logic: it hides pages that are private or duplicative.

For most Shopify stores the default needs no changes at all. The usual damage comes from over-blocking: merchants pasting a template from another platform that cuts off crawlable, rankable URLs. If you see /products/ or /collections/ in a Disallow line without a parameter after it, something is wrong.

What to Allow and What to Block

Always allow (these need to be crawlable)

Always block (these waste crawl budget)

Sorted and filtered collection URLs deserve a note. Shopify blocks the sort_by and multi-filter variants, but single-filter URLs stay crawlable and rely on the canonical tag instead. If your filtered pages are showing up in Search Console as duplicates, the fix is in our Shopify canonical tags guide, not in robots.txt.

What to add if you have specific needs

Beyond the default, there are a few cases where you might legitimately add custom rules:

• Hide a specific product or collection from search: use a noindex tag, not robots.txt. A Disallow keeps Google from seeing the noindex, and the URL can still be indexed from links.
• Block specific bots (for example, AI training crawlers): add a separate User-agent: GPTBot group with Disallow: / if you want to opt out of OpenAI's training crawler. This doesn't affect Google. Think twice before blocking AI search crawlers such as OAI-SearchBot or PerplexityBot, which can send shoppers.
• Block a staging store: password-protect the staging store instead. A development store's robots.txt is separate from your live store's anyway.

How to Edit Robots.txt in Shopify (robots.txt.liquid)

Shopify lets you customise the file through a Liquid template in the theme code editor. No app is required, and Shopify's help page notes that Shopify Support won't make these edits for you, so it pays to be careful.

1. Open the theme code editor. In your Shopify admin go to Online Store > Themes, open the ... menu on your live theme and choose Edit code.
2. Create the template. In the Templates folder click Add a new template, select robots, and click Create template. Shopify creates robots.txt.liquid prefilled with Liquid that prints the default rules.
3. Add your rules around the Liquid, not instead of it. Keep the {% for group in robots.default_groups %} loop so you keep Shopify's defaults, including future ones, and add conditions inside it or plain-text lines after it.
4. Save and verify. Open yourstore.com/robots.txt in a browser and compare it with a copy you saved before editing. Then check the robots.txt report in Search Console and request a recrawl of the file.

Copy-paste examples

These come from Shopify's customize robots.txt documentation. Each goes inside the {% for group in robots.default_groups %} loop unless stated.

One important note: creating robots.txt.liquid replaces Shopify's generated output for that file. If you create the template and delete everything in it, your store serves an empty robots.txt: nothing is blocked, including cart and admin. Always keep the default loop and add to it. Shopify also warns the rendered template doesn't always match the generated file exactly, which is why you save a copy of the live file first.

How to Remove or Reset Shopify's Robots.txt

A common search is "how to remove robots.txt from Shopify". You can't. Every Shopify store serves one, and you wouldn't want to delete it: without it, crawlers would spend their budget on cart, checkout and sorted-collection URLs.

What people usually mean is one of three things, and each has a clean fix:

After a reset, open yourstore.com/robots.txt to confirm the default is back, then request a recrawl in Search Console's robots.txt report so Google picks up the change without waiting for its next scheduled fetch.

Common Robots.txt Mistakes on Shopify

These range from minor to catastrophic. For a broader technical SEO picture, our Shopify SEO checklist covers robots.txt alongside the other technical foundations worth auditing regularly.

Disallowing everything with Disallow: /

This is the worst case: it blocks all crawlers from your entire site. Google stops crawling, and pages fall out of the index as they are re-evaluated. Shopify's help page puts it bluntly: incorrect use of the feature can result in loss of all traffic.

It happens most often when merchants copy a robots.txt "template" from an unrelated platform or edit the wrong line. If your store stops showing on Google after any theme or robots.txt change, check yourstore.com/robots.txt before anything else.

Blocking CSS and JS files

Blocking *.css or *.js was common advice in the early 2010s to speed up crawls. Google now renders pages with those files to evaluate them, and its JavaScript SEO basics state that it won't render JavaScript from blocked files. Blocking them degrades Google's understanding of your pages. Shopify's default correctly leaves /cdn/ assets crawlable.

Using robots.txt to hide sensitive content

Robots.txt is a public file; anyone can read it. If you disallow a path, you're telling every crawler, including malicious ones, exactly where it is. For genuinely sensitive pages, use password protection. Robots.txt is not a security layer.

Blocking Googlebot specifically while allowing others

A User-agent: Googlebot group with aggressive Disallow rules, while User-agent: * stays permissive, is almost always a mistake copied from somewhere else. Google only follows the most specific group that matches it, so those rules override everything in the * group. Unless you have a precise, deliberate reason, keep one * group and treat Googlebot like every other crawler.

Expecting robots.txt to fix indexing problems

If products are missing from Google, robots.txt is the first thing to rule out but rarely the cause. More often it's a missing or unsubmitted sitemap, a password-protected store, or thin pages Google chose not to index. Our guide on why Shopify SEO isn't working walks through the diagnosis in order.