Shopify Robots.txt: What to Include

Robots.txt is one of those files that most Shopify merchants never touch — and for good reason. Shopify's default is well-constructed. But understanding what's in it, what you can change, and what you absolutely should not change is worth fifteen minutes of your time. The downside risk of a bad robots.txt edit — accidentally deindexing your entire store — is serious enough to warrant a clear picture of how the file works before you open it.

🔍 What Is robots.txt?

Robots.txt is a plain text file at the root of your domain that tells search engine crawlers which pages or paths they should and should not crawl. It's part of the Robots Exclusion Standard — a protocol followed by Google, Bing, and most reputable crawlers.

The key technical distinction: robots.txt controls crawling, not indexing. A page blocked by robots.txt won't be crawled, but Google can still index it if another page links to it. If you want to prevent indexing, you need a noindex meta tag — not a robots.txt disallow rule. John Mueller has confirmed this at Google Search Central more than once, and it's a point that trips up even experienced developers.

The file format is simple. Each block starts with a User-agent line specifying which crawler it applies to (* means all), followed by Allow and Disallow directives for specific paths. Most robots.txt files end with a Sitemap line pointing to the XML sitemap.

⚙️ Shopify's Default Robots.txt

Your store's robots.txt file lives at https://yourstore.com/robots.txt. Open it in any browser right now — it's public and readable. Shopify's default output looks broadly like this:

User-agent: *
Disallow: /admin
Disallow: /cart
Disallow: /orders
Disallow: /checkouts/
Disallow: /checkout
Disallow: /account
Disallow: /collections/*sort_by*
Disallow: /*/collections/*sort_by*
Disallow: /collections/*+*
Disallow: /collections/*%2B*
Disallow: /collections/*%2b*
Disallow: /*/collections/*+*
Disallow: /*/collections/*%2B*
Disallow: /*/collections/*%2b*
Disallow: /blogs/*+*
Disallow: /blogs/*%2B*
Disallow: /blogs/*%2b*
Disallow: /tools/
Disallow: /search
Allow: /search/predictive

Sitemap: https://yourstore.com/sitemap.xml

This default is good. It blocks functional pages (admin, cart, checkout, account) that have no ranking value and would waste crawl budget, blocks parameter-heavy sorted collection URLs that create near-duplicate pages, and blocks your internal search results. It allows the predictive search endpoint used by theme JS.

For most Shopify stores, the default robots.txt needs no changes at all. Gary Illyes from Google has said that the most common robots.txt mistakes are over-blocking — merchants adding disallow rules that cut off crawlable, rankable content — rather than under-blocking.

💡 What to Allow and What to Block

Always allow (these need to be crawlable)

Always block (these waste crawl budget)

What to add if you have specific needs

Beyond the default, there are a few cases where you might legitimately add custom rules:

• Block a specific product or collection you don't want indexed: Use a noindex meta tag rather than robots.txt — it's more reliable and doesn't affect crawling of other content on the page.
• Block specific bots (e.g. AI training crawlers): Add a separate User-agent: GPTBot block with Disallow: / if you want to opt out of OpenAI's training crawler. This doesn't affect Google.
• Block a staging subdomain: Staging is better handled by password-protecting the staging environment, not robots.txt, since the staging domain's robots.txt would need its own separate configuration.

🛠️ How to Customise Shopify's Robots.txt

Since August 2021, Shopify allows merchants to customise the robots.txt file using a Liquid template. This is done through the theme editor — no app required.

One important note: editing robots.txt.liquid replaces Shopify's default output for that file. If you create the template and leave it blank, your store will have an empty robots.txt — blocking nothing, but also providing no Disallow rules for cart and admin. Always start from Shopify's default content and add to it rather than replacing it entirely.

⚠️ Common Robots.txt Mistakes on Shopify

These mistakes range from minor to catastrophic. For a broader technical SEO picture, our Shopify SEO checklist covers robots.txt alongside the other technical foundations worth auditing regularly.

Disallowing everything with Disallow: /

This is the most catastrophic error — it blocks all crawlers from your entire site. Google will process it and stop crawling your store within days. Traffic typically drops 60–80% over 2–4 weeks as pages fall out of the index. It happens most often when merchants copy a robots.txt "template" from an unrelated platform or accidentally edit the wrong line. If you see a ranking crash after any theme or robots.txt change, check yourstore.com/robots.txt immediately.

Blocking CSS and JS files

Blocking Disallow: *.css or Disallow: *.js was common advice in the early 2010s to speed up crawls. Google now explicitly recommends allowing these files — Googlebot renders pages using the CSS and JS to evaluate page quality. Blocking them degrades Google's ability to understand your pages and can suppress rankings. Shopify's default correctly allows all CSS and JS.

Using robots.txt to hide sensitive content

Robots.txt is a public file — anyone can read it. If you disallow a path, you're telling every crawler (including malicious ones) exactly what's there. For genuinely sensitive paths, use server-level access controls. Robots.txt is not a security layer.

Blocking Googlebot specifically while allowing others

Setting User-agent: Googlebot with aggressive disallow rules while leaving User-agent: * permissive creates an asymmetry that Google's spam policies flag as cloaking — showing different content to Google versus users. Only do this if you have a very specific, legitimate reason and you understand the risk.

❓ Frequently Asked Questions