robots.txt: Control what search engines are allowed to crawl

Your online store is visited by far more than just your customers. Google, Bing, price comparison services, AI bots, and a wide variety of other programs automatically crawl your pages. The robots.txt file is like a little note you hang on the door for them: it tells them which parts of the store they’re welcome to look at—and which ones they should avoid.

Shoporama automatically creates and maintains your robots.txt file, so you don’t need to do anything to get off to a good start. If you want extra control, you can add your own lines. Here’s a complete guide: what the file does, what’s in it by default, how to edit it, and what mistakes to avoid.

What is robots.txt?

robots.txt is a plain text file located in the root directory of your domain. You can view your own by typing /robots.txt after your store’s address in the browser—for example , https://dinshop.dk/robots.txt. Try it as you read along; it’ll make it easier to follow the examples below.

The file contains lines in a very simple format. A robot reads the file before it starts fetching pages and follows the instructions it finds there. The most important terms in the file are:

• User-agent: who the rules apply to. An asterisk means “all robots.”
• Disallow: a path the robot must not crawl.
• Allow: an exception—that is, a path the robot is allowed to crawl even if a broader scope is blocked.
• Sitemap: a reference to your sitemap, which is the list of all the pages you want to be found.

It’s important to understand one thing right away: robots.txt is a polite request, not a lock. The file is public, and anyone can read it. The major search engines respect it, but it doesn’t prevent anyone from opening a page manually. Therefore, never include anything in robots.txt that you don’t want strangers to see, and never use it to protect sensitive content.

Here’s what your robots.txt looks like by default

If you haven’t made any changes, your robots.txt in Shoporama looks like this, with your own domain in the Sitemap line:

User-agent: *
Sitemap: https://dinshop.dk/sitemap.xml
Disallow: /basket$
Disallow: /address
Disallow: /payment
Disallow: /shipping
Disallow: /approve
Disallow: /thanks
Disallow: /order
Disallow: /download-file
Disallow: /product_review

# Powered by Shoporama

In plain English, it says: "Dear all bots. Here is the list of my pages. Please don’t mess around with the shopping cart, the address form, the shipping options, the payment page, the approval page, the receipt page, the order pages, file downloads, or product reviews.”

There are two good reasons why these specific pages are blocked. First, they make no sense in a search result. No one benefits from landing directly in an empty shopping cart from Google. Second, robots spend a limited amount of time on your store—what’s known as a crawl budget. Every time a bot fetches a page that shouldn’t appear in search results, that’s time it isn’t spending on your products and categories. In other words: these settings help ensure your actual sales pages are discovered faster.

The “Sitemap” line points to your sitemap, which Shoporama generates automatically based on your products, categories, pages, landing pages, and blog posts. You don’t need to maintain it yourself, and you don’t need to add it manually to robots.txt.

You cannot delete the lines that Shoporama adds itself. This is intentional. These are precisely the kinds of mistakes that are most costly to fix, and these standard rules apply to virtually all online stores.

How to Edit Your robots.txt

1. Go to Account, Web Store, robots.txt in the menu on the left side of the admin panel.
2. You’ll see the Robots.txt page with the “Customize robots.txt” box.
3. Enter your additional lines in the “Additional Rules” field. One rule per line.
4. Click Save.
5. Open https://dinshop.dk/robots.txt in your browser and check that the result looks as expected.

The change takes effect immediately. There’s no waiting period and no cache to clear. At the bottom of the page in the admin panel, there’s a small help box with examples of rules you can add.

Note that the field is called “Additional Rules” for a reason: what you enter is added on top of Shoporama’s own lines, not in place of them. Your lines are inserted right after `User-agent: *` and before the `Sitemap` line and the default rules.

This field applies to one online store at a time. If you have multiple stores in the same account, you’ll need to set this up in each individual store. However, the content will be carried over if a store is duplicated.

Rules That Make Sense for an Online Store

Most online stores never need to enter anything in this field. The default setup works well. However, there are some situations where adding an extra line is a good idea.

Internal pages no one should find on Google

If you have pages that are for internal use only, a temporary test page, or a campaign page that should only be accessible via a direct link from your newsletter, you can exclude them:

Disallow: /internal-page/
Disallow: /temp/
Disallow: /newsletter-only/

A path without a trailing slash matches everything that starts with it. So, ` Disallow: /temp ` excludes `/temp`, `/temp/test`, and `/temperature`. Be careful where you place the slash.

Filtered and Sorted Views

If your theme has filters or sorting options that create many variations of the same category page, it may be worth excluding those variations so that the crawlers focus their time on the category itself. Whether this is relevant depends entirely on how your theme is built, so check your own URLs first, or read more about filtering on your online store.

A specific exception

If you want to grant access to something that’s otherwise located under a blocked path, you can use `Allow`. A more specific rule takes precedence over a less specific one, regardless of their order:

Disallow: /internal-page/
Allow: /internal-page/guide

What you should NOT block

• Your products and categories. These are what you rely on to be found.
• Images, CSS, and JavaScript. Google renders your pages just like a real browser. If you block the design files, Google will see a broken page and rank it accordingly—especially on mobile.
• Your sitemap. The robots must be able to crawl it.

The difference between robots.txt and noindex

This is the point where most people go wrong, so it’s worth reading twice. The two terms sound similar but do something completely different:

• robots.txt says: “You may not crawl this page.”
• noindex says: “You may crawl the page, but you must not display it in search results.”

The important implication: if you block a page in robots.txt, Google can’t crawl it, and therefore Google can’t see that it has a “noindex” tag. If the page already has external links, Google might still show the URL in search results—just without a title or description. The result is an empty, ugly line in Google instead of nothing at all.

Rule of thumb: If you want a page out of Google, use `noindex` and don’t block it in `robots.txt`. If you just want to save the crawlers from wasting time on something that’s never going to appear in search results anyway, use `robots.txt`.

In Shoporama, you set `noindex` by checking the “Enable ‘noindex, follow’” checkbox. This option is available for products, categories, pages, landing pages, and blog posts. Read the full guide in “Set noindex on a page, product, or category.”

Shoporama also automatically sets “noindex” on pages where it’s appropriate, including the cart, address, shipping, payment, receipt page, and search results page. All newer themes include this. If you have an older or custom-built theme, it’s worth having your theme developer check that the meta tag is actually included.

Common Mistakes

• Writing ` Disallow: /`. That single slash blocks the entire store from search engines. It’s the most costly typo in the entire field. If your traffic from Google disappears over the course of a few weeks, this is the first place you should look.
• Blocking a page to get it out of Google. It doesn’t work. Use `noindex`, as described above.
• Entering a User-agent line in the middle of the field. Your lines will be placed in the group that applies to all robots. If you start a new group in the middle of the field, Shoporama’s default rules will end up in the new group instead, and then they suddenly no longer apply to all robots. If you need rules for a specific robot, please contact us at support@shoporama.dk first.
• Using robots.txt for security. The file is public. If you write ` Disallow: /secret-folder/`, you’ve just told the whole world that the folder exists.
• Blocking design files. Without CSS and JavaScript, Google will consider your page broken.
• Forgetting to check afterward. Always open /robots.txt in your browser after you’ve saved it. It takes five seconds.
• Thinking it takes effect immediately in Google. The file updates instantly, but it can take days or weeks before Google has crawled it again and indexed your entire store.
• Uppercase and lowercase letters. The words “Disallow” and “Allow” themselves are case-insensitive, but the paths are not. /Tilbud and /tilbud are two different things.

Demo stores and shoporama.dk addresses

If your store is still a demo store, is closed, or is located at a temporary address under shoporama.dk, Shoporama automatically generates a robots.txt file that says:

User-agent: *
Disallow: /

# Powered by Shoporama

This is intentional, and it works to your advantage. A store under construction shouldn’t appear in Google, and your temporary address shouldn’t end up competing with your actual domain later on. In this situation, the “Extra Rules” field has no effect, since the entire file is overwritten.

As soon as the store goes live on your own domain, the file automatically switches to the normal version with a sitemap and standard rules. If you haven’t moved to your own domain yet, read “Move Your Store to Your Own Domain.” If you’re moving an existing store to Shoporama, “Avoid Losing Google Rankings When Moving ” and “The Comprehensive Guide to Redirects ” are the two articles you should read first.

For the technically inclined

• The file is generated on every request and served as text/plain with UTF-8 encoding. It is not cached by Shoporama, so any changes are visible immediately.
• The order in the final file is: User-agent: *, followed by your content from the “Extra Rules” field, then the Sitemap line, and finally Shoporama’s default rules.
• The default rule `Disallow: /basket$ ` uses the dollar sign as the "end point." This means it only affects the shopping cart page itself, not URLs that start with the same string.
• Google and Bing resolve conflicts between “Allow” and “Disallow” by giving priority to the most specific rule, not by the order in the file. If rules are equally specific, “Allow” takes precedence.
• The sitemap is located at /sitemap.xml and is automatically split into multiple files with an index if the store has a very large number of URLs.
• This field can also be read and written via Shoporama’s API along with the other online store settings if you want to manage it from your own tools.

Frequently Asked Questions

I’ve never touched my robots.txt. Is my store set up incorrectly?

No. Shoporama creates the file for you, and the default content works for the vast majority of online stores. The “Extra Rules” field is for cases where you have something very specific that you want to keep out of search engines. If you’re unsure, just leave it blank.

How do I know if my change has taken effect?

Open https://dinshop.dk/robots.txt in your browser and refresh the page. The change will be visible immediately, since the file isn’t cached. If you’re still seeing the old version, it’s your browser’s own cache—try a hard refresh or opening a private window.

Google is still showing a page I’ve blocked. What should I do?

That’s the classic pitfall. A block in robots.txt prevents Google from crawling the page, not from displaying the URL. Remove the block in the “Extra Rules” field, set the page to “noindex” instead, and wait for Google to crawl it again. Then it will disappear on its own.

Can I create rules that apply only to one specific search engine?

You shouldn’t do that in this field, because your lines will be added to the group that applies to all robots. If you write your own User-agent line, Shoporama’s default rules will be moved over to the new group. If you have a specific need, please email support@shoporama.dk, and we’ll find a solution together.

I have multiple online stores. Do I need to do the same thing in all of them?

Yes, the field applies to each individual online store, so you’ll need to go into each store separately. If you create a new store by copying an existing one, the contents of the field will be carried over to the copy, so you won’t have to do anything there.

Can I prevent order and receipt pages from appearing in Google?

They’re already excluded by default. The cart, address, shipping, payment, and receipt pages have a “Disallow” line, and Shoporama also sets “noindex” on them. The order pages have a “Disallow” line. You don’t need to do anything. Just remember that robots.txt is never, by itself, a form of data protection. The file is public, and anyone can read it.

Could I end up ruining my traffic from Google with this field?

Yes, if you write ` Disallow: /`. That tells all search engines to stay away from the entire store, and over the course of a few weeks, your visibility will drop. Therefore: always check /robots.txt in your browser after saving to make sure there isn’t a single slash in it.

Our store is for business customers only and requires a login. Should we block everything?

No, that’s rarely necessary. The robots can’t crawl content behind a login anyway, and blocking the entire store also means that the homepage and any public information pages will disappear from Google. If you want to completely remove the shop from search results, please contact us so we can assess the right solution for your setup.

Can I remove the lines that Shoporama automatically adds?

No, they are always added. They cover the shopping cart, checkout, order pages, file downloads, and product reviews—in other words, pages that should never appear in search results. If you have a very specific reason for wanting one of these pages to be crawled, please email support@shoporama.dk.

Do I need to add my sitemap to the field myself?

No. Shoporama automatically inserts a sitemap line that points to /sitemap.xml on your own domain, and the sitemap is kept up to date for you. If you add a line yourself, you’ll simply end up with the same sitemap listed twice.

If you have questions about a specific rule, or if you’d like us to review your robots.txt file, please feel free to email us at support@shoporama.dk.