What is gmlink.txt and how to use it for SEO?
Gmlink.txt is a file that you can upload to your website’s root directory to tell Google and other search engines how to crawl and index your site. It is similar to robots.txt, but it has some additional features and benefits that can help you improve your SEO performance.
In this article, we will explain what gmlink.txt is, how it works, and how to create and use it for your website. We will also show you some examples of gmlink.txt files and how they can help you optimize your site for search engines.
What is gmlink.txt?
Gmlink.txt is a text file that contains instructions for Google and other search engines on how to crawl and index your website. It is an extension of the robots.txt protocol, which is a standard way of communicating with web crawlers.
Gmlink.txt allows you to specify more details and preferences for your site’s crawling and indexing, such as:
- Which pages or sections of your site you want to allow or disallow for crawling
- How often you want your site to be crawled
- How much bandwidth you want to allocate for crawling
- Which sitemap files you want to submit to Google
- Which canonical URLs you want to use for your pages
- Which alternate versions of your pages you want to link to (such as mobile, AMP, or multilingual versions)
- Which parameters or filters you want to exclude from crawling (such as session IDs, sorting options, or pagination)
- Which meta tags or headers you want to use for your pages (such as noindex, nofollow, or noarchive)
Gmlink.txt can help you improve your SEO performance by:
- Preventing duplicate content issues by telling Google which version of your page is the preferred one
- Increasing your crawl budget by avoiding unnecessary crawling of low-value or irrelevant pages
- Enhancing your site’s visibility by submitting your sitemap files and linking to your alternate versions
- Controlling your site’s indexing by using meta tags or headers to instruct Google how to treat your pages
How does gmlink.txt work?
Gmlink.txt works by following a simple syntax and logic. The file consists of one or more records, each starting with a user-agent line that specifies which web crawler the record applies to. The user-agent line is followed by one or more directives that tell the crawler what to do with the pages or sections of your site. Each directive consists of a field name and a value, separated by a colon. The field name indicates the type of instruction, and the value indicates the target or parameter of the instruction.
For example, here is a simple gmlink.txt file that tells Googlebot (Google’s web crawler) not to crawl any pages on your site that have the word “secret” in their URL:
User-agent: Googlebot Disallow: /*secret*
The user-agent line specifies that this record applies to Googlebot. The disallow directive tells Googlebot not to crawl any pages that match the pattern /*secret*, which means any page that has the word “secret” anywhere in its URL.
You can also use wildcards (*) and end-of-URL markers ($) to create more complex patterns. For example, here is a gmlink.txt file that tells Googlebot not to crawl any pages on your site that have the word “secret” at the end of their URL:
User-agent: Googlebot Disallow: /*secret$
The $ symbol indicates the end of the URL, so this pattern matches any page that has the word “secret” as the last part of its URL.
You can also use multiple user-agent lines and directives in one record, or create multiple records for different web crawlers. For example, here is a gmlink.txt file that tells Googlebot not to crawl any pages on your site that have the word “secret” in their URL, but allows Bingbot (Bing’s web crawler) to crawl all pages on your site:
User-agent: Googlebot
Disallow: /*secret*User-agent: Bingbot