Bing Introduces Improved Robots.txt Testing Tool
Errors in the Robots.txt file can prevent search bots from correctly indexing the site, which in the future may affect the ranking results and the amount of organic traffic. The document contains information on what content search engine crawlers can crawl. This is one of the few ways to control search engines.
Errors in Robots.txt can also lead to negative results. For example, the Ryanair website was closed from search engines in 2020. The reason was the directive forbidding of all bots from crawling the site. Therefore, no webmaster is immune to errors when composing or making changes to a text file.
To help users, the search engine Bing announced an improved tester for Robots.txt that allowed the identification of the main errors in the document. According to official representatives, the use of the service became possible after the transition to a new, more advanced platform that provides new functionality. The tool shows how the directives are drawn up for all crawlers and whether the name of the crawler is spelled correctly.
Bing Tester
Search engine officials said they have upgraded the service to test the Robots.txt file. It is better at identifying the correctness of the directives and identifies any problems that interfere with the effective crawling and indexing of the site.
Even if you have not blocked some URLs or the entire site from crawling by robots, sometimes changes can be made to the file without your knowledge and permission. Robots.txt can be accessed by the developer, site owner, or agency staff where you work. Therefore, you need to constantly monitor the updates in the document, which may appear there by mistake. Otherwise, the site may lose its position in the search results or become available for indexing unwanted content.
What Changes Has Bing Made to the Robots.txt Tester?
The new service will be useful for SEO specialists to check the Robots.txt document for errors. It also lets you see which URL is blocked for which Bing search agent. This will allow you to identify content that is closed from indexing by mistake. In the future, this may have a positive effect on the ranking results.
The improved service is ideal for preparing an SEO audit. You will be able to show the client the existing problems in the Robots.txt file. Based on the information received, you will be able to make recommendations, indicate which directives are written incorrectly, and see what content is better to close so that bots do not scan duplicate pages or 404 errors.
The algorithm for working with the tool is simple. All you need to do is copy and paste the URL into the corresponding line of the tester. The system will show whether Bingbot and BingAdsBot and other Bing agents are allowed access to it. A list of all available crawlers is located in the special window on the right.
Blogs Bing
The process of scanning a file is similar to the work of the Bingbot and BingAdsBot crawlers, according to the developers of the tool on the official blog of the Bing search engine. You can find more details here.
The system checks for bots directives and displays robots.txt in the editor in four options:
- http: //
- https: //
- http: // www
- https: // www
The tester shows all variants of the file corresponding to versions of the site with and without a security certificate in the results.
You can make changes to the text version of the document and you can also download it to check for updates offline. This will save you time when putting together SEO-friendly Robots.txt.
The developers have added the latest Fetch option that gives webmasters access to previous versions of the file. If changes have been made to the file without your knowledge, you can return the latest Robots.txt and use it to configure access for search robots.
This is how the function looks in practice:
Blogs Bing
The system displays the entire step-by-step process of updating the document, including uploading the edited Robots.txt to the root of the site’s domain and the corrections made after that. If no new directives have been added to the document, the tool will only show the original version. This is a very useful feature when you start working on a new and existing project because the webmaster can see all of the actions of other specialists. You can analyze how to improve Robots.txt to get higher positions in the site rankings.
Bing webmasters say they are willing to listen to their users’ comments and upgrade the Robots.txt tester if necessary in the future. You can report possible errors or share your opinion in the official Twitter account or write a letter to the search engine technical support.
It is difficult to say at this point how good Bing’s improved Robots.txt tester is in practice because it became available to webmasters in early September. However, judging by the description of the main functions, the service will greatly facilitate the work with the ranking of sites in the search engine.
The tool allows you to adjust the directives for Bingbot and to see how other Bing crawlers are crawling the site. You can make the necessary changes right in the editor, return the previous version if necessary. You can also load the final data into the Robots.txt text file. This makes it much easier for novice SEO specialists to work with the document. There will always be an opportunity to promptly correct the information in order to exclude possible negative results for SEO. Read the related article – Robots.txt: How to Create the Perfect File for SEO.
