Increase Your Google Crawl Budget

Written by: Jason Bayless | July 24, 2017

First of all, it is crucial to clarify that not every site has to deal with the crawl budget. For websites that have more pages, the crawl budget may be a problem that requires some additional planning. For the most part, sites with fewer than a few thousand pages are typically not impacted by crawl budget problems.

The crawl budget allocated to a site depends on several factors. If the response time is very fast, the crawl speed will be higher, conversely, if the server is slow, Google will slow down the crawl speed. Google itself optimizes the rate at which it will crawl a site according to its response time.

However, from the “”Site Settings”” section in the Search Console, webmasters can ask Google to decrease its maximum crawl speed. They can also configure the maximum scan speed acceptable by their server. The response time of the site is also a factor. The faster the server will respond, the higher the volume of pages that the Googlebot can crawl simultaneously.

The volume of new pages to crawl is important. A site that frequently has new pages added will require more crawl budget than a static site that has not changed for several months. The volume of popular pages to crawl is a factor as well. Google tends to crawl popular pages much more frequently than the less popular pages so that they are as up to date as possible in the possible index.

Frequently webmasters are worried that the Googlebot will not crawl the site as often as it should, and that this will hurt their ranking. Google has confirmed multiple times that the crawl budget is not linked to the ranking, however a lot of webmasters still worry that the a crawl budget increase for a site will equal a higher ranking. Google has stated several times before that the indexing frequency is not related to ranking in search results.

That said, boosting the crawl budget may result in a few pages on very large sites that have not yet been indexed can end up being indexed. That can help to position keywords and generate traffic. For bulky sites, a poorly managed crawl budget can make a difference between your pages being indexed or not being indexed at all. This is the reason for why it may be rather valuable for bulky sites to make sure they get as large a budget as they are able.

A crawl budget is determined by the health of the crawl. The health of the crawl is the way the server responds to a Googlebot crawl and frequently by the highest rate that the site owner gives for the Googlebot via the console. The crawl speed limit is a rate that the Googlebot utilizes when it crawls the site.

However, you’ll find there isn’t a way to make sure that the Googlebot will scan all of the pages of a site. It doesn’t matter if you’re lucky enough to have a server that responds quickly. The crawl request concerns the volume of URLs that the Googlebot plans to crawl. They base that on how frequently they need to be scanned, which URLs to crawl, and how URLs that are less frequently consulted are treated.

If a server doesn’t have any errors and is quick to respond, it may have a better crawl budget when compared with a site that displays server errors or is slow to reply. Remember that it is not always the speed of the server that is the most important. Your CMS may be old and obsolete, and this may result in a slower display of your pages. Your database may also be responding slowly.