Google dispels myths about crawling budget and website loading speed



The most important thing about the crawling budget
  • Google has to choose what the search engine crawls in order to use resources wisely.
  • The crawler crawls news sites more often because they update more often. Accordingly, the bot scans less frequently sites that are updated less frequently.
  • Google keeps a fingerprint of the content to track when it was last updated and tries to determine if this site requires more crawling.
  • To track updates on sites, Google analyzes structured data and date elements.
  • Etag and Last-Modified headers can help the crawler track the frequency of site updates, but in general this is more of a clue for Google, so it can ignore them. If you change the date on the page and do not change the content, the site will not be crawled more often.
  • Only large online sites with a million URLs should worry about the crawling budget: first of all, these are e-commerce and publishers.
  • If the site is not indexed, owners may think about exceeding the crawling budget, but more often than not the problem is not crawling, but low quality content.
  • It is also important to ensure that the servers run smoothly and do not give error codes.
  • If your site has a page that does not have a URL in the sitemap, Google may not immediately find it. First, the crawler will need to go to another page containing a link to the new URL.
  • Important changes need to be made gradually. Do not change the server, content, URL, domain at the same time.
  • Google does not cache POST requests - it quickly wastes the crawling budget. It is worth using GET requests.
  • If your site has pages that you don't need to crawl, they need to be blocked using robots.txt.
  • Nothing can be done to get Google to crawl your site more often. If a crawler finds out that the page has a lot of high-quality content, and there are many URLs in the sitemap, it will crawl as much as it can and will continue to do the same in the future. As a result, this can lead to the fact that the crawling budget will increase to the values that are required by the site owner or webmaster.

Key points about website loading speed
  • Loading speed is not as important a ranking factor as many SEOs think. Pages with more relevant content will rank higher than pages with faster loading speeds.
  • There are many regions in the world with slow internet connections. This must be borne in mind when developing websites.
  • On average, a web page usually weighs several megabytes. The recommended size is about 500 kilobytes.
  • There are several problematic points in the tactics of SEO specialists: for example, many people think that images of the minimum size have a positive effect on loading speed, while rejecting "lazy" loading.
  • No need to get hung up on reports and recommendations from services like Lighthouse. Changes implemented as recommended by such a tool do not always result in an immediate acceleration of downloads.
  • Different types of devices load sites at different speeds. Therefore, you should not think that the site is fast if it opens quickly on a smartphone of the latest model. You need to keep track of which devices your users open the site from, and improve the speed for them.
  • AMP is not a ranking factor.
  • AMP is important in the same way that ranking speed is important.
  • If there are two pages with approximately the same content, the page with the faster loading speed will appear higher in the search results.