CAPTCHAFORUM
Administrator
The most important thing about the crawling budget
- Google has to choose what the search engine crawls in order to use resources wisely.
- The crawler crawls news sites more often because they update more often. Accordingly, the bot scans less frequently sites that are updated less frequently.
- Google keeps a fingerprint of the content to track when it was last updated and tries to determine if this site requires more crawling.
- To track updates on sites, Google analyzes structured data and date elements.
- Etag and Last-Modified headers can help the crawler track the frequency of site updates, but in general this is more of a clue for Google, so it can ignore them. If you change the date on the page and do not change the content, the site will not be crawled more often.
- Only large online sites with a million URLs should worry about the crawling budget: first of all, these are e-commerce and publishers.
- If the site is not indexed, owners may think about exceeding the crawling budget, but more often than not the problem is not crawling, but low quality content.
- It is also important to ensure that the servers run smoothly and do not give error codes.
- If your site has a page that does not have a URL in the sitemap, Google may not immediately find it. First, the crawler will need to go to another page containing a link to the new URL.
- Important changes need to be made gradually. Do not change the server, content, URL, domain at the same time.
- Google does not cache POST requests - it quickly wastes the crawling budget. It is worth using GET requests.
- If your site has pages that you don't need to crawl, they need to be blocked using robots.txt.
- Nothing can be done to get Google to crawl your site more often. If a crawler finds out that the page has a lot of high-quality content, and there are many URLs in the sitemap, it will crawl as much as it can and will continue to do the same in the future. As a result, this can lead to the fact that the crawling budget will increase to the values that are required by the site owner or webmaster.
Key points about website loading speed
- Loading speed is not as important a ranking factor as many SEOs think. Pages with more relevant content will rank higher than pages with faster loading speeds.
- There are many regions in the world with slow internet connections. This must be borne in mind when developing websites.
- On average, a web page usually weighs several megabytes. The recommended size is about 500 kilobytes.
- There are several problematic points in the tactics of SEO specialists: for example, many people think that images of the minimum size have a positive effect on loading speed, while rejecting "lazy" loading.
- No need to get hung up on reports and recommendations from services like Lighthouse. Changes implemented as recommended by such a tool do not always result in an immediate acceleration of downloads.
- Different types of devices load sites at different speeds. Therefore, you should not think that the site is fast if it opens quickly on a smartphone of the latest model. You need to keep track of which devices your users open the site from, and improve the speed for them.
- AMP is not a ranking factor.
- AMP is important in the same way that ranking speed is important.
- If there are two pages with approximately the same content, the page with the faster loading speed will appear higher in the search results.
