Archive | March, 2016

Managing crawl budget for large sites

If you manage a site with millions or even billions of URLs, it’s important to consider that Google and Bing have a crawl budget, a limit in the number of URLs they are prepared to crawl, for every domain, determined by its authority. If a less-authoritative domain has billions of URLs, Google won’t crawl potentially […]

Continue Reading 0