If You’re fighting the preparing approach, a framework that I like to recommend is the START Scheduling Process (complete disclosure: I developed it). For example, you could see which web pages and folders are crawled by far the most. Which might enable determine when you’re throwing away crawl funds (Google https://bowsalon.com