Post 998304395

DaveNet: Google upgrades the Web » "I call it a JIT-SE or Just-In-Time-Search-Engine. The JIT-SE feature is particularly suited to weblogs, which are time-oriented websites. The Google crawler notices that I update my site every day, so it knows it should come back and re-index my site every day. It finds lots of links to new stuff. Presumably it indexes those pages too.

They incorporate knowledge of past behavior in the algorithm that determines how often they index sites. We use the same technique in several UserLand Web services, and it's a good heuristic, it offers a reasonable tradeoff between scaling and currency. Beautiful."