|
Search engines like Yahoo! and Google "spider" the site, which is to say they read and index everything on the front page and then read and index everything that the front page links to, and then everything that every other page links to, and so on.
In any given week Yahoo! and Google read and index the entire site several times. because they have found that doing so doesn't slow down the connection, and because they aren't smart enough not to read old threads. Little things like the cookies, the date stamps, etc. change and this tricks them into believing the whole site has changed.
A "slurp spider" is one process at Yahoo! reading. When there are a hundred of them, they are reading the entire site again.
|