Crawlers (or bots) are used to gather info available on the web. By utilizing web site navigation menus, and studying internal and external links, the bots begin to grasp the context of a page. Of course, the words, photographs, and other knowledge on pages also help search engines like google and yahoo understand what they’re about. Follow these