Crawlers (or bots) are used to assemble information available on the internet. By utilizing website navigation menus, and studying inner and external hyperlinks, the bots begin to grasp the context of a web page. Of course, the words, photographs, and different data on pages also help search engines