In This Chapter

• Understanding the Google crawl

• Preparing your site for the Google spider

• Avoiding Google’s wrath

• Keeping Google (and other search engines) out

From the inception of the Web, it has been the goal of every person with even the most modest Web page to attract visitors. Advertising, reciprocal linking, word of mouth, getting listed in directories — all have been tools in the mad scramble for traffic. In the Google era, getting into the index has become the single most important task of Webmasters large and small.

Search engine listing has always been crucial. Page owners have spent hours submitting requests to innumerable directories and engines. Coding a page in a way that attracts a search engine crawler and puts the page high on the search results list became a crafty art form in the late 1990s. Google has become so dominant in the search field that if your page can’t be Googled, it might as well not exist — that’s today’s presumption.

Getting your site into the Google index requires patience and networking skill, but it’s not hard. Improving your position in the index — how high your site places on search result lists — is another matter. Old coding tricks don’t work in Google, which means bad news and good news. The bad news is that there are no shortcuts to prominence in Google. The good news is that the index is utterly democratic, affording any Web site, large or small, a chance to gain good positioning based on merit. In this way Google is different than, and superior to, other search engines.

This chapter covers how Google crawls the Web, how a new page can get into the index, and how a new or established site can improve its position in Google’s search results.