Search Engine Spiders
We all want the search engine spiders (also called “bots” or “spiders”) to rank us highly. Have you ever stopped to think about what they actually are? What functions do these search engine spiders perform?
Scenario
You have created this fantastic website web page. You have examined all your top ranking competitors and you have far more content than they do and yet your ranking does not move. You know that the spiders are visiting you yet no love.
I think that we have all been in this position. Understanding search engine spiders a little better may help us all prepare a little better for the inevitable. The crawling (or not) of your site or blog.
What is a search engine spider?
A search engine spider is simply a program that examines almost every web page on the internet (eventually). These programs are appropriately called spiders because they basically crawl the entire web. The internet is connected via a vast network on addresses and links. A spider can basically start at any address or link and crawl (examine) all connecting pages or connecting websites.
Now keep in mind that there are billions of web pages out there. A search engine spider has to be able to document all of this information and feed it back to the search engines so that this information can be used to provide its ultimate users with valuable information.
Now obviously there is very little if any human intervention anymore. Search engine spiders do all the work. The most important thing to get out of all of this is that…just as people see two seemingly identical website differently, so will the spiders. The trick is understanding how the spiders view your site since it is they that help determine your ranking.
What is it that Search engines and their spider actually do?
1. Crawling - As mentioned above, the bots crawl the internet. Your websites, your blog, internal links, external links you name it. Now they may not crawl every single webpage and the timing of each page may vary but… they do develop a pretty good idea of what your site is about. The names given to the top three search engine spiders are as follows:
a. Google - Googlebot
b. Yahoo - Yahoo! Slurp
c. MSN - Msnbot
2. Indexing - Spiders also index the content that they find. This content is fed to and managed in the search engine’s index database. Just imagine how many documents are in this database!
3. Queries - Anytime a user requests/queries information from the search engine, an examination of millions of items within the index is begun. Using complex algorithms, the most relevant information is obtained and returned to the user.
4. Page Ranking – Most searches these days yield either millions of resulting entries or very few. In an attempt to provide users with the most relevant data, search engines rank their results. It is this ranking that is displayed to information seeking users.
So why do seemingly equal web pages rank differently?
As you have probably already guessed, there are tons of reason why this could happen. If you are certain there are no differences, consider the following:
1. On page factors – Compare your on page factors vs. those of your competitor.
a. Is your website’s name/main keyword contained within your domain name.
b. Are you keywords embedded in your meta title, meta description and meta keyword list?
c. Are your keywords embedded within your heading?
d. Have you used your keyword
enough within your web page. You want
to use your keywords enough so
that the bots know what your web
page is about.
However you
don’t want your page to become spamy. Use your
keywords sparingly within your page. The text
should make
sense. Shoot for one main keyword for every
one hundred twenty
words approx.
e. Check to make sure your meta tags are placed in the head of your s ource code, not the body. Inappropriately place tags could potentially reduce the relevance of your page. Also check to make sure there are no transpositions in your code.
f. Make sure all of your internal links work. This one is serious. Check them all. If the bot can’t crawl your site then you can’t rank for some of your hard work.
2. Off page factors – There are several off page factors which help determine your ranking. Some you can control and some you cannot. For example Domain age. Your domain age does count as part of your ranking. If your domain is new, the only thing that you can do is wait while continuing to build content. Other items to work on include links, your incoming links.
a. Links – The more links you have coming from other websites, the higher you will rank. These links count as votes for your site.
b. Backlinks – Backlinks are usually referred to as those links where your site is linking to another site. Make sure that you check these often. Links to bad sites can negatively affect your ranking also.
3. Your website’s content. – Check for misspellings. Your ranking is based on relevancy. This relevancy is determined in part using Latent Semantic Indexing (LSI). If you have misspellings on your site, search engine spiders may not be able to accurately determine relevancy.
Conclusion:
Don’t worry about the search engine spiders. Rankings changes all the time. Today’s number one may be tomorrow’s number 10. Just keep adding quality content and building links. This will attract visitors. The search engines spiders will notice and rankings will take care of themselves.
