Help the Googlebot understand your web site

A list of recommendations.

Right now I'd like to lock myself in a small padded room, froth at the mouth, and make menacing faces through barred windows at innocent passer-by. But my shrink says I should channel my anger into something productive.

So let's talk about Google.

Google is the best search engine on the 'net right now. The Googlebot is Google's indexing software. The Googlebot visits billions of web sites over time and records their contents, which makes them available to search. The Googlebot is very smart and works really well. But, like everyone, it could use a little help from its friends.

When authoring a web site, keep in mind that the Googlebot is software, which means it has a set of capabilities and limitations and algorithms it uses to index content. There are lots of effective ways to trip up the Googlebot and make it impossible for it to index your content. Alternately, the Googlebot can index your site well, and then people will find it when searching for words it contains.

As a web site author, there are a few simple things you can do to help the Googlebot understand your web site as fully as possible.

Here's a list.**

  • Make every single page on your site accessible via a text-based link - as opposed to Javascript, Flash, DHTML, etc. The Googlebot only speaks text.
    [See #6 in Nine things you can do to make your web site better]
  • Keep the number of links on a given page less than 100.
    [See Google's Webmaster Guidelines]
  • Give every single page on the site a complete and meaningful <title>. Google offers the allintitle syntax, which lets users search only text that appears in a page title. There are over 3 million results returned for Untitled Document.
  • Avoid frames. Avoid frames like the plague.
    [See #3 in Nine things you can do to make your web site better]
  • Use URLs with query strings sparingly, if at all. When using dynamic URLs, like
    http://www.ginastired.com/index.php?howtired=very&currentime=1330&hourssleep=3
    keep in mind that the shorter the list of query string parameters, the better.
    [See #5 in Nine things you can do to make your web site better]
  • Make sure that the title and alt tag attributes exist and are complete and meaningful in each page's markup. For example, the markup for that picture of your goldfish should be something like
    <img src="/imgs/goldie.jpg" alt="my beloved goldfish, Goldie" />
  • Make all relevant information on a page textual. Don't embed page content into images or objects like Flash movies. Did I mention the Googlebot only speaks text?
    [See #6 in Nine things you can do to make your web site better]
  • Make sure your web server supports the If-Modified-Since HTTP header. This feature allows your web server to tell Google whether your content has changed since the Googlebot last crawled your site. Supporting this feature saves you bandwidth and overhead.
    [Verbatim from Google's Webmaster Guidelines]
  • Use robots.txt and meta robots tags to show the Googlebot around your site. These standard mechanisms for directing well-behaved robots like the Googlebot will allow you to specify important things like whether or not Google will cache your page content and/or images, and whether or not the Googlebot will index content on pages that maybe you don't want available to the searching public.

    Webloggers: use the meta tags to help the Googlebot index only your permalinks, not your constantly changing front page. To do this, use
    <meta name="robots" content="noindex,follow" >
    on your front page and
    <meta name="robots" content="index,follow" >
    on your posts' permanent locations.
    [See #7 in Nine things you can do to make your web site better]
  • Use meaningful text inside your tags so the Googlebot can associate that text with that href link. Meaning, if I am going to link my pictures from the war protest, I should say "Take a look at my photos from the war protest" instead of "My war protest pictures are here." Now, Google doesn't explicitly recommend this. But I have a friend named Damion who has a weblog which I link with the word "Damion" on my Bookmarks list. If you do a Google search for the word Damion, this weblog is the third result. So what, you say? Well, Damion doesn't mention his name anywhere on his site.

    So don't use link text like read more or go here or download it or, God help us, click here. Don't click here.

    Webloggers: take heed of this when you display the permanent link for a post. You should link the title of a post which presumably contains words which indicate what the post is about instead of a [+] or the word permalink or, common amongst Blogger users, the date and time.
  • Include a
    <meta name="description" content="[insert your site's description here]">
    tag in your page header to summarize your site; even better, include descriptive text on the site's front page where users can actually read it, like, "Scribbling.net is a self-documentation project, occasionally interrupted by misdirected attempts at explaining the vaguely technical." This text will appear as the description for the site in Google results.
  • Forget <meta name="keywords"> ever existed. Really. It's meaningless.
  • Place more important content higher in the markup than less important content in a page.
  • Don't try to fool the Googlebot with hidden links or duplicate content or irrelevant pages of words like "sex" and "hot girls." The Googlebot doesn't like being played. The Googlebot will make you sorry.

Every few days Scribbling.net is ripe with new content, just waiting and wanting to be indexed and searched. Scribbling.net trembles with anticipation for it's weekly-or-so Googlebot visit, and when the big G arrives, let me tell you, it's like a well-choreographed dance. The Googlebot and Scribbling.net have all the elements of a healthy relationship: love, trust, respect, honesty and understanding. It's beautiful, really. Your site can know this kind of bliss too.

** Disclaimer: I'm no Google expert or employee, and I'm no SEO. As a matter of fact, SEOs who charge exorbitant amounts of money for "proprietary, secret" methods of upping your site's placement in Google results are thieves. There aren't secrets or tricks to any of this; my sources of information for this list of recommendations are the freely available Google Webmaster Guidelines and my own piddly server logs.

peace

39 Comments
  • # yael:
    scribbling.net + googlebot 2gether 4eva!

    thanks, gina! this is great.

    ps: how many people have come to scribbling via a google search for "eminem's dick"?
  • # Joost Schuur:
    I see a slight problem in If-Modified-Since and using most blogging software. It's not uncommon to use the 'repulish all my content' button, be it in Movable Type or other systems. This essentially re-creates every single page, causing the modified date to undoubtedly be changed.

    I have a blog (martiansoil.com) that has 'related entries' built with Kalsey's MTSql suggestion, so I rebuild all the individual archives pretty much every day. I'm downright slammed by Googlebot with a couple of hundred indexes every few days.

    Considering the amount of traffic I get from Google (I have 5-10 updates a day, with meaningful title-tags), I'm not really complaining though.
  • # Adam Rice:
    I don't think it's true that the googlebot can only follow conventional A HREF=... links. In my blog, the archives are only available via a javascript-based menu. I have made back-dated posts that would only be linked to the front page via that menu, and Google still finds them.

    If it looks for anything that seems like a path, it will be able to find those links, though, and it might be doing that.
  • # Andy:
    This is a great article. I tried writing something similar, although less structured, a couple of days ago: The power of semantics and meta. I still haven't quite given up hope on the keywords yet, though -- but that's mostly because they aid MovableType's internal search-engine more than anything.
  • # chris:
    Adam,

    Others (as in, other people on the web?) may have linked to your content using standard hrefs.

    Does your blog engine ping www.weblogs.com?
  • # stone:
    google who
    google you
    google me
    google too
  • # Adam Kalsey:
    Oh the irony... :)

    "Give every single page on the site a complete and meaningful <title>"

    <title>Scribbling.net</title>

    The other advantage of a meaningful title is that it makes it easier for people to link to you when they use their blog tool's bookmarklet. The bookmarklet typically grab's the page's title, so including the entry title in there improves your incoming link quality.
  • # Adam Kalsey:
    Of course I make a pedantic comment about your page and misspell "grabs" when I do it.
  • # Gina:
    Hahaha - You're absolutely right Adam.

    I was waiting to see if anyone would call me on that one. ;)

    It's on the todo list.
  • # Gerben:
    I doubt if Google does anything with the description meta. I use these on all my pages, and Google seems to like my site, but I've never seen my descriptions appear on the Google listings.
  • # EG:
    Adam,

    Your menus are in a javascript downdown, but the URLs are still normal. Whereas on sites like http://www.optionsxpress.com, they use links like javascript:AppendSessionID('welcome_home.htm'); ... those don't make it past Mr. Google
  • # Matt Brubeck:
    Don't overuse alt tags. Most images should probably have alt="". Consider how your page will look or sound to users of text browsers, braille terminals, and screen-readers.

    For example:

    A monkey One time I hired a monkey to take notes for me in class.

    In a non-visual browser, that becomes "A monkey One time I hired a monkey to take notes for me in class." It would be better to have alt="". Use the 'alt' attribute only for replacement text. Use the 'title' attribute for descriptive or additional text.
  • # Matt Brubeck:
    (The example should read: <img src="monkey.png" alt="A monkey"> One time I hired a monkey...)
  • # Kyle Pula:
    Are you saying that Google does not use meta keywords at all to produce its result sets?

    I assume that not much weight is given to these tags but are they really disregarded?

    Do you have a source for this?

    Thanks.
  • # Matt Brubeck:
    According to Google Hacks, "Google sometimes relies on META tags for a description when there's a logt of navigation code that wouldn't make sense to a human searcher."

    Google Hacks also recommends using keywords in the META "description". They say that the META "keywords" tag is ignored by Google, but may be used by other search engines.
  • # Matthias:
    Is'nt it jugded a bit to hard about SEOs?! I think there are different kinds of SEOs. These with the "known secrets" and that which provide their service (knowledge) to help website owners to make their sites more "google-effective". And it may be a lot of work to optimize a website! And why not to get payed for that service?!

    **Disclaimer: I'm not a SEO too! At least no professional... ;)
  • # Anders Jacobsen:
    Good list of stuff! I've also written a couple of entries;

    for example you can set the file names of your archives to be more google friendly:
    http://www.jacobsen.no/anders/blog/archives/2002/08/06/my_movable_type_installation.html
    http://www.jacobsen.no/anders/blog/archives/2002/08/07/optimizing_movable_type_part_2_more_googlejuice.html
    + some info on meta tags:
    http://www.jacobsen.no/anders/blog/archives/2002/08/30/optimizing_movable_type_part_5_meta_tags.html
  • # Muraii:
    In addition to the "<title>Scribbling.net</title>" note above, the linked-to tutorial, "Nine things you can do to make your web site better," betrays your recommendation to use meaningful text in links. For instance, when suggesting validators, you reference "this <a>one</a> and this <a>one</a>".

    For shame.
  • # TimM:
    Muraii is right, you could have gone for a link like this one. Also, adding the title attribute to an image will provide a tooltip to Mozilla users.
  • # Matt Brubeck:
    (Internet Explorer also uses the title attribute as a tooltip, when available.)
  • # Richard Igoe:
    One thing that I don't think you mentioned - and which I have found to be Google's most important page-ranking factor - is the number of high-quality sites linking to yours - and that means sites with content related to yours.
  • # iKOOKi:
    The most important thing - write best content.
  • # WebSolutions1:
    I agree with iKOOKI. Good content, relavant links for relavant users and suitable search results are the best way.

    Thank you for this great web site.
  • # Alex:
    keyword tag should be present to get higher link popularity and to be indexed by a couple dozens of other s.e.
  • # firestarter:
    Hmm

    I think its ok for an SEO to take good money for this! People pays a lot of money for Webdesign - so why they shouldn`t pay good Money for getting good Listings?
  • # Alex:
    Search engine optimization - is like a game with it's own rules; there's need just to learn these rules and success will come gradually . But what's most disappointing about it - finding out all the nuances and details sometimes takes a lot of time - not less than half a year of your own experience. Sometimes it is still common fact that webdesign and s.e. optimization go separately, though to be perfect and to gain the final aim - "high positioning + attracting custimers" they should be present together.
  • # John:
    Webpage design and optimization are the first steps to do, but without proper 3rd step - promotion they will be in vain. Everything is changing rapidly - 6-8 months ago many popular search engines were free for submitting. Now ither they are sold, or turned into "paid only inclusion".
    Gina, we are waiting for your 9_tips_for_proper_free_promotion_for_high_positioning_in_Google for Google has its own relation to promotion.
  • # GoogleBot:
    The Official GoogleBot Fan Listing http://googlebotfan.2ya.com/
  • # Marc:
    Good article. Keep it up!
  • # Bill Brown:
    You may not be aware that Google considers underscores to be part of the word when they're in file names and doesn't treat them as word separators. That means that your file name looks like "helpthegooglebotunderstandyourwebsite" and thus doesn't actually help you (file name is an important part of the PageRank algorithm).

    In case you don't believe me, this came from Google itself:

    http://www.markcarey.com/googleguy-says/archives/discuss-underscores-are-not-word-seperators-in-google.html

    Good article otherwise.
  • # Bill Brown:
    To make that link clickable (sorry, didn't look like you could do a tags):

    GoogleGuy Says
  • # Michael Heraghty -- Search Engine Optimization Consultant:
    You said:
    >>As a matter of fact, SEOs who charge exorbitant amounts of money for "proprietary, secret" methods of upping your site's placement in Google results are thieves. <<

    I couldn't agree more! Unfortunately, there are a lot of charlatans out there who are giving us genuine, committed SEOs a bad name.

    However, people still have a need for Search Engine Optimsation Consultants, as there is a long and detailed learning curve to promoting your site, which most people don't have the time or patience (or desire) for. Most people just want the results, and are willing to pay for good service.
  • # Todd:
    Great article Gina - thanks for putting this together!
  • # Jay Currie:
    Great Article! Google loves blogs and one of the stranger things to do is to set one up, submit it to google and be indexed in a week. Basically by following the directions you are giving here.
  • # Dirk (1-Hit-SEO):
    Agree with Michael; most people have simply no 'time, clue and patience' to read all about the ways to get a higher rank in search engines ! And there are many good un-expensive services around.
  • # About weblog:
    I agree with Bill - use "-" not "_".
  • # Gina:
    Spam upsets me.
  • # Jorge:
    Dear Experts,

    I am about to upload my new website, in replacement of the the good old one. Im in the process of developing it and have a few queestions that i will like to discuss with people that knows about Google.

    1. My new website is in 2 languages English and Spanish.
    2. If the client clicks the link "english" will be forwarded to a webpage that will set a cookie and redirect the client back to the homepage.
    2. If the cookie is present, homepage will be in english, If the cookie is not present homepage will be in spanish (as it was by default).


    The homepage will display information with IF ... THEN clauses, ie.

    IF the cookie is present THEN
    "Hello Guys!"
    ELSE
    "Hola Amigos"
    END IF

    - My questions -

    1. When google indexes my website it will find the language link and will get the cookie, then it will index all the information in english, am I right?

    2. Since it has already indexed the "set-or-remove-cookie-depending-if-its-present-or-not.asp" webpage it will not go again to that page, am i right?

    - My options -

    1. Create 2 domains "spanish.mywebsite.com" and "english.mywebsite.com" pointing to the same folder in my webserver. Keep my original domain "www.mywebsite.com" that will redirect the user to spanish.mywebsite.com if the cookie is not present.

    Ive seen that if my default page is just an automatic redirect google will never page-rank it. (it happens in my actual website) so maybe this is not an option.

    Some advice or ideas will be very apreciated.

    bandolex (AT) hotmail (DOT) com

    Thanks.

    Jorge.-
  • # Jorge.-:
    Forgot to say that I need that Google indexes my website in both languages.

    Jorge.-

This discussion has been closed. Thanks to everyone who participated. See Scribbling.net's comment policy for more information.

Help the Googlebot understand your web site is 1044 words published on June 29, 2003, filed under Web, filed under Technology, filed under Concept, filed under Scribbling.net.


Next: Happy Pride
Previously: Photos from P-town

Browse all writing by date.
 

RSS