Donklephant » Blog Archive » New White House Website Open To Search Engines

New White House Website Open To Search Engines

By Justin Gardner | Related entries in Barack, Bush, Technology

Web guru Jason Kottke points out that the Bush administration had almost 2,400 different rules governing what search engines could crawl on the site and report in their results. This was most likely done so that no caches could be created to later compare with any changes that were made.

But seriously, most of them aren’t necessary and some of them are just plain odd.

For instance, the White House didn’t want search engines crawling pages about Bush’s dog Barney?

Disallow: /barney/barney/text
Disallow: /barney/barneycam/text
Disallow: /barney/newmedia/text
Disallow: /barney/photoessay/beazley/text
Disallow: /barney/photoessay/text
Disallow: /barney/photoessay2/text
Disallow: /barney/text
Disallow: /barneycam/text

Or how about the pages about Easter?

Disallow: /easter/2002/eggsbystate/text
Disallow: /easter/2002/text
Disallow: /easter/2003/eggsbystate/text
Disallow: /easter/2003/text
Disallow: /easter/2004/eggsbystate/text
Disallow: /easter/2004/media/text
Disallow: /easter/2004/text
Disallow: /easter/2005/eggsbystate/text
Disallow: /easter/2005/photoessay/text
Disallow: /easter/2005/text
Disallow: /easter/2005/video/text
Disallow: /easter/2006/eggsbystate/text
Disallow: /easter/2006/photoessay/text
Disallow: /easter/2006/text
Disallow: /easter/2006/video/text
Disallow: /easter/2007/eggsbystate/text
Disallow: /easter/2007/photoessay/easter-2007/text
Disallow: /easter/2007/photoessay/photoessays/text
Disallow: /easter/2007/photoessay/text
Disallow: /easter/2007/text
Disallow: /easter/2007/video/text
Disallow: /easter/2008/eastervideo/text
Disallow: /easter/2008/eggsbystate/text
Disallow: /easter/2008/photoessay/preparing/text
Disallow: /easter/2008/photoessay/text
Disallow: /easter/2008/text
Disallow: /easter/2008/video/text
Disallow: /easter/states/text
Disallow: /easter/text

On the other hand, Obama’s new site has one rule for search engines…

Disallow: /includes/

Perhaps this will change, but it’s an encouraging sign nonetheless.

This entry was posted on Wednesday, January 21st, 2009 and is filed under Barack, Bush, Technology. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

2 Responses to “New White House Website Open To Search Engines”

Togakangaroo Says:
January 21st, 2009 at 9:53 am
Don’t need no gall-darn robot spiders spolin’ our Easter party.

And Barney? He just eats them like they’re candy, then who has to stay up all night with him crying of a stomach ache? Bushy J thas who!

You try fostering world peace on two hours of sleep.
Craig McGill: writings » Blog Archive » New White House 2.0 site is good PR Says:
January 22nd, 2009 at 12:46 am
[...] thing, for others it may be a stunt (for example, there’s already a debate going on about the robots.txt file) but it’s another symbol of change being brought in at all levels and another example of how [...]

Name (required)

Mail (will not be published) (required)

Website

NOTE TO COMMENTERS:

You must ALWAYS fill in the two word CAPTCHA below to submit a comment. And if this is your first time commenting on Donklephant, it will be held in a moderation queue for approval. Please don't resubmit the same comment a couple times. We'll get around to moderating it soon enough.

Also, sometimes even if you've commented before, it may still get placed in a moderation queue and/or sent to the spam folder. If it's just in moderation queue, it'll be published, but it may be deleted if it lands in the spam folder. My apologies if this happens but there are some keywords that push it into the spam folder.

One last note, we will not tolerate comments that disparage people based on age, sex, handicap, race, color, sexual orientation, national origin or ancestry. We reserve the right to delete these comments and ban the people who make them from ever commenting here again.

Thanks for understanding and have a pleasurable commenting experience.