Jason Kottke pointed out that the whitehouse.gov robots.txt file was changed almost immediately after the inauguration …
It went from this …
User-agent: * Disallow: /cgi-bin Disallow: /search Disallow: /query.html Disallow: /omb/search Disallow: /omb/query.html Disallow: /expectmore/search Disallow: /expectmore/query.html Disallow: /results/search Disallow: /results/query.html Disallow: /earmarks/search Disallow: /earmarks/query.html Disallow: /help Disallow: /360pics/text Disallow: /911/911day/text Disallow: /911/heroes/text
… to this …
User-agent: * Disallow: /includes/
Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol.
One small step towards the new attitude of openness and transparency.
Wow! Thats interesting that someone even bothered to look at this point. (I’d have left the query/search results probably)
(Wait a minute. I’ll be back..)
Now I’m not sure if those directories exist any more. I wonder if they just dropped in a new server (they obviously changed a lot).
And it’s tripled in size now!
Disallow: /includes/
Disallow: /search/
Disallow: /omb/search/