Searchme's Spider (Charlotte) Help

Who are you and why are you accessing my site?

Charlotte is a spider created by Searchme, Inc. in Mountain View, CA, a startup that is indexing the web for sites to include in its search engine index.

We are not attempting to steal any copyrighted information from your site and will not be re-distributing your content. We will only be allowing users to find your website more easily.

top

You are requesting pages too fast, what can I do?

If you feel we are causing an undue strain on your server or bandwidth, please send an email to charlotte@searchme.com.

Please include:

  • The domain name(s) of the servers in question.
  • What exactly is happening.
  • A few lines from your webserver logs showing the activity, if possible.
We will respond within 24 hours.

top

Why are you asking for robots.txt?

The robots.txt file is an industry-standard method to control our spiders access to your website. Charlotte requests this file to see if there are any special instructions for her with regard to how to crawl or where not to crawl on your website. More information on robots.txt can be found at: www.robotstxt.org

top

How can I control Charlotte’s access to my site?

The easiest way to prevent Charlotte from accessing your site is via the robots.txt file, which should be located at the root of your website, ie: http://www.mysite.com/robots.txt. If you add the following lines, requests to your website will soon stop: User-Agent: Charlotte
Disallow: /

It may take up to an hour for this change to take effect. A common mistake people make is to deny access via robots.txt and also deny via another method, such as blocking the IP from which the requests are coming. This ends up blocking the spider’s access to robots.txt, and when robots.txt can't be retreived, the spider continues requesting other pages. It’s best to wait for roughly an hour for the robots.txt changes to take effect before attempting the more drastic measure of IP-based blocking.

top