automation - Is automated web site access legal?

08
2014-07
  • cs.

    Many web sites include in their terms of service things about automated access being prohibited. One example is in ebay's robots.txt file:

    The use of robots or other automated means to access the eBay site\n without the express permission of eBay is strictly prohibited. Notwithstanding the foregoing, eBay may permit automated access to access certain eBay pages but soley for the limited purpose of including content in publicly available search engines. Any other use of robots or failure to obey the robots exclusion standards set forth at is strictly prohibited.

    What does their prohibition of automated access actually mean? Does it make automated access illegal or just frowned upon?

  • Answers
    Know someone who can answer? Share a link to this question via email, Google+, Twitter, or Facebook.

    Related Question

    automation - Are there any tools to periodically check if a web site is available?
  • Umber Ferrule

    Either online or local applications (Windows XP preferably). I'm specifically looking to be notified by email if the web site in question goes down.

    I've tried the Firefox add on, Mr Uptime, but it doesn't seem to have any options for polling frequency. There's also Pingdom who appear to produce this add-on and their online service of the same name.

    Does anyone know of anything else which could fit the bill?


  • Related Answers
  • Ludwig Weinzierl

    You already mentioned pingdom.com

    The good thing with pingdom is that they are the only free service I know of that offers a 1 minute check resolution. Their free service is restricted to one site, if you want more checks you'll either have to pay or participate in GIGRIB. All other services I found are either paid or their they check only every 5 or 10 minutes.

    EDIT: site24x7.com also offers to check every minute for free, but only from california. pingdom checks from all around the world.

  • arathorn

    Any of these online services will do the trick:

  • Swish

    We use Webmetrics and Keynote is another that has been recommended to me, both online services

  • Dirk Paessler

    On serverfault.com I have written an extensive description of how we monitor our own production webservers here at Paessler. This may be helpful here, too.

    http://serverfault.com/questions/71441/what-is-the-best-way-to-monitor-a-production-server/72731#72731

  • 8088

    I would recommend using WhatsUpGold.

    It allows for you to set a polling frequency for each system you want to monitor. It also allows you to specify specific things you are monitoring for. A good example is that this will connect to a webserver via HTTP and confirm it can do a GET command. If the webserver goes down but the IP is still pinging, this would still catch it.

    alt text

  • GollyJer

    Update Scanner is a firefox add-on that checks websites and allows per site polling interval settings. It's generally meant for checking changes to a page, but being down would be a change I guess. :-)

  • Paul Sheldrake

    I use Basic State for my website monitoring. It sends you daily uptime reports and when your site goes down it can send you an email and an SMS message. It's also free.

    The only downside is the user interface is not very friendly.

  • warren

    This is a bash solution that I wrote a couple years ago to see if the server is up:

    #! /bin/bash
    # check to see if site is up
    #   if it is, don't worry
    #
    # v.1 Warren M Myers - initial stab
    #     31 Aug 06
    #
    
    # 7 is the return code of `curl` if it cannot load the page
    ERRCOD='7'
    WHEN=`date +%d%b%y`
    REPT="/var/tmp/$1.$WHEN.txt"
    STARS='********************'
    
    # $1 is the command-line arg for the domain
    # you could hardcode this, too
    curl -I $1 > /var/tmp/curlret.txt
    
    # $? is the exit code of the last-run script
    if [ "$?" = "$ERRCOD" ]; then
        # return was unable to connect to host: save ps -aux; mail report
        echo $STARS >> $REPT
        echo 'curl return results' >> $REPT
        echo >> $REPT
        cat curlret.txt >> $REPT
        echo >> $REPT
        echo $STARS >> $REPT
        # mail the whole works to myself
        mail -s "failed to connect to $1" [email protected] < $REPT
    fi
    
    rm -f /var/tmp/curlret.txt
    rm -f $REPT
    
  • fjyaniez

    It's self promotion, but I have developed an online tool for this too:

    Port Monitor

    It also provides a REST API if you want to integrate the checking in your own app or site.