automation - Is automated web site access legal?
2014-07
Many web sites include in their terms of service things about automated access being prohibited. One example is in ebay's robots.txt file:
The use of robots or other automated means to access the eBay site\n without the express permission of eBay is strictly prohibited. Notwithstanding the foregoing, eBay may permit automated access to access certain eBay pages but soley for the limited purpose of including content in publicly available search engines. Any other use of robots or failure to obey the robots exclusion standards set forth at is strictly prohibited.
What does their prohibition of automated access actually mean? Does it make automated access illegal or just frowned upon?
Either online or local applications (Windows XP preferably). I'm specifically looking to be notified by email if the web site in question goes down.
I've tried the Firefox add on, Mr Uptime, but it doesn't seem to have any options for polling frequency. There's also Pingdom who appear to produce this add-on and their online service of the same name.
Does anyone know of anything else which could fit the bill?
You already mentioned pingdom.com
The good thing with pingdom is that they are the only free service I know of that offers a 1 minute check resolution. Their free service is restricted to one site, if you want more checks you'll either have to pay or participate in GIGRIB. All other services I found are either paid or their they check only every 5 or 10 minutes.
EDIT: site24x7.com also offers to check every minute for free, but only from california. pingdom checks from all around the world.
Any of these online services will do the trick:
We use Webmetrics and Keynote is another that has been recommended to me, both online services
On serverfault.com I have written an extensive description of how we monitor our own production webservers here at Paessler. This may be helpful here, too.
I would recommend using WhatsUpGold.
It allows for you to set a polling frequency for each system you want to monitor. It also allows you to specify specific things you are monitoring for. A good example is that this will connect to a webserver via HTTP and confirm it can do a GET command. If the webserver goes down but the IP is still pinging, this would still catch it.
Update Scanner is a firefox add-on that checks websites and allows per site polling interval settings. It's generally meant for checking changes to a page, but being down would be a change I guess. :-)
I use Basic State for my website monitoring. It sends you daily uptime reports and when your site goes down it can send you an email and an SMS message. It's also free.
The only downside is the user interface is not very friendly.
This is a bash
solution that I wrote a couple years ago to see if the server is up:
#! /bin/bash
# check to see if site is up
# if it is, don't worry
#
# v.1 Warren M Myers - initial stab
# 31 Aug 06
#
# 7 is the return code of `curl` if it cannot load the page
ERRCOD='7'
WHEN=`date +%d%b%y`
REPT="/var/tmp/$1.$WHEN.txt"
STARS='********************'
# $1 is the command-line arg for the domain
# you could hardcode this, too
curl -I $1 > /var/tmp/curlret.txt
# $? is the exit code of the last-run script
if [ "$?" = "$ERRCOD" ]; then
# return was unable to connect to host: save ps -aux; mail report
echo $STARS >> $REPT
echo 'curl return results' >> $REPT
echo >> $REPT
cat curlret.txt >> $REPT
echo >> $REPT
echo $STARS >> $REPT
# mail the whole works to myself
mail -s "failed to connect to $1" [email protected] < $REPT
fi
rm -f /var/tmp/curlret.txt
rm -f $REPT
It's self promotion, but I have developed an online tool for this too:
It also provides a REST API if you want to integrate the checking in your own app or site.