Monitoring Tools for your Website/Server

By Scott | Published: November 22, 2015

sitemonitoring_snapshotDo you have a website hosted on a server somewhere – either in your home or in the cloud? Has it ever gone down and you didn’t know about it for hours? I have, and it has happened to me too many times.

As a result, I have often wanted/needed a way to verify that the server which is hosting my websites is up and running properly. While many fine graphical tools exist such as Munin for this purpose, I wanted a simple set of utilities that could return just the parameters I needed and how I needed them.

What’s Needed

There are two different ways in which I want to get my metrics. The first is a command-line utility that I can run from my local computer to test the parameter needed. The second is a set of python functions that deliver the same metrics that I can then use to create a custom webpage.

This webpage (a future project) would run on a local server such as a Raspberry Pi, which could then display all my metrics on a tablet display in a convenient location. A quick glance could then show that everything was running smoothly or that problems have arisen.

Many other possibilities exist as well – sending an SMS message when a site goes down, lighting up LEDs to indicate site performance, etc.

The various metrics needed to maintain server and website health can be grouped into two categories:

  1. Server performance – these metrics indicate the health of the server itself. Metrics such as cpu load and memory usage are most desired.
  2. Website performance – these metrics indicate the health of the webserver application as well as the websites themselves. Metrics include page load speed, site verification, and number of requests per hour.

The Solution

In order to meet the goals cited, I decided to create a custom site monitoring library. This library is a collection of python scripts that provide monitoring tools for servers and websites. These scripts can be:

  1. Run from the command line on a local computer in order to get the needed information any time it is needed without having to login to the server. An added benefit is once the library is downloaded to the local computer, it can be used to monitor any server without needing any further installation on that server.
  2. Individual functions in each file can be loaded and called from custom python applications to provide customizable output. I mentioned some possibilities already. I hope to implement some of those in a future project.

Each of these utilities require a host to be specified. The host can be either a named host in your ~/.ssh/config file, or it can be specified as USER@DOMAIN/IP. Server utilities can also be called to monitor your local computer. In this case, use either local or localhost for the host specification.

Note: if you do not have a computer running Linux or Mac, you may have difficulty using them due to the lack of SSH support in Windows. In that case, I suggest using either a Raspberry Pi to run them on, or upgrade your computer to a more modern operating system.

Getting Started

Installation

Download the library from github, or use git to clone the repo via:

$ git clone https://github.com/provideyourown/SiteMonitoring.git

To execute these utilities as standalone scripts, make them executable via:

$ chmod +x FILENAME.py

You can also execute the standalone scripts repeatedly with a simple bash function (add to your ~/.bashrc or .bash_aliases file):

repeat() { INT=$1; shift; while true; do $*; sleep $INT; done; }

Usage, e.g. repeat systemload every 15 secs:

repeat 15 ./systemload.py local

The repeat function can be very handy to monitor changes in server loads, memory usage or any other parameter provided in this library.

Now let’s examine the utilities themselves. They are grouped into Server utilities and Website utilities.

Server Utilities

System Load

This utility shows the system load as a percentage over a period of time.

Python Funcs

getCpuLoad(server, interval)

Given the host definition for the server, and an interval (in secs), getCpuLoad returns the average cpu load over that interval as a percentage.

Standalone usage:

$ ./systemload.py HOST [-i 5.0]

-i optionally specifies the interval. The default is 1.0 seconds.

Prints the cpu load as a percentage. It is really handy to determine when your desktop or laptop is in need of a reboot.

Memory Usage

This utility shows the memory used out of total available and the percentage of swap memory used (if any).

Python Funcs

getMemoryUsage(server)

Given the host definition for the server, return the memory used, total and swap. Used is given as a percentage, total memory in Megabytes, and swap is in percentage.

Standalone usage:

$ ./memoryusage.py HOST

Prints the memory used as a percentage of total available, the total available, and the percentage of swap memory used.

Website Utilities

Site Test

This utility is a wonderful way of making sure your website is up and healthy. It indicates if a site is up, verifies the site’s page is valid, and provides the total page load time.

Python Funcs

getSite(url, testStr)

Tests to see if the site is up, is valid and its loading time.

Parameters:

  • url: can be just the site’s domain name or can optionally include the protocol (http/https)
  • testStr: a string within the page to verify the page loaded correctly. If this string is not found on the page returned, the site is considered invalid.

Returns:

isUp (bool), isCorrect (bool), load-time (secs – float)

Standalone usage:

$ ./sitetest.py [http://]example.com 'this is an example site'

Prints up status, verifies integrity (is valid), and provides the page load time in seconds.

Get Requests

This utility counts the number of requests to a site over a period of time. It is really useful for determining the load on your webserver at any given time.

Python Funcs

countRequests(server,  logfile,  interval)

Counts the requests to this site over a period of minutes, with a maximum of 1440 mins (60 mins * 24 hours).

Parameters:

  • server: host name as described in the beginning of the article
  • logfile: full path name for the access.log file (can be either apache or nginx standard log file)
  • interval: time interval in minutes over which to count requests

Returns: number of requests (int)

Standalone usage:

$ ./getrequests.py MYSERVER /var/www/MYSITE/logs/access.log [-i 20]

The time interval is specified by -i in minutes, and is optional. If not specified, the utility will return the number of requests over three periods of time: 10 mins, 1 hour, 24 hours (1 day).

Conclusion

Server and website monitoring is a big challenge. Many tools exist, but they often are either difficult to use or provide way more information than what is needed. These simple, open-source, python utilities will meet the most common and pressing monitoring needs. Since they have been kept simple and are written in python, you can easily customize them or write additional ones, using them for a starting point.

If you have useful comments to contribute, please do so in the comments. Any code/feature suggestions or bugs to report should be made on the github pages.

Monitoring Tools for your Website/Server by Provide Your Own is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

This entry was posted in Tech and tagged , , , . Section: . Bookmark the permalink. Both comments and trackbacks are currently closed.

One Trackback

  • By Get Your Website In 24 Hrs | 7Wins.eu on November 28, 2015 at 8:52 am

    […] How to Get Your Website Indexed by Search Engines in 24 Hours Or Less! | PlusCaptcha Homepage24-Hour House, In-Kind Sponsors Recognized | HabiJax How To Get On The First Page Of Google Within 24 Hours Free – Blogging Tips and Tricks Why Buick’s Free 24 Hour Test Drive Might Cost You Bargaining Power – Game Theory Tuesdays | Mind Your DecisionsChecking Your Resource UsageRock online reviews with one widget! – localbiz360Monitoring Tools for your Website/Server […]