Monitor your production systems and application analytics using Graphite. This article will help you setup these tools on Ubuntu 14.04 on a Nginx webserver with PostgreSQL as backend.
What is what
What is Graphite?
Graphite is an open source software that is used for monitoring any system. The monitored data is the numeric information that represents any performance metric. Graphite also as a renderer which renders this information as graphs. Official Documentation
What is Nginx?
NGINX is a very fast Webserver, its faster than most web servers available in the market. The biggest advantage of Nginx is its concurrency (because of asynchronous nature). It can also act as
- HTTP Cache
- Reverse Proxy
- Load Balancer
For more information visit Nginx Wiki
Graphite Ubuntu Package Installation
Install Graphite packages
NOTE: During the installation, you will be asked if during uninstallation of Graphite you also like to remove its files. Please select NO because anyways you can delete them manually. The files are kept in
Install and Configure PostgreSQL Database
Graphite internally uses carbon and whisper database library for storing data. But the web application is a Django application which needs some data store for its own purpose. The default data store configured is SQLite3 database files. But this is not a full fledged database system hence we will use PostgreSQL.
Script to install database and libs used by Graphite to communicate with PostgreSQL
Once our PostgreSQL is installed we will create a user and a database
Login to PostgreSQL console
Create a user graphite which will be used by Django to operate on our database.
Please make sure you select a secure password for your user.
Create a database graphite and give our new user graphite ownership of it.
Please verify is database is created or not by connection to it
If you can successfully connect to the database graphite then you are good to go to next step.
Exit from the PostgreSQL console
Configure Graphite Web Application
Now, as we have our PostgreSQL database and user ready to go we can now move to configuring the web application.
Open the Graphite web app configuration file:
Uncomment the SECRET_KEY and give a nice random value to it
Uncomment the TIMEZONE and set it to some appropriate value. I have set it to UTC, but you may choose any one you like
Uncomment the USE_REMOTE_USER_AUTHENTICATION and set tot to True so that remote user will be authenticated first before making any DB changes
Change the database dictionary definition:
Save and close this file.
Sync the Database
Once your web application is configured, it is time to sync your database, create a super user and create the correct structure.
NOTE: It will ask you to create a superuser. Make sure you remember the credentials with which you create one. This user will be used to connect to Graphite application and be admin of it. Being admin you will change interface of Graphite and create graphs.
Carbon is the Graphite storage backend.
Open the configuration file:
Change value of CARBON_CACHE_ENABLED to true
This enables the carbon service to start at boot
Save and close the file.
Next, open the Carbon configuration file:
Set ENABLE_LOGROTATION to True to turn on log rotation
Save and close the file
Configuring Storage Schemas
Now, open the storage schema file. This tells Carbon how long to store values and how detailed these values should be:
Inside you will find entries like
which implies: pattern that matches regular expression ^carbon. should retain the data with retention policy 60:90d which is
- how often a metric is recorded: 60 seconds
- length of time to store those values: 90 days
For detail information on retention policy visit here
Now we need to add our own entry. Let’s take an example test i.e. we need to monitor data points and our data point entries will start with string test.
NOTE: This entry should be added before the default entry mentioned at the bottom of the file
This will match any metrics beginning with “test.”. It will store the data it collects two times, in varying detail.
The first archive definition (1s:10m) will create a data point every ten seconds. It will store the values for only ten minutes.
The second archive (1m:1h) will create a data point every one minute. It will gather all of the data from the past minute (six points, since the previous archive creates a point every ten seconds) and aggregate it to create the point. By default, it does this by averaging the points, but we can adjust this later. It stores the data at this level of detail for one hour.
This example is taken from this link
Save and close the file.
Storage Aggregation Methods
This aggregation methods are used when we try to fetch data that is less detailed (In our previous example we saw 6 data points were aggregated to create 1 data point). Understanding aggregation is important is we want accurate metrics.
Default aggregation method is taking out mean of values which implies that all retention policies other than most detailed one will create data points by taking mean of all data points it received.
We can specify the aggregation configuration in file called storage-aggregation.conf . A sample file is already provided by Carbon, so you can simply copy-paste it for default behaviour.
You can view official documentation to understand it better.
Save and close the file.
Start the carbon service
Setup uwsgi and init script
To install uwsgi globally you can run following command
If pip is not installed in your system, you can run following command
The entrance file for Django application is stored in directory /usr/share/graphite-web and is by default named as graphite.wsgi. You should rename it to graphite_wsgi.py.
You can do this by executing following command
Create log files and socket files with appropriate permissions
Once you have uwsgi setup in your system its time to set up the init script which will make it easier to manage the service.
Create the following file
And put following content in it
Now you can start the Graphite Web application service using following command
But before you can see anything on browser you need to setup Nginx configuration
Setup Nginx for Graphite
Let us first create all files and links
Now we are ready for configuring Nginx server for Graphite
Open file /etc/nginx/sites-enabled/graphite and put following content in it
Now you can view a working Graphite on your server. Just hit http://graphite.yourservername.com from your favourite browser.
See it working
Login to the system with credentials that you provided while creating the superuser.
Once you are logged in, you should see a screen like this.
First we need to add data into the system. Remember we added a pattern matcher in Storage Schema, according to which any pattern that starts with test. will be recorded as our pattern. Lets add some random data
In order to add data we need to run following
This will add one data metric of value 9 in system. Lets add some more data; this time wee loop through values
Now you should see something like this.
I already had some data in the system, your graph will look a bit different but should be similar
This completes the setup of Graphite on your machine. Although the UI does not look good but when used along with Grafana, it gives a complete experience along with high level of customization and metric analytics.