Monitor Different Systems with Zabbix
This is the last post from the “Network Monitoring Software Comparison” series. As I mentioned in the main page here is what Zabbix is about:
Zabbix is an enterprise-class open source distributed monitoring solution.
Zabbix is software that monitors numerous parameters of a network and the health and integrity of servers. Zabbix uses a flexible notification mechanism that allows users to configure e-mail based alerts for virtually any event. This allows a fast reaction to server problems. Zabbix offers excellent reporting and data visualisation features based on the stored data. This makes Zabbix ideal for capacity planning.
Zabbix supports both polling and trapping. All Zabbix reports and statistics, as well as configuration parameters, are accessed through a web-based front end. A web-based front end ensures that the status of your network and the health of your servers can be assessed from any location. Properly configured, Zabbix can play an important role in monitoring IT infrastructure. This is equally true for small organisations with a few servers and for large companies with a multitude of servers.
So let’s try it out.
Install Zabbix Server on Ubuntu
Most of the instructions can be found in this Zabbix page. First let’s install the software:
kerch:~>sudo apt-get install zabbix-server-mysql zabbix-frontend-php
Now we need to setup the MySQL database. Instructions for the setup can be found online here, or in the /usr/share/doc/zabbix-server-mysql/README.Debian file (after the zabbix-server package has been installed). Here are the commands I ran to setup the MySQL database for Zabbix:
kerch:~>mysql -u root -p -e "create database zabbix character set utf8"
kerch:~>mysql -u root -p -e "grant all on zabbix.* to 'zabbix'@'localhost' identified by 'password'"
kerch:~>zcat /usr/share/zabbix-server-mysql/{schema,images,data}.sql.gz | mysql -uzabbix -ppassword zabbix
Now let’s add the MySQL credentials to the Zabbix configuration files and enable the service. First edit the /etc/zabbix/zabbix_server.conf file, add the MySQL stuff, and set the IP of the Zabbix server. Here is how my configuration file looked like after I was done:
kerch:~>grep -vE '^#|^$' /etc/zabbix/zabbix_server.conf
SourceIP=192.168.1.100
LogFile=/var/log/zabbix-server/zabbix_server.log
PidFile=/var/run/zabbix/zabbix_server.pid
DBHost=localhost
DBName=zabbix
DBUser=zabbix
DBPassword=password
AlertScriptsPath=/etc/zabbix/alert.d/
FpingLocation=/usr/bin/fping
Fping6Location=/usr/bin/fping6
Then let’s enable the service by editing the /etc/default/zabbix-server file and modifying the following line to look like this:
kerch:~>grep -vE '^#|^$' /etc/default/zabbix-server
START=yes
Now let’s start the service:
kerch:~>sudo service zabbix-server start
kerch:~>sudo service zabbix-server status
* zabbix_server is running
Lastly let’s enable the web front-end. First, create the directory where the files will be hosted:
kerch:~>sudo mkdir /var/www/zab
then copy the PHP files:
kerch:~>sudo rsync -avzP /usr/share/zabbix/. /var/www/zab/.
Lastly visit the site at 127.0.0.1/zab. When I did, I saw the following:
I then clicked “Next” and I saw the following:
It looks like some of my PHP options were not up to Zabbix’s standards. So I edited the /etc/php5/apache2/php.ini file and fixed all of the above options, then I restarted Apache like so:
kerch:~>sudo service apache2 restart
* Restarting web server apache2
... waiting [ OK ]
Then clicking Retry, I saw the following:
then I clicked “Next” and saw the database page, I filled out the settings and clicked “Test Connection” and it was successful, like so:
Then, I filled out the server connection details:
And then I saw the pre-installation summary page:
it then tried to auto create a configuration file under /etc/zabbix but it failed:
So I downloaded it manually and copied the file to the appropriate location:
kerch:~>sudo cp zabbix.conf.php /etc/zabbix/.
Then hitting “Retry” I saw the following page:
Hitting “Next” yielded the following page:
You can login with Username: Admin, Password: zabbix. After you login, you can create another user; instruction on user setup and default password are here. After I logged in, I saw the following:
Now let’s start monitoring some systems.
Monitor an Ubuntu Machine with Zabbix
First let’s install the zabbix-agent package:
kerch:~>sudo apt-get install zabbix-agent
then edit the /etc/zabbix/zabbix_agentd.conf file and add/modify the following:
kerch:~>grep -vE '^#|^$' /etc/zabbix/zabbix_agentd.conf
PidFile=/var/run/zabbix/zabbix_agentd.pid
LogFile=/var/log/zabbix-agent/zabbix_agentd.log
LogFileSize=0
Server=192.168.1.100
ListenIP=192.168.1.100
ServerActive=192.168.1.100
Hostname=kerch.dnsd.me
Include=/etc/zabbix/zabbix_agentd.conf.d
Then start the agent and make sure it’s running:
kerch:~>sudo service zabbix-agent start
kerch:~>sudo service zabbix-agent status
* zabbix_agentd is running
This node is a special case since it will be both the server and a node. So make sure both are running:
kerch:~>sudo netstat -antp | grep zabbix
tcp 0 0 192.168.1.100:10050 0.0.0.0:* LISTEN 683/zabbix_agentd
tcp 0 0 0.0.0.0:10051 0.0.0.0:* LISTEN 4784/zabbix_server
That looks good. Now let’s add the host to be monitored with Zabbix via the PHP frontend. Most of the instructions are laid out here. So in the Zabbix Dashboard go to “Configuration” -> “Hosts” and you will see the following:
Then click on “Create Host” and fill out all the information, mine looked like this:
Under the template section choose “Template OS Linux”:
Then click “Save” and you will see the following:
At first I checked out a graph but the font was missing for some reason, I found a fix on the Zabbix forums from here. After I pointed to the appropriate path for the fonts, I was able to see the graphs like so:
Also going to “Monitoring” -> “Overview” you can see current values for our hosts:
Now let’s add our FreeBSD machine to the mix.
Monitor FreeBSD with Zabbix
First let’s install the zabbix-agent:
freebsd:~>cd /usr/ports/net-mgmt/zabbix2-agent/
freebsd:/usr/ports/net-mgmt/zabbix2-agent/>sudo make install clean
after the install is finished enable the service by adding:
zabbix_agentd_enable="YES"
to /etc/rc.conf. Next let’s copy the default configuration file:
freebsd:~>sudo cp /usr/local/etc/zabbix2/zabbix_agentd.conf.sample /usr/local/etc/zabbix2/zabbix_agentd.conf
Now edit the /usr/local/etc/zabbix2/zabbix_agentd.conf file and set it up to connect to our server. Here is how my file looked like at the end:
freebsd:~>grep -vE '^#|^$' /usr/local/etc/zabbix2/zabbix_agentd.conf
LogFile=/tmp/zabbix_agentd.log
Server=192.168.1.100
ListenIP=192.168.1.101
ServerActive=192.168.1.100
Hostname=freebsd.dnsd.me
Lastly start the agent:
freebsd:~>sudo service zabbix_agentd start
Starting zabbix_agentd.
freebsd:~>sudo service zabbix_agentd status
zabbix_agentd is running as pid 60263 60264 60265 60266 60267 60268.
Lastly ensure it’s listening on the correct port:
freebsd:~>sockstat -4 | grep zabbix
zabbix zabbix_age 60268 4 tcp4 192.168.1.101:10050 *:*
zabbix zabbix_age 60267 4 tcp4 192.168.1.101:10050 *:*
zabbix zabbix_age 60266 4 tcp4 192.168.1.101:10050 *:*
zabbix zabbix_age 60265 4 tcp4 192.168.1.101:10050 *:*
zabbix zabbix_age 60264 4 tcp4 192.168.1.101:10050 *:*
zabbix zabbix_age 60263 4 tcp4 192.168.1.101:10050 *:*
Now let’s add the host to the Zabbix server. Go back to the dashboard (127.0.0.1/zab) and then go to “Configuration” -> “Hosts” -> “Create Host” and fill out all the necessary information:
And under templates choose, “Template OS FreeBSD”:
After a little bit of time you should see the host as monitored with a green Z under “Configuration” -> “Hosts”:
If you want to check what items will be monitored, you can go to “Configuration” -> “Host” -> Pick a Host -> “Items” and you will see a list of all the checks:
From there you can add triggers and graphs depending on the items, but we will get into that later. Now let’s add another machine to monitor.
Monitor Fedora with Zabbix
As always, let’s install the software:
moxz:~>sudo yum install zabbix-agent
Now let’s configure the agent by editing the /etc/zabbix_agentd.conf file. Here is how my file looked like:
moxz:~>grep -vE '^#|^$' /etc/zabbix_agentd.conf
PidFile=/var/run/zabbix/zabbix_agentd.pid
LogFile=/var/log/zabbix/zabbix_agentd.log
LogFileSize=0
Server=192.168.1.100
ListenIP=192.168.1.102
ServerActive=192.168.1.100
Hostname=moxz.dnsd.me
Now let’s enable the agent and start it up:
moxz:~>sudo service zabbix-agent enable
Redirecting to /bin/systemctl enable zabbix-agent.service
ln -s '/usr/lib/systemd/system/zabbix-agent.service' '/etc/systemd/system/multi-user.target.wants/zabbix-agent.service'
moxz:~>sudo service zabbix-agent start
Redirecting to /bin/systemctl start zabbix-agent.service
moxz:~>sudo service zabbix-agent status
Redirecting to /bin/systemctl status zabbix-agent.service
zabbix-agent.service - Zabbix Monitor Agent
Loaded: loaded (/usr/lib/systemd/system/zabbix-agent.service; enabled)
Active: active (exited) since Wed 2013-03-06 12:54:25 PST; 6h ago
Process: 14065 ExecStart=/usr/sbin/zabbix_agentd (code=exited, status=0/SUCCESS)
Open up the firewall for TCP port 10050 by adding the following :
-A INPUT -s 192.168.1.0/24 -p tcp -m state --state NEW -m tcp --dport 10050 -j ACCEPT
to the /etc/sysconfig/iptables file and restarting the iptables service like so:
sudo service iptables restart
Lastly add the host using the Zabbix front-end and make sure “Template OS Linux” is selected. When it’s all done we should see something like this under “Configuration” -> “Hosts”:
Monitor raid status on FreeBSD with Zabbix
There are good examples in this post. We will have to use “User Parameters” to get this working. The first link from above describes the process pretty well. I will define two parameters
- raid.active_drives
- raid.controller_status
First let’s run the commands that will provide the above paramaters:
freebsd:~>/usr/local/sbin/arcconf getconfig 1 PD | grep State | grep Online| /usr/bin/wc -l
2
and here is the second one:
freebsd:~>/usr/local/sbin/arcconf getconfig 1 AD | grep "Controller Status" | grep -vc Optimal
0
if the value is non zero then we know we are not at an Optimal state. So let’s add both to our zabbix_agentd config under /usr/local/etc/zabbix2/zabbix_agentd.conf:
## Return Number of online drives
UserParameter=raid.active_drives,/usr/local/sbin/arcconf getconfig 1 PD | grep State | grep Online| /usr/bin/wc -l
## Return 1 if status of raid is not optimal
UserParameter=raid.controller_status,/usr/local/sbin/arcconf getconfig 1 AD | grep "Controller Status" | grep -vc Optimal
Now let’s restart the zabbix-agentd process:
freebsd:~>sudo service zabbix_agentd restart
Stopping zabbix_agentd. Waiting for PIDS: 60263 60264 60265 60266 60267 60268.
Starting zabbix_agentd.
Now let’s see if we can get those values locally as the zabbix user:
freebsd:~>sudo su -m zabbix -c '/usr/local/sbin/arcconf getconfig 1 PD | grep State | grep Online| /usr/bin/wc -l'
2
freebsd:~>sudo su -m zabbix -c '/usr/local/sbin/arcconf getconfig 1 AD | grep "Controller Status" | grep -vc Optimal'
0
That looks good. Now let’s see if we do it via the zabbix_agentd process:
freebsd:~>zabbix_agentd -t raid.active_drives
Alarm clock
Initially it looks like I was getting a SIGARLM signal, I then updated the ‘timeout’ value to 30 seconds:
freebsd:~>grep ^Time /usr/local/etc/zabbix2/zabbix_agentd.conf
Timeout=30
and restarted the zabbix_agentd service and after that the command ran:
freebsd:~>zabbix_agentd -t raid.active_drives
raid.active_drives [/usr/local/sbin/arcconf getconfig 1 PD | grep State | grep Online| /usr/bin/wc -l] [t| 2]
freebsd:~>zabbix_agentd -t raid.controller_status
raid.controller_status [/usr/local/sbin/arcconf getconfig 1 AD | grep "Controller Status" | grep -vc Optimal] [t|0]
Both look good. Now trying the same thing from the Zaabix server:
kerch:~>zabbix_get -s 192.168.1.101 -k raid.active_drives
2
kerch:~>zabbix_get -s 192.168.1.101 -k raid.controller_status
0
That looks good. Now to add items for our new values. From the Zabbix front-end go to “Configuration” -> “Hosts” -> Select FreebSD -> “Items” -> “Create Item” and fill out all the settings:
When finished, click “Save”. Then from the same window click on “Graphs” -> “Create Graph” and fill out the information:
Then checking out the graph (by going to “Monitoring” -> “Graphs” -> “Host” -> “FreeBSD” -> “Graph” -> “Raid_#_Disks_Online”), here is what I saw:
Now let’s setup a new item for the controller status. Go to “Configuration” -> “Hosts” -> Select host -> “Items” -> “Create New Item” and fill out all the information:
Then from the same page, click on “Triggers” -> “Create Trigger”. In the new window select “Add” under Trigger Expressions and make the follow expression:
then click “Save”. Your final result will look like this:
You can go to “Monitoring” -> “Latest Data” and then expand the Name when you groupped your item, in my case I put it under OS. Expanding OS, I saw the latest value of that item:
Also forcing a non-zero return value from that script yeilded the trigger to be fired. Here is what I saw under “Monitoring” -> “Triggers”:
and then fixing the script to return the correct value, allowed the trigger to be fixed. Here is what I saw under “Monitoring” -> “Events” after the issue was fixed:
Zabbix Vs. Zenoss, Munin, and Collectd
PROs
- Lightweight compared to Zenoss and for all the functionality that it provides
- Doesn’t Depend on Python or Perl
- Supports MySQL, PSQL, and SQlite
- Very easy to customize Triggers and Graphs
- Has the capability to monitor with agents and without
- Has great support for most OSes, even FreeBSD
- Works like an Intrusion Detection System (IDS). I installed a new software and it automatically added a new user, I then saw the following under events:
CONs
- Depends on PHP, the front-end a little slow and not as cosmetic as Zenoss
- Doesn’t support Nagios plugins
- Doesn’t load multiple Graphs at ones, only one aspect at a time (CPU or Memory but not both)
- Requires an external Database (MySQL)