Zabbix 101: Distributed Monitoring with Proxies

In the interest of getting this tutorial published more quickly, I've decided to split it into multiple parts. This part (part one) covers the basic installation of proxies and how to get them to communicate with the Zabbix Server. Part two covers agent configuration and additional configuration for proxies.

While it is not the intent of this article to fully explain the ins and outs of Zabbix Proxies, here's a brief overview of them.

Proxies provide the ability to distribute monitoring. There are many use cases for Proxies, some of which include:

Remote environments where it would be impossible to give all the monitored devices access access to the Zabbix Server.
Environments with strict security requirements.
Large environments.

There are two types of Zabbix Proxies: active and passive.

An active Proxy periodically connects to the Zabbix Server to request its configuration (the timing is configurable). It will then collect data and then send it on to the Zabbix Server every so often (also configurable).

A passive Proxy on the other hand does not make any connections to the Zabbix Server on its own. The Zabbix Server will periodically send the proxy its configuration (timing is configurable). The Zabbix Server will also pull data from the Proxy at a configured interval.

For a little more description about Proxies and how they work, take a look at the documentation.

Based on how proxies are used, it makes little to no sense to install one on a Zabbix Server. This tutorial is based on installing it on a separate device. I make the assumption that you are already SSH'd into the server that will become a Zabbix Proxy.

Now onto the tutorial!

Installation

If you really want to install Zabbix from source, be my guest, but I assume you don't want to. The commands to install the Zabbix Proxy really depend on what flavor of Linux you're running and are pulled directly from the documentation. If you happen to be running Debian/Ubuntu or some other distro, take at the Zabbix documentation for specific instructions for your case.

RedHat 6/CentOS 6/<Derivatives>

As you probably have guessed, you'll add the Zabbix Yum Repository to your server and then use Yum to install the proxy. Run these commands to do that:

rpm -ivh http://repo.zabbix.com/zabbix/2.4/rhel/6/x86_64/zabbix-release-2.4-1.el6.noarch.rpm
yum install zabbix-proxy-sqlite3

Done! You may have noticed that we explicitly installed the SQLite3 proxy binary. That's because in the vast majority of cases the SQLite Proxy database will suffice. Unless you are pushing a LOT of data through a Proxy (I believe the technical term would be "crap-ton") the SQLite database will work just fine and requires less maintenance/configuration than using MySQL or Postgresql.

Configuration File

While it's nice to get the Proxy installed so quickly, it does require some specific configuration. If you installed from Linux Packages (ie: RPM/deb) then your configuration files will be in /etc/zabbix. The Proxy configuration file is most likely located at /etc/zabbix/zabbix_proxy.conf. Go ahead and open up that file in your favorite text editor.

First we'll take a look at two options that need to be configured on all proxies.

### Option: ProxyMode
#       Proxy operating mode
#       0 - proxy in the active mode
#       1 - proxy in the passive mode
## Mandatory: no
# Default:
# ProxyMode=0

The "ProxyMode" option defines what type of proxy this is. By default proxies are set to be "active". If you want a "passive" proxy, go ahead and uncomment this option and set the value to "1".

### Option: DBName 
#       Database name. 
#       For SQLite3 path to database file must be provided. DBUser and DBPassword are ignored. 
#       Warning: do not attempt to use the same database Zabbix server is using. 
## Mandatory: yes
# Default: 
# DBName=

DBName=zabbix

Make sure you read the comment for the "DBName" parameter. Notice that for SQLite (which we're using) this parameter is the absolute path to the SQLite database the proxy will use. Something neat about SQLite proxies is that the proxy will automatically create the database if it doesn't exist.

By default the configuration file is not setup with a path in which to write the SQLite database. The "DBName" option should point to a valid location. A location that most will most likely work on a RedHat/CentOS box is "/var/lib/zabbix/proxy_db.sqlite3".

Caveat: Make sure that SELinux and friends do not prevent the proxy from writing to whatever location you give it! If the proxy has issues creating the database, you will see errors like the last one shown here in the proxy log file:

7063:20150324:173741.311 cannot open database file "/var/lib/zabbix/proxy_db.sqlite3": [2] No such file or directory
7063:20150324:173741.312 creating database ...
7063:20150324:173741.312 [Z3002] cannot create database '/var/lib/zabbix/proxy_db.sqlite3': [0] unable to open database file

Of course, normal Unix permissions apply here too, so it could be simply that the user the proxy runs as ("zabbix" by default) does not have rights to write files in whatever path you gave.

The next important configuration options really depend on the type of Proxy that you wish to setup. As such, the following two sections cover the relevant options for each type of proxy.

Active Proxy

### Option: Hostname 
#       Unique, case sensitive Proxy name. Make sure the Proxy name is known to the server! 
#       Value is acquired from HostnameItem if undefined.
## Mandatory: no
# Default: 
# Hostname= 

Hostname=Zabbix proxy

The importance of the "Hostname" option cannot be understated. When an active proxy talks to the Zabbix Server, it will pass this value to the server to identify which proxy is talking. The value here MUST match exactly with another one that will be discussed later. Suffice to say, just make sure you remember what you put here.

### Option: Server 
#       IP address (or hostname) of Zabbix server. 
#       Active proxy will get configuration data from the server. 
#       For a proxy in the passive mode this parameter will be ignored. 
## Mandatory: yes (if ProxyMode is set to 0)
# Default:
# Server=

Server=127.0.0.1

The "Server" option indicates the address (IP/FQDN) of the Zabbix Server. The proxy uses this option to know from where to retrieve its configuration.

Passive Proxy

In the case of a passive proxy, there isn't anything additional to setup on the proxy itself. You're done!

Starting the service

Go ahead and start up the proxy. Assuming RPM/Deb packages were used, run this command:

service zabbix-proxy restart

We do a restart here for consistency's sake. This will make sure none of you somehow had some random proxy processes already running that cause issues :).

Take a look at the proxy log file. It probably exists at /var/log/zabbix/zabbix_proxy.log. Note that this output is from Zabbix 2.4.x! Depending on your version of Zabbix, the output could be different. You should see output like this:

1588:20150326:094411.616 Starting Zabbix Proxy (active) [Zabbix proxy]. Zabbix 2.4.4 (revision 52341). 
1588:20150326:094411.617 **** Enabled features **** 
1588:20150326:094411.617 SNMP monitoring:       YES 
1588:20150326:094411.617 IPMI monitoring:       YES
1588:20150326:094411.617 WEB monitoring:        YES 
1588:20150326:094411.617 VMware monitoring:     YES 
1588:20150326:094411.617 ODBC:                  YES 
1588:20150326:094411.617 SSH2 support:          YES 
1588:20150326:094411.617 IPv6 support:          YES
1588:20150326:094411.617 ************************** 
1588:20150326:094411.617 using configuration file: /etc/zabbix/zabbix_proxy.conf 
1588:20150326:094411.634 current database version (mandatory/optional): 02040000/02040000 
1588:20150326:094411.634 required mandatory version: 02040000
1588:20150326:094411.644 proxy #0 started [main process]
1590:20150326:094411.645 proxy #1 started [configuration syncer #1]
1591:20150326:094411.645 proxy #2 started [heartbeat sender #1] 
1592:20150326:094411.645 proxy #3 started [data sender #1]
1593:20150326:094411.646 proxy #4 started [poller #1]
1591:20150326:094411.647 sending heartbeat message to server failed: error:"negative response: "failed"", info:"proxy "Zabbix proxy" not found"
1594:20150326:094411.651 proxy #5 started [poller #2]
1595:20150326:094411.664 proxy #6 started [poller #3] 
1596:20150326:094411.647 proxy #7 started [poller #4]
1598:20150326:094411.659 proxy #9 started [unreachable poller #1]
1599:20150326:094411.650 proxy #10 started [trapper #1]
1600:20150326:094411.663 proxy #11 started [trapper #2]
1602:20150326:094411.669 proxy #13 started [trapper #4] 
1603:20150326:094411.659 proxy #14 started [trapper #5]
1604:20150326:094411.664 proxy #15 started [icmp pinger #1] 
1605:20150326:094411.651 proxy #16 started [housekeeper #1]
1606:20150326:094411.670 proxy #17 started [http poller #1] 
1609:20150326:094411.651 proxy #20 started [history syncer #2]
1610:20150326:094411.670 proxy #21 started [history syncer #3] 
1612:20150326:094411.663 proxy #23 started [self-monitoring #1]
1608:20150326:094411.669 proxy #19 started [history syncer #1]
1601:20150326:094411.670 proxy #12 started [trapper #3]
1611:20150326:094411.670 proxy #22 started [history syncer #4]
1597:20150326:094411.670 proxy #8 started [poller #5]
1590:20150326:094411.719 cannot obtain configuration data from server: proxy "Zabbix proxy" not found
1607:20150326:094411.727 proxy #18 started [discoverer #1]
1591:20150326:094411.647 sending heartbeat message to server failed: error:"negative response: "failed"", info:"proxy "Zabbix proxy" not found"
1590:20150326:094411.719 cannot obtain configuration data from server: proxy "Zabbix proxy" not found

Notice the log entries that show the message "proxy 'Zabbix proxy' not found". Remember that "Hostname" parameter in zabbix_proxy.conf? An active Zabbix proxy passes that value to the server (as mentioned earlier). In this case, we haven't finished setting everything up for the proxy to work right, so these log messages are expected.

The log messages from 2.0.x are slightly different. The log file will not indicate which configuration file it read and any errors talking to the server will look like this and mean the same thing:

1953:20150326:193529.347 Heartbeat message failed
1952:20150326:193529.425 Cannot obtain configuration data from server. Proxy host name might not be matching that on the server.

Configuration (UI)

Go ahead and fire up a browser and load the Zabbix UI.

For Zabbix 2.4+, go to "Administration" -> "Proxy".
For Zabbix 2.0 or 2.2, go to "Administration -> "DM". Once the page loads, ensure that the dropdown box in the upper-right corner has "Proxies" selected and not "Nodes".

Now go ahead and click the "Create proxy" button so that we can create the configuration for the proxy.

The manner in which to configure the proxy depends on the type of proxy that is wanted.

Active Proxy

Remember that "Hostname" parameter in the zabbix_proxy.conf file? This is where it must match exactly. In the "Proxy name" box, make 100% certain that the value entered matches that "Hostname" parameter EXACTLY (beware of white spaces in here).
Verify that the "proxy mode" option is set to "active".
Click "Save".

On the proxy itself, restart the proxy service by running:

service zabbix-proxy restart

After the restart, you should now see a line similar to this:

1862:20150330:213908.399 received configuration data from server, datalen 2594

If you see that log entry, congrats! Your proxy will now retrieve its configuration from the server on a given interval.
If you still see entries indicating a failure (like the ones mentioned before in the "Starting the service" section), then most likely your "Hostname" configuration option and the "Proxy name" entry in the UI don't match. Another thing to check could be the "Server" parameter in zabbix_proxy.conf to be sure that it points to the correct Zabbix server.

Passive Proxy

The "Proxy name" field can be anything. Make sure it's something that you'll recognize later on though!
Set "Proxy mode" to "passive".
In the "interface" section only one of the following is required

"IP Address" - Set the IP address for the proxy and then click the "IP" button under the "Connect to" option.
"DNS" - Set the DNS for the proxy and then click the "DNS" button under the "Connect to" option.

Change the port number if necessary. By default proxies listen on port 10051, so probably no change is necessary.
Click "Save".

You might have to wait a while for the Zabbix Server to send the Proxy its configuration data (up to one hour by default). When it does send the proxy it's configuration, you'll see a line like this in the Zabbix server log (/var/log/zabbix/zabbix_server.log):

1901:20150430:212155.013 sending configuration data to proxy "my_cool_proxy", datalen 2588

Tidying up

At this point the proxy can communicate with the Zabbix server. Congratulations! In part two we'll discuss how to change the frequency in which the the proxy is given/gets its configuration. We'll also discuss how to change hosts to be monitored by a proxy.

If it appears your proxy is still unable to talk to the Zabbix server, make sure to look for these gotchas:

If using an active proxy, can that proxy talk to the zabbix server on the port it listens (port 10051 by default)?
If using a passive proxy, can the Zabbix server talk to the proxy on the port it listens (port 10051 by default)?
If the proxy log indicates a failure to create the SQLite database, did you change the "DBName" parameter in zabbix_proxy.conf to point to a valid location?

Does the user the Zabbix proxy runs as ("zabbix" by default) have write permissions to this path?
Is SELinux (or other programs like it) preventing the Zabbix proxy process from writing to this path?

If using an active proxy, does the "Hostname" parameter in zabbix_proxy.conf EXACTLY match the proxy name given in the Zabbix UI?
In the Zabbix UI, does the proxy configuration correctly indicate the kind of proxy being used (Active/Passive)?
If using a passive proxy, does the Zabbix UI show that the proxy is configured with the correct IP or FQDN?