Zabbix

Links

• Homepage
• zabbix 4 database schema
• https://www.digitalocean.com/community/tutorials/introduction-to-queries-mysql
• compilation instructions
• Documentation
• Examples of Common Queries
• Custom scripts
• Various scripts to automate tasks in Zabbix
• Tuning mysql for zabbix
• https://huyabbix.com
• Migrating zabbix database with minimal downtime
• Bug tracker
• Clean up database
• Zabbix and selinux
• Apache/SSL checks
• Zabbix on RHEL/Centos
• Grafana
• https://blog.zabbix.com/zabbix-ha-cluster-setups/8264/ Zabbix HA cluster]

Zabbix API

• The Zabbix API
• API and python

Installing from git

git clone https://github.com/zabbix/zabbix.git
cd zabbix 
./bootstrap.sh

FAQ

SERVER

Adjust loglevel

zabbix_server --runtime-control log_level_increase=trapper
      

Reload zabbix server configuration

zabbix_server -c /etc/zabbix/zabbix_server.conf -R config_cache_reload

 

PROXY

Zabbix does not support SQLite3 database upgrade.

Stop the proxy and (re)move the .db file

[Z3002] cannot create database 'zabbix_proxy': [0] unable to open database file

fix path

 

Cannot parse heartbeat from active proxy

TBD, probably name issue

Troubleshoot high queue on proxy

• https://blog.zabbix.com/how-to-troubleshoot-zabbix-proxy-high-queue/12244/

 

cannot obtain data from proxy "proxybox": ZBX_TCP_READ() failed: [104] Connection reset by peer

?????

SNMP

Cannot find host interface on "esxhost" for item key foo

Might mean you're trying to import an SNMP template before configuring SNMP for the host

 

Agent side ping check

UserParameter=pingtime[*],fping -e $1|sed 's/^.*(\([0-9].*\) ms).*$/\1/g'
UserParameter=pingalive[*],fping $1|grep -q alive;echo $?

 

IPMI errors

cannot connect to IPMI host: [22] Operation canceled

Usually temporary because of broken ipmi lib, ignore it

cannot connect to IPMI host: [16777411] Unknown error 16777411

another classic

Filters

The regular expressions referred to in discovery are found under Administration->General, and then "Regular expressions" in the dropdown at top right of the page

cannot connect to IPMI host: [125] Operation canceled

possibly authentication method issue

Discover: value must be a JSON object

Could mean you need to escape slashes

 

Cannot create item: Invalid first parameter

Problably a calculated item, try doublequoting the item key:

last("foo[bar]")

 

Install recent zabbix on CentOS/RHEL

rpm -ivh https://repo.zabbix.com/zabbix/3.4/rhel/7/x86_64/zabbix-release-3.4-2.el7.noarch.rpm
yum install zabbix-agent

Discovery data example

Output of a discovery script should look like:

{"data":[
  {"{#VAR1}":"value11","#{VAR2":"value12"},
  {"{#VAR1}":"value21","#{VAR2":"value22"}
]}

Backing up tables

https://www.zabbix.org/wiki/Docs/howto/mysql_backup_script

cannot send list of active checks

Most likely ServerActive is defined in agent config, while not used at all

Latest 20 issues

DEFAULT_LATEST_ISSUES_CNT in/usr/share/zabbix/include/defines.inc.php

Zabbix unreachable poller processes more than 75% busy

Increase StartPollersUnreachable

More than 100 items having missing data for more than 10 minutes

Could be high load. Also check Administration->Queue

Zabbix escalator processes more than 75% busy

probably high system load overall

Check agent

zabbix_get -s my.host.com -k agent.version

ZBX_NOTSUPPORTED

Could be anything, enable logging on agent. It could be version mismatch. Check

zabbix_get -s yourhost -k agent.version

If that works, you're calling for an undefined or unsupported key.

Incorrect trigger expression. Host "xx" does not exist or you have no access to this host.

Means there's no related item.

zabbix_get returns nothing

best look at log on agent side

run playbook on single host

ansible_playbook -l somehost somplay.yml

 

Zabbix server is not running: the information displayed may not be current

Might be selinux: http://sysads.co.uk/2013/11/zabbix-server-running-alert/

 

 

 

Monitoring vmware

vmware.hv.cpu.usage[{$URL},{HOST.HOST}]" became not supported: Couldn't resolve host name

Set macro {$URL} to https://your.ip/sdk/ (shouldn't discovery figure that out from {$HOST} ?
      

Couldn't resolve host name

Sometimes it's a matter of waiting a few hours

 

vmware events collector returned empty result

???

No "vmware collector" processes started.

Check StartVMwareCollectors on server or proxy

unsupported item key

This might mean it's expecting a value from the script you're calling.

echo 1

ansible or API not showing host groups

Permissions!! See Administration->User Groups

 

failed to update local proxy configuration copy: invalid field name "items.lastlogsize"

check everything :)

Received value [11] is not suitable for value type [Numeric (unsigned)] and data type [Decimal]

This probably means the agent returned 1\n1

 

database is down: retrying in 10 seconds

try upping max_connections

[Incorrect key file for table 'items'; try to repair it

Could be something /tmp related