Zabbix: Difference between revisions

From DWIKI
 
(137 intermediate revisions by the same user not shown)
Line 1: Line 1:
=Links=
*[http://www.zabbix.org/ Homepage]
*https://www.digitalocean.com/community/tutorials/introduction-to-queries-mysql
*[http://zabbix.org/wiki/Compilation_instructions compilation instructions]
*[http://www.zabbix.com/documentation Documentation]
*[https://dev.mysql.com/doc/mysql-tutorial-excerpt/8.0/en/examples.html Examples of Common Queries]
*[http://zabbixzone.com/zabbix/easy-update-on-custom-scripts/ Custom scripts]
*[https://github.com/q1x/zabbix-gnomes Various scripts to automate tasks in Zabbix]
*[https://www.percona.com/blog/2014/11/14/optimizing-mysql-zabbix/ Tuning mysql for zabbix]
*https://huyabbix.com
*[http://techblog.procurios.nl/k/n618/news/view/56429/14863/how-to-migrate-mysql-databases-without-downtime.html Migrating zabbix database with minimal downtime]
*[https://support.zabbix.com/browse/ZBX/ Bug tracker]
*[http://www.michaelfoster82.co.uk/zabbix-database-cleanup-delete-old-data/ Clean up database]
*[https://linux.die.net/man/8/zabbix_selinux Zabbix and selinux]
*[[ Zabbix on RHEL/Centos ]]
*[[Grafana]]


=Installing from git=
= Links =
  git clone https://github.com/zabbix/zabbix.git
 
*[http://www.zabbix.org/ Homepage]
*[https://zabbix.org/wiki/Docs/DB_schema/4.0 zabbix 4 database schema]
*[https://www.zabbix.com/documentation/current/en/manual/appendix/compatibility Zabbix compatibility matrix]
*[https://www.digitalocean.com/community/tutorials/introduction-to-queries-mysql https://www.digitalocean.com/community/tutorials/introduction-to-queries-mysql]
*[http://zabbix.org/wiki/Compilation_instructions compilation instructions]
*[http://www.zabbix.com/documentation Documentation]
*[https://dev.mysql.com/doc/mysql-tutorial-excerpt/8.0/en/examples.html Examples of Common Queries]
*[http://zabbixzone.com/zabbix/easy-update-on-custom-scripts/ Custom scripts]
*[https://github.com/q1x/zabbix-gnomes Various scripts to automate tasks in Zabbix]
*[https://www.percona.com/blog/2014/11/14/optimizing-mysql-zabbix/ Tuning mysql for zabbix]
*[https://huyabbix.com https://huyabbix.com]
*[http://techblog.procurios.nl/k/n618/news/view/56429/14863/how-to-migrate-mysql-databases-without-downtime.html Migrating zabbix database with minimal downtime]
*[https://support.zabbix.com/browse/ZBX/ Bug tracker]
*[http://www.michaelfoster82.co.uk/zabbix-database-cleanup-delete-old-data/ Clean up database]
*[https://linux.die.net/man/8/zabbix_selinux Zabbix and selinux]
*[https://github.com/a-schild/zabbix-ssl Apache/SSL checks]
*[[Zabbix_on_RHEL/Centos|Zabbix on RHEL/Centos]]
*[[Grafana|Grafana]]
*[https://blog.zabbix.com/zabbix-ha-cluster-setups/8264/ https://blog.zabbix.com/zabbix-ha-cluster-setups/8264/] Zabbix HA cluster]
*[https://blog.zabbix.com/zabbix-agent-active-vs-passive/9207/ Active vs Passive]
*[https://geofrogger.net/review/zabbix20.svg Very old network diagram]
*[https://blog.zabbix.com/fighting-notification-floods-and-misleading-alerts-in-distributed-zabbix-deployments/11600/ Fighting zabbix alert floods]
*[https://github.com/unioslo/zabbix-cli zabbix-cli]
 
=Documentation=
==Triggers==
*[https://www.zabbix.com/documentation/current/en/manual/appendix/functions Trigger functions]
Function str() searches for substrings
 
==AND OR case==
not, and and or operators are case-sensitive and must be in lowercase. They also must be surrounded by spaces or parentheses.
 
 
==Installation==
*[https://repo.zabbix.com/ Zabbix repositories]
*[https://wiki.gentoo.org/wiki/User:MalakymR/Zabbix Zabbix on Gentoo]
 
== Installing Zabbix from git ==
 
  git clone [https://github.com/zabbix/zabbix.git https://github.com/zabbix/zabbix.git]
  cd zabbix  
  cd zabbix  
  ./bootstrap.sh
  ./bootstrap.sh
./configure --help
autoreconf -fvi
If you don't have a Makefile, try
./config.status Makefile
and then
./configure
again
== Zabbix API ==
*[https://www.zabbix.com/documentation/current/manual/api The Zabbix API]
*[https://www.zabbix.com/integrations/python API and python]




==Zabbix agent paths==
Ubuntu:
/etc/zabbix/zabbix_agentd.conf.d/
/etc/zabbix/zabbix_agentd.conf


=Templates=
==Simple check==
==Mysql template==
*[https://www.zabbix.com/documentation/6.4/en/manual/config/items/itemtypes/simple_checks Simple checks]




=FAQ=
==Zabbix and SNMP==
*[https://bestmonitoringtools.com/tutorial-snmp-traps-on-zabbix/ Zabbix SNMP Traps: A Step-by-Step Guide]


= Zabbix error codes =


== Z3005 ==


Database issue
= Items =
[https://www.zabbix.com/documentation/6.0/en/manual/config/items/itemtypes/zabbix_agent Agent item keys]
==Item dialog==
*[https://www.zabbix.com/documentation/5.4/en/manual/config/items/item Item documentation]
*[https://www.zabbix.com/documentation/5.4/en/manual/config/items/itemtypes/zabbix_agent Zabbix agent items]
===Units===
*B
*uptime
*unixtime
*s
== proc.mem ==
proc.mem[<name>,<user>,<mode>,<cmdline>,<memtype>]
=== name ===
??
=== cmdline ===
regex like php-fpm:
===memtype===
*[https://www.zabbix.com/documentation/5.0/en/manual/appendix/items/proc_mem_notes Notes on proc.mem memtypes]
==Item preprocessing==
===Preprocessing regular expressions===
See [https://www.zabbix.com/documentation/current/en/manual/regular_expressions#example Regular expressions: example]
===XML/xpath preprocessing===
https://blog.zabbix.com/zabbix-xpath-preprocessing/7936/
NOTE xq -x does not want the number() bit
===Incorrect value for field "Prev. time": a relative time is expected.===
Prev. Time should be something like
now-30s
==Windows performance counters==
https://www.zabbix.com/documentation/current/en/manual/config/items/perfcounters
= Templates =
*[https://github.com/zabbix/community-templates/ Community templates]
==Template App MySQL==
https://github.com/tiramiseb/zabbix-templates/blob/master/Template%20App%20MySQL.txt
TODO shouldn't this be user zabbix?
mysql user account:
create user 'monitor'@'localhost' identified by auth_socket;
grant PROCESS,SHOW DATABASES,SHOW VIEW on *.* to 'monitor'@'localhost';
flush privileges;
= Configuration =
== Zabbix agent active ==
=== On client ===
Have port 10051 open and:
ActiveServer zabbix.ser.ver
=== On server ===
Set Agent IP to 0.0.0.0
&nbsp;
=Zabbix and SQL=
==Find hosts with hostmacro defined==
select h.host, m.macro, m.value from hosts h, hostmacro m where macro like '%FOO%' and h.hostid = m.hostid;
==most frequent items in history_uint==
select itemid,count(itemid) as freq from history_uint group by itemid order by freq desc limit 5;
and then
select name from items where itemid = whateveryoufind;
= HOWTO =
==Define discovery filters==
== LLD with JSON ==
*[https://blog.zabbix.com/low-level-discovery-with-dependent-items/13634/ LLD with JSON and dependent items]
*https://www.zabbix.com/forum/zabbix-help/383827-json-and-lld-understanding
*https://www.zabbix.com/forum/zabbix-troubleshooting-and-problems/456663-lld-macros-with-json
if you want multiple keys, use jsonpath like
$[?(@.share=='{#FSTYPE}' && @.name=='{#NAME}')].size.first()
===testing jsonpath preprocessing===
In Value paste valid json, then name {#NAME} value somevalue
== Test trapper ==
==Reset admin password==
Mysql prompt:
select * from user where username='Admin';
bcrypt your new password:
htpasswd -nbBC 10 USER YOURPASSWORD|awk -F ':' '{ print $2 }'
Mysql prompt:
update user set passwd = 'your bcrypted pass' where userid = 1
= FAQ =
== SERVER ==
=== Adjust loglevel ===
zabbix_server --runtime-control log_level_increase=trapper
     
&nbsp;
=== Reload zabbix server configuration ===
You can't, but you might want
zabbix_server -c /etc/zabbix/zabbix_server.conf -R config_cache_reload
&nbsp;
=== No media defined for user ===
=== The frontend does not match Zabbix database. ===
Probably version conflict between frontend and server
&nbsp;
=== value cache working in low memory mode ===
Increase ValueCacheSize
=== Message from 1.2.3.4 is missing header. Message ignored. ===
&nbsp;
&nbsp;
== PROXY ==
[[ Zabbix Proxy ]]
===cannot send proxy data to server===
empty string received
===failed to update local proxy configuration copy: unexpected field "host_inventory.type"===
== Front end ==
===Round numbers===
Preprocessing javascript
2 decimals:
return Math.round(value* 100) / 100
0 decimals:
return Math.round(value)
=== Visable name vs hostname ===
Visible name: {HOST.NAME}
Hostname: {HOST.HOST}
Host IP: (as defined in Interface->IP/DNS) {HOST.CONN}
&nbsp;
=== Acknowledge multiple items ===
Monitor->Problems apply filters, select all, mass update
&nbsp;
=== No permissions to referred object or it does not exist! ===
Graph no longer exists. Probably items no longer discovered
Maybe you've been editing a template file, remember to replace template name everywhere
=== Cannot add host ===
??
== Monitoring SNMP ==
=== Cannot find host interface on "xxxhost" for item key foo ===
Might mean you're trying to import an SNMP template before configuring SNMP for the host
&nbsp;
&nbsp;
=== No SNMP data ===
=== snmp_parse_oid(): cannot parse OID "IF-MIB::ifSpeed.3 ===
===Timeout while connecting===
Could be wrong community string, remember delay when using proxy.
== Agent side ping check ==


==Agent side ping check==
  UserParameter=pingtime[*],fping -e $1|sed 's/^.*(\([0-9].*\) ms).*$/\1/g'
  UserParameter=pingtime[*],fping -e $1|sed 's/^.*(\([0-9].*\) ms).*$/\1/g'
  UserParameter=pingalive[*],fping $1|grep -q alive;echo $?
  UserParameter=pingalive[*],fping $1|grep -q alive;echo $?


==IPMI errors==
&nbsp;
===cannot connect to IPMI host: [22] Operation canceled===
 
Usually temporary because of broken ipmi lib, ignore it
&nbsp;
 
 
==LLD/Discovery==
=== Discover: value must be a JSON object ===


Could mean you need to escape slashes, check output with zabbix_get


===cannot connect to IPMI host: [125] Operation canceled===
possibly authentication method issue


==Discover: value must be a JSON object==
===Cannot create item: item with the same key already exists===
Could mean you need to escape slashes
make sure the key contains "{#SOMENAME}"


==Install recent zabbix on CentOS/RHEL==
== Discovery data example ==
rpm -ivh https://repo.zabbix.com/zabbix/3.4/rhel/7/x86_64/zabbix-release-3.4-2.el7.noarch.rpm
yum install zabbix-agent


==Discovery data example==
Output of a discovery script should look like:
Output of a discovery script should look like:


Line 59: Line 314:
  ]}
  ]}


==Backing up tables==
https://www.zabbix.org/wiki/Docs/howto/mysql_backup_script


==cannot send list of active checks==
Most likely ServerActive is defined in agent config, while not used at all


==Latest 20 issues==
 
== IPMI ==
 
=== IPMI Monitoring account for zabbix ===
 
[https://www.thomas-krenn.com/en/wiki/Configuring_IPMI_under_Linux_using_ipmitool https://www.thomas-krenn.com/en/wiki/Configuring_IPMI_under_Linux_using_ipmitool]
 
ipmitool user set name 3 monitor
ipmitool user set password 3
ipmitool channel setaccess 1 3 link=on ipmi=on callin=on privilege=2
ipmitool user enable 3
 
 
 
===Zabbix credentials===
Authentication algorithm
Default?
 
=== cannot connect to IPMI host: [22] Operation canceled ===
 
Usually temporary because of broken ipmi lib, ignore it
 
&nbsp;
 
=== cannot connect to IPMI host: [16777411] Unknown error 16777411 ===
 
classic, probably authentication problem
 
=== cannot connect to IPMI host: [22] Invalid argument ===
 
== zabbix_sender ==
 
=== processed: 0; failed: 1 ===
 
Possible causes:
 
*incorrect hostname
*incorrect item key
*item not in the server configuration cache yet
*Allowed hosts in trapper item
*phase of moon
*aliens
 
&nbsp;
 
=== Testing zabbix_sender ===
 
zabbix_sender stuff
 
== Filters ==
 
The regular expressions referred to in discovery are found under Administration->General, and then "Regular expressions" in the dropdown at top right of the page
 
=== cannot connect to IPMI host: [125] Operation canceled ===
 
possibly authentication method issue
 
 
&nbsp;
== Calculated items ==
See [https://blog.zabbix.com/zabbix-monitoring-with-calculated-items-explained/9950/ Calculated items explained]
=== Cannot create item: Invalid first parameter ===
For calculated items use last("youritemkey")
 
=== Cannot create item, error in formula ===
Problably a calculated item, try doublequoting the item key:
 
last("foo[bar]")
 
===Invalid parameter "/1/params"===
Maybe forgot to use last()?
You might need to doublequote your items, or prepend with double slashes
 
 
== Reset trigger/alert ==
For example when you changed the settings
Just disable, wait a bit and enable again.
== Install recent zabbix on CentOS/RHEL ==
 
rpm -ivh [https://repo.zabbix.com/zabbix/3.4/rhel/7/x86_64/zabbix-release-3.4-2.el7.noarch.rpm https://repo.zabbix.com/zabbix/3.4/rhel/7/x86_64/zabbix-release-3.4-2.el7.noarch.rpm]
yum install zabbix-agent
 
 
== Backing up tables ==
 
[https://www.zabbix.org/wiki/Docs/howto/mysql_backup_script https://www.zabbix.org/wiki/Docs/howto/mysql_backup_script]
 
&nbsp;
 
== cannot send list of active checks ==
Verify hostname:
/usr/sbin/zabbix_agentd -t 'agent.hostname'
 
If in agent log: most likely ServerActive is defined in agent config, while not used at all
 
It is also possible agent is sending some active check to server while host is monitored via proxy.
 
In proxy/server log:
most likely '''Hostname''' in agent config does not match hostname used on server.
 
==cannot send list of active checks to "127.0.0.1": host [Zabbix server] not found==
??
 
== active check configuration update started to fail ==
 
??
 
== Latest 20 issues ==
 
DEFAULT_LATEST_ISSUES_CNT in/usr/share/zabbix/include/defines.inc.php
DEFAULT_LATEST_ISSUES_CNT in/usr/share/zabbix/include/defines.inc.php


==Zabbix unreachable poller processes more than 75% busy==
&nbsp;
 
== Zabbix unreachable poller processes more than 75% busy ==
 
Increase '''StartPollersUnreachable'''
Increase '''StartPollersUnreachable'''


==More than 100 items having missing data for more than 10 minutes==
&nbsp;
 
== Zabbix poller processes more than 75% busy ==
 
another mystery
 
== More than 100 items having missing data for more than 10 minutes ==
 
Could be high load. Also check Administration->Queue
Could be high load. Also check Administration->Queue


==Zabbix escalator processes more than 75% busy==
== Zabbix escalator processes more than 75% busy ==
 
probably high system load overall
probably high system load overall


==Check agent==
== Check agent ==
 
  zabbix_get -s my.host.com -k agent.version
  zabbix_get -s my.host.com -k agent.version


==ZBX_NOTSUPPORTED==
== ZBX_NOTSUPPORTED ==
 
Could be anything, enable logging on agent. It could be version mismatch. Check
Could be anything, enable logging on agent. It could be version mismatch. Check
  zabbix_get -s yourhost -k agent.version
  zabbix_get -s yourhost -k agent.version
If that works, you're calling for an undefined or unsupported key.
If that works, you're calling for an undefined or unsupported key.


==Incorrect trigger expression. Host "xx" does not exist or you have no access to this host.==
== Incorrect trigger expression. Host "xx" does not exist or you have no access to this host. ==
 
Means there's no related item.
Means there's no related item.


==zabbix_get returns nothing==
== zabbix_get returns nothing ==
 
best look at log on agent side
best look at log on agent side


==run playbook on single host==
== run playbook on single host ==
 
  ansible_playbook -l somehost somplay.yml
  ansible_playbook -l somehost somplay.yml


[[:Category:Monitoring]]
[[:Category:Monitoring]]


&nbsp;
== Zabbix server is not running: the information displayed may not be current ==
Might be selinux: [http://sysads.co.uk/2013/11/zabbix-server-running-alert/ http://sysads.co.uk/2013/11/zabbix-server-running-alert/]
&nbsp;
&nbsp;
&nbsp;
== Monitoring vmware ==
=== vmware.hv.cpu.usage[{$URL},{HOST.HOST}]" became not supported: Couldn't resolve host name ===
Set macro {$URL} to [https://your.ip/sdk/ https://your.ip/sdk/] (shouldn't discovery figure that out from {$HOST}&nbsp;?
     
=== Couldn't resolve host name ===
Sometimes it's a matter of waiting a few hours


==Zabbix server is not running: the information displayed may not be current==
&nbsp;
Might be selinux: http://sysads.co.uk/2013/11/zabbix-server-running-alert/


==vmware.hv.cpu.usage[{$URL},{HOST.HOST}]" became not supported: Couldn't resolve host name==
=== vmware events collector returned empty result ===
Set macro {$URL} to https://your.ip/sdk/ (shouldn't discovery figure that out from {$HOST} ?
 
???
 
=== No "vmware collector" processes started. ===
 
Check StartVMwareCollectors on server or proxy
 
&nbsp;
 
== unsupported item key ==


==unsupported item key==
This might mean it's expecting a value from the script you're calling.
This might mean it's expecting a value from the script you're calling.
  echo 1
  echo 1
remember: not supported is not disabled, server/proxy will try again after interval
=== became not supported: Not supported by Zabbix Agent ===
probably output by userparameter/script
== ansible or API not showing host groups ==


==ansible or API not showing host groups==
Permissions!! See Administration->User Groups
Permissions!! See Administration->User Groups


&nbsp;
== failed to update local proxy configuration copy: invalid field name "items.lastlogsize" ==
check everything&nbsp;:)
== Received value [11] is not suitable for value type [Numeric (unsigned)] and data type [Decimal] ==
This probably means the agent returned 1\n1
&nbsp;
== database is down: retrying in 10 seconds ==
try upping max_connections
== [Incorrect key file for table 'items'; try to repair it ==
Could be something /tmp related
&nbsp;
&nbsp;
== another network error, wait for 8 seconds ==
'''UnreachableDelay'''=8
&nbsp;
== failed: first network error ==
Setting '''Timeout '''in server configuration
also Timeout in agents?
&nbsp;
== no active checks on server ==
*Hostname in agent config (-kagent.hostname) must match name on server
*simple no connection possible? firewall?
???
&nbsp;
&nbsp;
== show cpu utilization ==
Monitoring->host->graphs
===fuzzytime on command line===
TS=lotsofseconds
#output in hours
echo $(( ($(date +%s) - $TS) / 3600 ))
==duplicate entry adding user/group==
Check table 'ids'
==Troubleshooting==
===zabbix_get: no route to host===
Check the firewall


==failed to update local proxy configuration copy: invalid field name "items.lastlogsize"==
[[Category:Monitoring]]
check everything :)

Latest revision as of 11:56, 26 March 2024

Links

Documentation

Triggers

Function str() searches for substrings

AND OR case

not, and and or operators are case-sensitive and must be in lowercase. They also must be surrounded by spaces or parentheses.


Installation

Installing Zabbix from git

git clone https://github.com/zabbix/zabbix.git
cd zabbix 
./bootstrap.sh
./configure --help
autoreconf -fvi

If you don't have a Makefile, try

./config.status Makefile

and then

./configure

again

Zabbix API


Zabbix agent paths

Ubuntu:

/etc/zabbix/zabbix_agentd.conf.d/
/etc/zabbix/zabbix_agentd.conf

Simple check


Zabbix and SNMP

Zabbix error codes

Z3005

Database issue

Items

Agent item keys

Item dialog

Units

  • B
  • uptime
  • unixtime
  • s

proc.mem

proc.mem[<name>,<user>,<mode>,<cmdline>,<memtype>]

name

??

cmdline

regex like php-fpm:

memtype


Item preprocessing

Preprocessing regular expressions

See Regular expressions: example

XML/xpath preprocessing

https://blog.zabbix.com/zabbix-xpath-preprocessing/7936/

NOTE xq -x does not want the number() bit


Incorrect value for field "Prev. time": a relative time is expected.

Prev. Time should be something like

now-30s

Windows performance counters

https://www.zabbix.com/documentation/current/en/manual/config/items/perfcounters

Templates

Template App MySQL

https://github.com/tiramiseb/zabbix-templates/blob/master/Template%20App%20MySQL.txt TODO shouldn't this be user zabbix?

mysql user account:

create user 'monitor'@'localhost' identified by auth_socket;
grant PROCESS,SHOW DATABASES,SHOW VIEW on *.* to 'monitor'@'localhost';
flush privileges;

Configuration

Zabbix agent active

On client

Have port 10051 open and:

ActiveServer zabbix.ser.ver

On server

Set Agent IP to 0.0.0.0

 

Zabbix and SQL

Find hosts with hostmacro defined

select h.host, m.macro, m.value from hosts h, hostmacro m where macro like '%FOO%' and h.hostid = m.hostid;


most frequent items in history_uint

select itemid,count(itemid) as freq from history_uint group by itemid order by freq desc limit 5;

and then

select name from items where itemid = whateveryoufind;

HOWTO

Define discovery filters

LLD with JSON

if you want multiple keys, use jsonpath like

$[?(@.share=='{#FSTYPE}' && @.name=='{#NAME}')].size.first()

testing jsonpath preprocessing

In Value paste valid json, then name {#NAME} value somevalue

Test trapper

Reset admin password

Mysql prompt:

select * from user where username='Admin';

bcrypt your new password:

htpasswd -nbBC 10 USER YOURPASSWORD|awk -F ':' '{ print $2 }'

Mysql prompt:

update user set passwd = 'your bcrypted pass' where userid = 1

FAQ

SERVER

Adjust loglevel

zabbix_server --runtime-control log_level_increase=trapper
      

 

Reload zabbix server configuration

You can't, but you might want

zabbix_server -c /etc/zabbix/zabbix_server.conf -R config_cache_reload

 

No media defined for user

The frontend does not match Zabbix database.

Probably version conflict between frontend and server

 

value cache working in low memory mode

Increase ValueCacheSize


Message from 1.2.3.4 is missing header. Message ignored.

 

 

PROXY

Zabbix Proxy

cannot send proxy data to server

empty string received

failed to update local proxy configuration copy: unexpected field "host_inventory.type"

Front end

Round numbers

Preprocessing javascript 2 decimals:

return Math.round(value* 100) / 100

0 decimals:

return Math.round(value)

Visable name vs hostname

Visible name: {HOST.NAME}

Hostname: {HOST.HOST}

Host IP: (as defined in Interface->IP/DNS) {HOST.CONN}

 

Acknowledge multiple items

Monitor->Problems apply filters, select all, mass update

 

No permissions to referred object or it does not exist!

Graph no longer exists. Probably items no longer discovered

Maybe you've been editing a template file, remember to replace template name everywhere

Cannot add host

??

Monitoring SNMP

Cannot find host interface on "xxxhost" for item key foo

Might mean you're trying to import an SNMP template before configuring SNMP for the host

 

 

No SNMP data

snmp_parse_oid(): cannot parse OID "IF-MIB::ifSpeed.3

Timeout while connecting

Could be wrong community string, remember delay when using proxy.

Agent side ping check

UserParameter=pingtime[*],fping -e $1|sed 's/^.*(\([0-9].*\) ms).*$/\1/g'
UserParameter=pingalive[*],fping $1|grep -q alive;echo $?

 

 


LLD/Discovery

Discover: value must be a JSON object

Could mean you need to escape slashes, check output with zabbix_get


Cannot create item: item with the same key already exists

make sure the key contains "{#SOMENAME}"

Discovery data example

Output of a discovery script should look like:

{"data":[
  {"{#VAR1}":"value11","#{VAR2":"value12"},
  {"{#VAR1}":"value21","#{VAR2":"value22"}
]}



IPMI

IPMI Monitoring account for zabbix

https://www.thomas-krenn.com/en/wiki/Configuring_IPMI_under_Linux_using_ipmitool

ipmitool user set name 3 monitor
ipmitool user set password 3
ipmitool channel setaccess 1 3 link=on ipmi=on callin=on privilege=2
ipmitool user enable 3


Zabbix credentials

Authentication algorithm

Default?

cannot connect to IPMI host: [22] Operation canceled

Usually temporary because of broken ipmi lib, ignore it

 

cannot connect to IPMI host: [16777411] Unknown error 16777411

classic, probably authentication problem

cannot connect to IPMI host: [22] Invalid argument

zabbix_sender

processed: 0; failed: 1

Possible causes:

  • incorrect hostname
  • incorrect item key
  • item not in the server configuration cache yet
  • Allowed hosts in trapper item
  • phase of moon
  • aliens

 

Testing zabbix_sender

zabbix_sender stuff

Filters

The regular expressions referred to in discovery are found under Administration->General, and then "Regular expressions" in the dropdown at top right of the page

cannot connect to IPMI host: [125] Operation canceled

possibly authentication method issue


 

Calculated items

See Calculated items explained

Cannot create item: Invalid first parameter

For calculated items use last("youritemkey")

Cannot create item, error in formula

Problably a calculated item, try doublequoting the item key:

last("foo[bar]")

Invalid parameter "/1/params"

Maybe forgot to use last()? You might need to doublequote your items, or prepend with double slashes


Reset trigger/alert

For example when you changed the settings Just disable, wait a bit and enable again.

Install recent zabbix on CentOS/RHEL

rpm -ivh https://repo.zabbix.com/zabbix/3.4/rhel/7/x86_64/zabbix-release-3.4-2.el7.noarch.rpm
yum install zabbix-agent


Backing up tables

https://www.zabbix.org/wiki/Docs/howto/mysql_backup_script

 

cannot send list of active checks

Verify hostname:

/usr/sbin/zabbix_agentd -t 'agent.hostname'

If in agent log: most likely ServerActive is defined in agent config, while not used at all

It is also possible agent is sending some active check to server while host is monitored via proxy.

In proxy/server log: most likely Hostname in agent config does not match hostname used on server.

cannot send list of active checks to "127.0.0.1": host [Zabbix server] not found

??

active check configuration update started to fail

??

Latest 20 issues

DEFAULT_LATEST_ISSUES_CNT in/usr/share/zabbix/include/defines.inc.php

 

Zabbix unreachable poller processes more than 75% busy

Increase StartPollersUnreachable

 

Zabbix poller processes more than 75% busy

another mystery

More than 100 items having missing data for more than 10 minutes

Could be high load. Also check Administration->Queue

Zabbix escalator processes more than 75% busy

probably high system load overall

Check agent

zabbix_get -s my.host.com -k agent.version

ZBX_NOTSUPPORTED

Could be anything, enable logging on agent. It could be version mismatch. Check

zabbix_get -s yourhost -k agent.version

If that works, you're calling for an undefined or unsupported key.

Incorrect trigger expression. Host "xx" does not exist or you have no access to this host.

Means there's no related item.

zabbix_get returns nothing

best look at log on agent side

run playbook on single host

ansible_playbook -l somehost somplay.yml

Category:Monitoring

 

Zabbix server is not running: the information displayed may not be current

Might be selinux: http://sysads.co.uk/2013/11/zabbix-server-running-alert/

 

 

 

Monitoring vmware

vmware.hv.cpu.usage[{$URL},{HOST.HOST}]" became not supported: Couldn't resolve host name

Set macro {$URL} to https://your.ip/sdk/ (shouldn't discovery figure that out from {$HOST} ?
      

Couldn't resolve host name

Sometimes it's a matter of waiting a few hours

 

vmware events collector returned empty result

???

No "vmware collector" processes started.

Check StartVMwareCollectors on server or proxy

 

unsupported item key

This might mean it's expecting a value from the script you're calling.

echo 1

remember: not supported is not disabled, server/proxy will try again after interval

became not supported: Not supported by Zabbix Agent

probably output by userparameter/script

ansible or API not showing host groups

Permissions!! See Administration->User Groups

 

failed to update local proxy configuration copy: invalid field name "items.lastlogsize"

check everything :)

Received value [11] is not suitable for value type [Numeric (unsigned)] and data type [Decimal]

This probably means the agent returned 1\n1

 

database is down: retrying in 10 seconds

try upping max_connections

[Incorrect key file for table 'items'; try to repair it

Could be something /tmp related

 

 

another network error, wait for 8 seconds

UnreachableDelay=8

 

failed: first network error

Setting Timeout in server configuration

also Timeout in agents?

 

no active checks on server

  • Hostname in agent config (-kagent.hostname) must match name on server
  • simple no connection possible? firewall?

???

 

 

show cpu utilization

Monitoring->host->graphs

fuzzytime on command line

TS=lotsofseconds
  1. output in hours
echo $(( ($(date +%s) - $TS) / 3600 ))

duplicate entry adding user/group

Check table 'ids'


Troubleshooting

zabbix_get: no route to host

Check the firewall