6.0 CERTIFIED SPECIALIST TRAINING DAY 1 Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved COPYRIGHT NOTICE Rules It is prohibited to make any video and/or audio recordings during the whole period of this course. This course is intended only for the officially enrolled student. Subject to the Copyright Notice below, the student is not allowed to share his credentials for attending this course, to allow others to join and take part, or otherwise make use of these Materials. Copyright notice © Zabbix, 2022. All rights reserved. Unless otherwise indicated, Zabbix owns the copyright and other intellectual property rights in the text, graphics, information, designs, data, verbal/audio/video presentations and files, comments, drawings, exam questions and exam answers, and other training content, lab manuals and practical tasks, and training courses themselves (further – Materials). The Materials are protected by watermarks, copyright statements, and other means. It is prohibited to remove any of watermarks and copyright statements, or in any other way to amend or change the content or appearance of the Materials. Any unauthorized reprint, publication, reproduction, sharing, or use of the Materials is prohibited. No part of the Materials may be reproduced, transmitted, or published in any form or by any means, electronic or mechanical, including photocopying, recording, or by any information storage and retrieval system without the express signed written permission from Zabbix. All course Materials made available to the student during the course of the training may be used solely by the student enrolled in the relevant course for personal and educational purposes only. Materials provided to the student should be treated as confidential information shared with the student only for the purpose of the student performing Zabbix Certified training. The student acknowledges that damages alone would not be an adequate remedy for the breach of this copyright and the student shall be entitled to the granting of equitable relief concerning any threatened or actual breach of any of the provisions of this Copyright notice. Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 2 GETTING STARTED Introduction Background/company Experience with CLI Unix-like systems Experience with Zabbix or other monitoring solutions Current Zabbix deployments All students are required to turn on their web cameras and use the name provided during training registration for the trainer to track your attendance. Questions at any moment are encouraged! We suppose that attendees of this course have basic Linux knowledge. For the practical tasks use the "Lab manual" - it provides all required details. Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 3 DAILY SCHEDULE Time Monday - Thursday 09:00 - 10:30 Zabbix 6.0 Certified Specialist 10:30 - 10:45 Coffee break 10:45 - 12:00 Zabbix 6.0 Certified Specialist 12:00 - 13:00 Lunch break 13:00 - 14:30 Zabbix 6.0 Certified Specialist 14:30 - 14:45 Coffee break 14:45 - 16:00 Zabbix 6.0 Certified Specialist 16:00 - 16:15 Coffee break 16:30 - 18:00 Zabbix 6.0 Certified Specialist Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Friday Certification Theory 4 AGENDA Introduction Components User interface Zabbix architecture Zabbix interfaces Data collection Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Installation Hosts and Host groups Zabbix agent 5 Introduction Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved 15 minutes 6 FUNCTIONALITY Zabbix is the ultimate enterprise - class monitoring platform IPMI monitoring Web monitoring Event correlation User roles and permissions Tags Java monitoring SNMP monitoring Pro-active monitoring Agent auto registration Network discovery Visualization Real-time monitoring Low Level Discovery Trend prediction Zabbix 6.0 Certified Specialist ● Day 1 Flexibility AD authentication Database monitoring Inventory Pre-processing API Native agents Encryption Aggregate monitoring Alerting Bulk metric collection IPv6 Problem detection SLA monitoring Templates Distributed monitoring © 2022 by Zabbix. All rights reserved Theory 7 WHAT TO MONITOR Zabbix offers solutions for different application areas and use cases: Hardware monitoring Infrastructure resources Sensors Industrial serial devices CLOUDS CONTAINERS OS level monitoring VIRTUAL MACHINES Available resources Service statuses Log file entries SERVICES HARDWARE Virtual machine and container monitoring Guest VM, hypervisors and datastores Docker, Podman, etc. Cloud monitoring APPLICATIONS DATABASES Application monitoring Business service monitoring Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 8 METRIC COLLECTION Zabbix supports wide range of metric collection: Using Zabbix agents Simple ping and port checks SNMP(v1, v2c, v3), SNMP traps HTTP checks and Web monitoring IPMI and SSH checks Database monitoring via ODBC Java applications monitoring Aggregation and calculated checks VMware monitoring of VM's, Hypervisors and Datastores Custom metrics/scripts and more Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 9 ABOUT ZABBIX Zabbix 1.0 was released 23 March 2004 Every 1.5 years a new LTS (long time support) version is released A major version with new and hot features is released every 6 months Patches/bugfixes are released monthly Zabbix is used by the world's biggest telco, finance, educational, retail, and healthcare companies Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 10 ZABBIX COMPANY The Zabbix LLC was established in 2005 There are developers, test engineers, integrators, technical consultants, HR, sales, marketing, and partnership departments working all around the globe Zabbix company offices are open worldwide: Europe USA Japan Latin America Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 11 DEVELOPMENT Zabbix is Open Source and comes at no cost: Released under the GPL (General Public License) version 2 Free for commercial and non-commercial use No limitations on the number of monitored devices The entire source code is fully open and available for everyone Key principles of development: Develop very high performance and high-quality product Have a low number of third-party dependencies Be efficient - use as few system resources as possible Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 12 PROGRAMMING LANGUAGES Different programming languages are used for the source code C language - to provide the best performance Zabbix server Zabbix proxy Zabbix agent Go language - to provide more options for plugin developers Zabbix agent 2 Zabbix web services PHP language - to provide platform independent frontend Zabbix frontend Java language - to communicate with Java applications natively Zabbix Java gateway Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 13 COMMERCIAL SERVICES Zabbix LLC and Zabbix certified partners offers commercial services: Technical support different tiers, 24x7 support Remote troubleshooting quickly get a solution Consulting get advice from Zabbix experts Upgrade minimize downtime and risks Training programs obtain valuable knowledge Turn-Key solutions deploy your instance professionally Integration interconnect Zabbix with other tools Template building effortlessly start monitoring your specific device Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 14 Zabbix components Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved 30 minutes 15 MAIN COMPONENTS Zabbix has three main components: Zabbix server Data collection Problem detection Notification Zabbix frontend Configuration management Visualization Zabbix database Configuration data Historical values Frontend Database Server Configuration Collected metrics Internal communications All components can be installed on the same or separate machines Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 16 ADDITIONAL COMPONENTS Additional Zabbix components: Zabbix Proxy Monitor remote locations Distribute load between proxies Zabbix Java gateway Monitor Java applications Can be attached to Zabbix server or Zabbix proxy Java gateway Zabbix web service Generate scheduled reports Zabbix web service Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Server Proxy Theory 17 ZABBIX SERVER Zabbix server is the central process of Zabbix software Central component to which Zabbix agents and proxies report data The server can collect data from networked devices by itself Zabbix server processes all collected data: Pre-processes received values (if required) Detects problems and sends out alerts Transfers collected values into a database Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 18 ZABBIX SERVER COMPONENTS Zabbix server has several components: Data collectors (pollers, trappers, pingers, etc.) Multiple instances of data collectors can be started (poller #1, poller #2, poller #3, etc.) Internal processes (history syncer, task manager, etc.) Some of the processes are divided into manager and worker processes Multiple instances of the same process can be started (alerter #1, alerter #2, alerter #3, etc.) Caches (Configuration cache, history cache, value cache, etc.) 26011 /usr/sbin/zabbix_server -c /etc/zabbix/zabbix_server.conf 26012 /usr/sbin/zabbix_server: configuration syncer [synced configuration in 3.18 sec, idle 60 sec] 26013 /usr/sbin/zabbix_server: preprocessing manager #1 [queued 0, processed 56 values, idle 5.06 sec] 26018 /usr/sbin/zabbix_server: preprocessing worker #1 started 26019 /usr/sbin/zabbix_server: preprocessing worker #2 started 26024 /usr/sbin/zabbix_server: housekeeper [deleted 0 hist/trends, 165 items/triggers, 56474 events, 0 sessions, 0 alarms, 4 audit items, 0 records in 7.567507 sec, idle for 1 26025 /usr/sbin/zabbix_server: timer #1 [updated 0 hosts, suppressed 0 events in 0.004086 sec, idle 59 sec] 26026 /usr/sbin/zabbix_server: http poller #1 [got 0 values in 0.000625 sec, idle 5 sec] 26028 /usr/sbin/zabbix_server: history syncer #1 [processed 0 values, 0 triggers in 0.000028 sec, idle 1 sec] 26029 /usr/sbin/zabbix_server: history syncer #2 [processed 3 values, 1 triggers in 0.002814 sec, idle 1 sec] 26036 /usr/sbin/zabbix_server: poller #1 [got 12 values in 0.031888 sec, getting values] 26037 /usr/sbin/zabbix_server: poller #2 [got 3 values in 0.023746 sec, idle 1 sec] Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 19 ZABBIX SERVER INTERNAL ARCHITECTURE (SIMPLIFIED) Configuration cache Data collectors Preprocessing History cache Configuration syncer History syncers Value cache Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 20 DATABASE ENGINES The following database engines are supported: MySQL / Percona (8.0.x) MariaDB (10.5.x - 10.8.x) PostgreSQL (13.x,14.x) TimescaleDB (2.0.1- 2.7.x) Oracle (19c-21c) SQLite3 (for proxies and agent2 only) Use recommended versions if possible: Offers the best performance, stability, and reliability Older versions may be no longer supported by the database developers Elasticsearch is supported as an alternative history storage (experimental) Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 21 SERVER/PROXY REQUIREMENTS Zabbix server/proxy can be installed on any *nix operational system: Linux RedHat / CentOS / Oracle Linux Debian / Ubuntu / Raspbian SUSE Linux Solaris, AIX, HP-UX FreeBSD, OpenBSD MacOS Other Some additional libraries will be required: Functionality Library Preprocessing libpcre Bulk metrics libevent Traffic compression zlib IPMI checks OpenIPMI SSH checks libssh SNMP checks net-snmp ICMP ping fping Web monitoring libcurl VMWare monitoring libxml2 Encryption openssl-libs libpcre, libevent and zlib libraries are always required Other libraries are optional if Zabbix is built from sources without some of the functionality Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 22 ZABBIX FRONTEND Zabbix frontend is used to: Display and modify Zabbix monitoring configuration Display collected values using dashboards, graphs, maps etc. Display problems detected by Zabbix server Changes made in the frontend are saved into Zabbix database: Frontend will display latest configuration immediately Zabbix server reads monitoring configuration changes once per minute (by default) It is possible to force Zabbix server read configuration with a runtime command immediately Zabbix 6.0 Certified Specialist ● Day 1 1 minute © 2022 by Zabbix. All rights reserved Theory 23 ZABBIX AGENT Zabbix agent is a process deployed on monitoring targets: May run on various supported platforms (Linux, UNIX, macOS, Windows, etc.) Collects data from a device or an application Has very low memory footprint and resource usage Can work in the passive or active mode (simultaneously) Supports native communication encryption Passive mode Active mode Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 24 ZABBIX COMMUNICATION All Zabbix components use TCP communication (JSON RPC protocol) Zabbix agent listens on port 10050 (in passive mode) Server and proxy listens on port 10051 Zabbix Java gateway listens on port 10052 Zabbix web service listens on port 10053 10052 10051 Zabbix 6.0 Certified Specialist ● Day 1 10050 10051 © 2022 by Zabbix. All rights reserved 10053 Theory 25 Zabbix proxies Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved 15 minutes 26 ZABBIX PROXY Zabbix proxy collects data on behalf of Zabbix server: Proxies will get monitoring configuration from Zabbix server automatically Collected data is sent to Zabbix server every second by default All data is stored into a local proxy database for a limited amount of time The data is still collected by the proxy in case of network issues between the server and proxy Proxies preprocess data (if required) to offload Zabbix server resources Zabbix proxy Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 27 PROXY BENEFITS Very useful to monitor hosts/devices behind firewalls or in remote locations Communication is faster between proxies and monitored hosts Collected data is sent in bulk in a single transmission Only single TCP port is required for communication between server and proxy Supports native communication encryption One firewall rule Multiple firewall rules With Zabbix proxy Zabbix 6.0 Certified Specialist ● Day 1 Without Zabbix proxy © 2022 by Zabbix. All rights reserved Theory 28 PROXY MODES Zabbix proxies can operate in one of the two modes: Active mode proxy is connecting to server Passive mode server is connecting to proxy Preferred proxy mode depends on the network configuration Proxies in both modes can be used by the same Zabbix server instance Monitored hosts Active Proxy Active mode Server Passive mode Remote site A Zabbix 6.0 Certified Specialist ● Day 1 Passive Proxy Monitored hosts Remote site B © 2022 by Zabbix. All rights reserved Theory 29 PROXY COMPATIBILITY Zabbix proxy major version must match Zabbix server major version: Zabbix proxy 6.0.0 and Zabbix server 6.0.1 are compatible Zabbix proxy 5.0.9 and Zabbix server 6.0.1 are not compatible An unlimited number of proxies can be deployed per Zabbix server instance: Proxies may be placed in different time zones, it will not affect monitoring London GMT +0 New York GMT -5 San Francisco GMT -8 Paris GMT +1 Tokyo GMT +9 Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 30 Zabbix installation Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved 30 minutes 31 OS PREPARATIONS It is recommended to prepare OS before installing Zabbix: Setup an NTP client: It is mandatory to have precise time on all systems with Zabbix components Unsynchronized time may lead to false problem detection, missing data and other issues Set up appropriate time zone on Zabbix server: Zabbix server time zone will be used for some of the functionality (repeating maintenance periods, time-based functions, reports, etc.) Zabbix log file entries will have timestamps in this time zone Setup firewall rules: Zabbix server will have a lot of outgoing/incoming connections to different network devices A firewall must be configured to allow this interaction Built-in access control systems may block Zabbix server components SELINUX, AppArmor Other security tools Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 32 DOWNLOAD ZABBIX There are multiple ways of getting Zabbix: Install it from the official packages Use Zabbix Cloud images Deploy from container images Download the virtual appliance Download the latest source archive and compile it yourself i https://www.zabbix.com/download Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 33 ZABBIX PACKAGES Choose the correct package for your OS: Zabbix version OS distribution OS version Database type Web server Instructions on the web page will change according to the selected options Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 34 ZABBIX CLOUD IMAGES Select the preferred cloud provider: Zabbix will be deployed on cloud using predefined settings Latest versions may not be available on some cloud providers i https://www.zabbix.com/cloud_images Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 35 DOCKER CONTAINERS Select and download the docker component: Choose between Zabbix docker hub or GitHub These are official docker files built by Zabbix LLC i https://www.zabbix.com/container_images Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 36 ZABBIX APPLIANCE Download the official Zabbix appliance: Zabbix software is pre-installed and pre-configured for a trouble-free deployment Evaluate Zabbix using your preferred virtualization platform The Appliance can be used for test environments i https://www.zabbix.com/download_appliance Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 37 ZABBIX SOURCE CODE Deploy Zabbix from the original source code: The latest source code is available in Zabbix Git repository or on the download page Zabbix binaries can be built using your preferred libraries and options The only way to deploy Zabbix server with Oracle database Building Zabbix from sources is included in the Zabbix Certified Expert course! i https://www.zabbix.com/documentation/6.0/manual/installation/install Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 38 MAIN ZABBIX PACKAGES Packages required for Zabbix installation (RHEL package name examples): zabbix-release-6.0 Zabbix 6.0 repository zabbix-server-mysql Zabbix server for MySQL database zabbix-server-pgsql Zabbix server for PostgreSQL database zabbix-sql-scripts Zabbix database schema zabbix-web-mysql Zabbix frontend for MySQL database zabbix-web-pgsql Zabbix frontend for PostgreSQL database zabbix-apache-conf Configuration settings for Apache web server zabbix-nginx-conf Configuration settings for Nginx web server ! Choose the correct OS, database engine, and web server when installing Zabbix! i https://repo.zabbix.com/zabbix/6.0/ Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 39 ADDITIONAL ZABBIX PACKAGES Additional packages provided by Zabbix: Functionality Package name Zabbix agent zabbix-agent zabbix-agent2 Zabbix proxy zabbix-proxy-mysql zabbix-proxy-postgresql zabbix-proxy-sqlite3 Zabbix Java gateway zabbix-java-gateway Zabbix web service zabbix-web-service Command-line utilities zabbix-get zabbix-sender zabbix-js SELINUX policies zabbix-selinux-policy Some OS distributions may provide their own Zabbix packages: This is more common to the Debian-based distributions Such packages are not supported by Zabbix, they are typically out-of-date It is recommended to use only official packages from https://repo.zabbix.com ! Zabbix web service is used only to generate scheduled reports! Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 40 Zabbix server installation Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved 20 minutes 41 ZABBIX SERVER INSTALLATION Installing Zabbix server from packages is the easiest way: Pre-built packages are available for most OS families, web servers and databases Debian Raspbian RHEL (Red Hat, Centos, Oracle Linux, AlmaLinux etc.) SLES Ubuntu Simple to install using package manager Zabbix can be automatically updated to the latest version Steps to deploy Zabbix server from packages: Install Zabbix official repository (use latest 6.0 release version) Use package manager (dnf, apt, zypper, etc.) to install required Zabbix components # dnf install https://repo.zabbix.com/zabbix/6.0/rhel/8/x86_64/zabbix-release-6.0-2.el8.noarch.rpm # dnf install zabbix-server-mysql zabbix-sql-scripts Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 42 DATABASE INSTALLATION Zabbix server can not work without a preconfigured database: Install and start preferred database engine Create Zabbix database with proper parameters: Zabbix requires a Unicode database character set and a UTF8 national character set Database which is created with incorrect settings will require downtime later to fix the issue Create database user with full access to the database: If a remote database is used, make sure zabbix user can access the database over the network MySQL # mysql mysql> create database zabbix character set utf8mb4 collate utf8mb4_bin; mysql> create user 'zabbix'@'localhost' identified by 'P455w0RD'; mysql> grant all privileges on zabbix.* to 'zabbix'@'localhost'; mysql> quit; PostgreSQL # sudo -u postgres createuser --pwprompt zabbix # sudo -u postgres createdb -O zabbix -E Unicode zabbix Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 43 ZABBIX DATABASE SCHEMA Zabbix database schema is required: Schema is delivered in a separate package zabbix-sql-scripts Database schema (Tables) Initial configuration (Zabbix server host, official templates, users, etc.) Pictures MySQL # cd /usr/share/zabbix-sql-scripts/mysql # zcat server.sql.gz | mysql -uzabbix -p zabbix PostgreSQL # cd /usr/share/zabbix-sql-scripts/postgresql # zcat create.sql.gz | sudo -u zabbix psql zabbix Zabbix will not start without the initial database schema 084221.231 using configuration file: /etc/zabbix/zabbix_server.conf 084221.237 query failed: Table 'zabbix.users' doesn't exist [select userid from users limit 1] 084221.237 cannot use database "zabbix": database is not a Zabbix database Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 44 ZABBIX SERVER CONFIGURATION Minimum requirements in the configuration file to start Zabbix server: Database host address Database name Database schema (only if a specific PostgreSQL schema is used) Database user Database password ### Option: DBHost DBHost=mysql.example.com ### Option: DBName DBName=zabbix ### Option: DBSchema # DBSchema= ### Option: DBUser DBUser=zabbix ### Option: DBPassword DBPassword=P455w0RD Zabbix 6.0 Certified Specialist ● Day 1 zbx.example.com © 2022 by Zabbix. All rights reserved mysql.example.com Theory 45 SERVER COMMANDS Zabbix server binary supports command-line parameters: More information (help) # zabbix_server -h # zabbix_server --help Zabbix server version # zabbix_server -V # zabbix_server --version Start with a different configuration file # zabbix_server -c /tmp/test_zabbix_server.conf # zabbix_server --config /tmp/test_zabbix_server.conf Run in the foreground # zabbix_server -f # zabbix_server --foreground Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 46 ZABBIX SERVER RUNTIME COMMANDS Zabbix server supports runtime controls: Runtime control syntax zabbix_server -R <option> zabbix_server --runtime-control <option> Reload configuration cache on demand # zabbix_server -R config_cache_reload Execute housekeeper # zabbix_server -R housekeeper_execute Reload SNMP cache # zabbix_server -R snmp_cache_reload Reload Vault secrets # zabbix_server -R secrets_reload Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 47 DIAGNOSTIC INFORMATION Diagnostic information can be collected and displayed in shell: For all processes # zabbix_server -R diaginfo For a single process or cache # zabbix_server -R diaginfo=historycache == history cache diagnostic information == Items:0 values:0 time:0.000011 Memory.data: size: free:16776832 used:0 chunks: free:1 used:0 min:16776832 max:16776832 buckets: 256+:1 Memory.index: size: free:4176368 used:17440 chunks: free:2 used:5 min:48 max:4176320 buckets: 48:1 256+:1 ......................................... Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 48 LOG LEVELS Zabbix components have 6 debug (log) levels: 0 - basic information about starting and stopping of Zabbix processes 1 - critical information 2 - error information 3 - warnings (default level - recommended) 4 - for debugging (produces lots of information) 5 - extended debugging (produces even more information) Default debug level is set in the Zabbix server configuration file: 3 is the recommended level for production environment ### Option: DebugLevel # Specifies debug level: DebugLevel=3 ! A Zabbix server restart is required if changing log level in the configuration file Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 49 LOG CONFIGURATION OPTIONS It is possible to adjust log options in Zabbix server configuration file: Where to write the log file ### Option: LogType: # system - syslog # file - file specified with LogFile parameter # console - standard output LogType=file Log file location ### Option: LogFile # Log file name for LogType 'file' parameter. # Mandatory only if LogType is set to file LogFile=/var/log/zabbix/zabbix_server.log Log rotation options Log file is moved to zabbix_server.log.old when the maximum size is exceeded ### Option: LogFileSize # Maximum size of log file in MB. # 0 - disable automatic log rotation. LogFileSize=512 Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 50 ZABBIX SERVER LOG LEVEL Log level for Zabbix server can be increased using the runtime controls: Increase/decrease the log level for all processes # zabbix_server -R log_level_increase # zabbix_server -R log_level_decrease It is possible to specify for which process to change the log level Increase the log level for a process type # zabbix_server -R log_level_increase=poller Increase the log level for a single process # zabbix_server -R log_level_increase="history syncer,2" Decrease the log level using process number (will not work with the 6-digit PID) # zabbix_server -R log_level_decrease=54362 Always set the log level back to the recommended (3 - warnings) after troubleshooting ! Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 51 LOG FILE FORMAT Zabbix components have a strict log file format: It is easy to track which process created the log entry at what time <Process id>:<Date yyyymmdd>:<Time hhmmss.ms> <information> PID Date Time Information 6621:20210927:143601.621 server #29 started [poller #4] 6622:20210927:143601.785 server #30 started [poller #5] 6623:20210927:143601.787 server #31 started [unreachable poller #1] 6624:20210927:143601.256 server #32 started [icmp pinger #1] 6625:20210927:143601.788 server #33 started [trapper #2] 6626:20210927:143601.802 server #34 started [availability manager #1] 6627:20211005:143601.903 server #35 started [housekeeper #1] 6621:20210927:143602.747 Zabbix agent item "system.cpu.util" on host "Zabbix server" failed: ... 6622:20210927:143617.789 Zabbix agent item "system.cpu.switches" on host "Zabbix server" failed: ... 6622:20210927:143632.791 Zabbix agent item "system.cpu.load on host "Zabbix server" failed: ... 6621:20210927:143641.892 item "zabbix[process,ipmi poller,avg,busy]" became not supported: ... 6622:20210927:143642.899 item "zabbix[process,java poller,avg,busy]" became not supported: ... 6624:20210927:143618.841 fping failed: dev-server.example.com: Name or service not known 6627:20210927:150601.670 server #14 started [housekeeper #1] Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 52 PRACTICAL SETUP 1) Setup operational system for Zabbix installation: Setup the correct time zone for operating system Check that SELinux and firewall are disabled 2) Create Zabbix database: Install the latest MySQL 8.0 server package and start the database engine Create a Zabbix database and set up database user Install Zabbix 6.0 official repository Upload Zabbix DB scheme to MySQL database, check for correct upload 3) Install Zabbix server 6.0: Set up required Zabbix settings in the configuration file Start Zabbix server. Check Zabbix server log file and service status Check Zabbix server version Check the process list for Zabbix processes Increase/decrease Zabbix log level for history syncer #3 Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved 45 minutes Practical task No: 1 53 Zabbix frontend installation Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved 30 minutes 54 FRONTEND REQUIREMENTS Zabbix frontend can be installed on any operational system: Web server Apache (1.3.12 or later) Nginx Other with PHP support (IIS, Tomcat etc.) PHP Version 7.4.0 or later is required PHP 8.x is supported since Zabbix 6.0.6 Some additional PHP libraries will be required: php-gd(2.0.28 or later), php-bcmath, php-ctype, php-xml(2.6.15 or later), php-xml, php-xmlreader, php-xmlwriter, php-session, php-net-socket, php-mbstring, php-gettext, php-mysql, php-pgsql or php-sqlora for database support php-ldap if LDAP authentication is used in the frontend php-openssl if SAML authentication is used in the frontend Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 55 WEB SERVER AND DATABASE Choose the correct database engine and web server package combination: zabbix-web-mysql or zabbix-web-postgresql zabbix-apache-conf or zabbix-nginx-conf Database engine Web server engine zabbix-web-mysql zabbix-apache-conf zabbix-web-postgresql zabbix-nginx-conf Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 56 ZABBIX FRONTEND INSTALLATION Zabbix frontend installation has two parts: Web server installation (from command line) Frontend setup wizard (GUI-based installation) Install the correct combination of frontend packages Start the web server when the packages are installed: Zabbix uses php-fpm (FastCGI Process Manager) by default on RHEL Additional PHP configuration is not required to start Zabbix frontend Web server and PHP parameters can be tuned later to match your environment After the web server and php-fpm are started, launch the setup wizard Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 57 BROWSER REQUIREMENTS The latest stable Zabbix versions are supported for the following browsers: Google Chrome Mozilla Firefox Microsoft Edge Apple Safari Opera Other browser requirements: Cookies and JavaScript must be enabled The minimum supported screen width for Zabbix frontend is 1200px Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 58 FRONTEND SETUP WIZARD Open http://<DNS or IP>/zabbix address in a web browser: Choose the default user interface language Check prerequisites (all must be OK) Configure Zabbix DB connection Enter Zabbix server name Choose the default time zone Review summary Finish the setup Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 59 FRONTEND LANGUAGE Zabbix frontend language: Default locale is defined in the setup wizard Additional locales need to be installed on the OS of a web server Language can be redefined per user in the user profile Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 60 FRONTEND PREREQUISITES All prerequisites must be OK: Installation can not continue with any failed pre-requisites: If a prerequisite is not met, fix the problem and restart php-fpm service Memory limit too low Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 61 ZABBIX DATABASE DETAILS Enter Zabbix database details: Choose the correct database type Enter the database host name and credentials The database connection can be encrypted (optional) Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 62 ZABBIX SERVER DETAILS Enter additional settings: Zabbix server name Default time zone for Zabbix frontend Default theme for all users Zabbix server name Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 63 ZABBIX THEMES Zabbix offers four built-in themes: Blue (standard) Dark High contrast light High contrast dark Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 64 PRE-INSTALLATION SUMMARY Summary page is displayed with all parameters: Database connection settings Zabbix server name Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 65 FRONTEND CONFIGURATION FILE Frontend settings are stored in the configuration file: Located at /usr/share/zabbix/conf/zabbix.conf.php Symlink created in /etc/zabbix/web/zabbix.conf.php // Zabbix GUI configuration file $DB['TYPE'] $DB['SERVER'] $DB['PORT'] $DB['DATABASE'] $DB['USER'] $DB['PASSWORD'] = 'MYSQL'; = 'mysql.example.com'; = '3306'; = 'zabbix'; = 'zabbix'; = 'P455w0RD'; // $ZBX_SERVER // $ZBX_SERVER_PORT $ZBX_SERVER_NAME = ''; = ''; = 'Zabbix Production'; The setup can be started again by opening http://<DNS or IP>/zabbix/setup.php Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 66 LOGIN SCREEN Default login credentials for a new Zabbix setup: Username: Admin Password: zabbix Cookie can be saved ! Default credentials are very unsafe and need to be changed! Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 67 PRACTICAL SETUP 1) Setup Zabbix frontend: Install Apache web server and Zabbix Web frontend Enable autostart for Apache and PHP-FPM manager and check service status 2) Complete the frontend setup wizard: Open http://<Training-VM-XX IP>/zabbix in your browser (use correct IP address) Check the prerequisites Setup all required parameters for a DB connection Use UTC as a default time zone Provide instance name as "Zabbix server XX" where XX corresponds to the number of your VM 3) Log into Zabbix web interface: Username: Password: Zabbix 6.0 Certified Specialist ● Day 1 Admin zabbix © 2022 by Zabbix. All rights reserved 15 minutes Practical task No: 2 68 Zabbix interface Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved 15 minutes 69 INTERFACE OVERVIEW Links Main menu Main window Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 70 MENU FUNCTIONALITY Menu functionality: Menu can be collapsed to small icons or hidden completely: Global search is used to find hosts, host groups and templates In the Monitoring section, frontend can be switched to kiosk mode by pressing To exit kiosk mode, point to the upper right corner and press Collapse or Hide Instance name Global search Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 71 GLOBAL SEARCH Type any string: If there is a host that includes the string, dropdown will appear, listing all such hosts Hosts Host Groups Templates Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 72 MENU SECTIONS Main menu has six sections: Monitoring: Dashboards, problems, latest data, etc. Services: Services and SLA settings Inventory: Overview of inventory data Reports: Predefined reports, scheduled reports and audit log Configuration: Monitoring setup Administration: Global settings, proxies, user management, scripts, etc. Links: Zabbix technical support information page Zabbix official integrations page https://www.zabbix.com/integrations Zabbix documentation https://www.zabbix.com/documentation/ Current user profile settings Log out current session Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 73 USER TYPES Zabbix has three built-in user types: Zabbix user limited user with access to collected data only Zabbix admin can change monitoring configuration Zabbix super admin provides access to global Zabbix settings and all hosts Access to the frontend menu sections depends on the user type: Zabbix User Zabbix 6.0 Certified Specialist ● Day 1 Zabbix Admin © 2022 by Zabbix. All rights reserved Zabbix Super Admin Theory 74 FILTERS Zabbix frontend pages may contain filters: Filter fields use AND logic to combine field values Some filter sections may have multiple fields with "AND/OR" or "OR" logic Or will use OR logic for all fields AND/OR will use OR logic for similar types and the And logic for different types Tags section with And/Or logic Show or hide filter by clicking on the filter icon Host group = Linux servers and Name contains "Database server" and Tags = ((component equals cpu or component equals memory) and database does not contain mssql) Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 75 NAMED FILTERS Some Zabbix frontend pages can save, and later reuse named filters: Pages with named filters will have multiple tabs: A tab with filter icon for the main filter Tabs for saved filters with a filter name Number of hosts can be displayed (optionally) on the tabs Filters are saved in the profile of current user, they are not available to other users Currently named filters are implemented for: Monitoring > Hosts Monitoring > Problems Monitoring > Latest data Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 76 TAB DISPLAY OPTIONS Zabbix configuration pages may have multiple tabs: Information on multiple tabs may be filled at once before adding / updating settings The currently active tab is highlighted For tabs which may contain multiple entries the number of entries is displayed If a tab has some settings enabled, it has a green dot Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 77 User profile Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved 15 minutes 78 USER SETTINGS User settings menu has two sections: Profile User tab Media tab Messaging tab API tokens User profile settings with three tabs General settings Media settings (Admins and SuperAdmins) Problem popup displaying options Tokens created to access Zabbix API Media tab is accessible to Admin and Super Admin users: Media is used to send out alerts Zabbix supports email, SMS, webhooks and other media types Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 79 USER SETTINGS Every Zabbix user has access to the following settings: Change own password Choose display language (from installed languages) Adjust the time zone Choose Zabbix theme Enable auto-login Logout after an inactivity period Auto refresh interval Used for Monitoring section 0 disables auto refresh Maximum rows per page Redirect after login Absolute or relative path supported Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 80 ZABBIX TIME FORMAT Zabbix always stores timestamps for all values in UNIXTIME: Number of seconds that have elapsed since the Unix epoch, 00:00:00 UTC, 1 Jan 1970 UNIXTIME is stored using UTC (GMT +0) time zone Frontend displays collected data using a time zone from the User profile: The date and time for collected value may be displayed differently for different users John Alice UNIXTIME 1629724769 Mon Aug 23 2021 15:19:29 Riga GMT +2 Zabbix 6.0 Certified Specialist ● Day 1 UNIXTIME 1629724769 Mon Aug 23 2021 13:19:29 UNIXTIME 1629724769 Mon Aug 23 2021 09:19:29 UTC (GMT +0) New York GMT -4 © 2022 by Zabbix. All rights reserved Theory 81 GLOBAL NOTIFICATIONS Global notifications are a way of displaying current problems: They will be displayed regardless of where you are in the Zabbix frontend Global notifications involve both showing a message and playing a sound. Only problems visible to the current user will be displayed Problems are displayed in a floating section on the right-hand side: This section can be repositioned freely by dragging the section header Several controls are available: Silence all currently active alarm sounds Mute/Unmute current and future alarm sounds Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 82 MESSAGING TAB Global notifications are configured in the Messaging section: Enable notifications Message display duration Play sound once, for 10 seconds or for the message timeout duration Define problem severities for which to play the sound: Select severities that are important Test the sounds from the frontend no_sound will disable the audio alert, but still display the message Display options for "suppressed" problems Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 83 PRACTICAL SETUP 1) Create a user group with the name "Training users" 2) Create a new Zabbix user for yourself with Super administrator role: You can use your name and surname to create login Set up user parameters: time zone, password, URL Use your local time zone Specify custom URL to open Monitoring > Hosts page after login 3) Change the default Admin password to P455w0RD 4) Log in with your username and password 5) Check your user profile settings Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved 10 minutes Practical task No: 3 84 Zabbix data flow Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved 20 minutes 85 DATA FLOW Zabbix data flow has multiple stages: Data collection Data are gathered from monitoring targets called Hosts Each host can collect hundreds of metrics using Items Data preprocessing Data are transformed if needed using Preprocessing steps defined in the item configuration Problem detection Each collected value is checked against problem conditions called Triggers and saved to DB On problem detection a trigger-based Event is created Alerting If a problem is detected, it is processed by Action if matched by its conditions Actions may perform multiple sequential commands called Operations Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 86 DATA FLOW DIAGRAM ACTION HOST ITEM TRIGGER EVENT CONDITIONS OPERATION ITEM ITEM OPERATION TRIGGER EVENT OPERATION ITEM Event may not match any action HOST ITEM TRIGGER ACTION CONDITIONS OPERATION EVENT ITEM ITEM Zabbix 6.0 Certified Specialist ● Day 1 OPERATION ACTION Trigger may belong to multiple hosts Event may match multiple actions © 2022 by Zabbix. All rights reserved CONDITIONS OPERATION Theory 87 ACTIONS An action is a set of operations which are executed when an event is created: Action is executed only if the event conditions met Operations are executed in the numbered order called operation steps Each operation step execution time can be scheduled Scheduled execution of operation steps is called escalation Operation steps can: Send messages (email, SMS, etc.) Execute remote commands ACTION TRIGGER EVENT CONDITIONS OPERATION 1 OPERATION 2 OPERATION 3 Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 88 TEMPLATES A template is a set of entities that can be applied to multiple hosts: Items, triggers and other entities can be created on a template This template can be linked to multiple similar hosts, which will inherit these entities Templates cannot be linked to a host group TEMPLATE HOST X HOST Y HOST Z ITEM A ITEM A ITEM A ITEM A ITEM B ITEM B ITEM B ITEM B ITEM C ITEM C ITEM C ITEM C TRIGGER 1 TRIGGER 1 TRIGGER 1 TRIGGER 1 Linkage Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 89 TAGS Tags are used to mark various entities in Zabbix: Hosts Templates Items Triggers Tagging is implemented as a pair of case sensitive tag name and value: You can use just the name or pair it with a value An entity may be tagged with the same name, but different values Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 90 TAG INHERITANCE If entities are tagged, the corresponding new events will inherit all tags: Host tags will be inherited by all items and triggers on the host An event will inherit all the host, item and trigger tags HOST Legend Host tag ITEM Item tag TRIGGER Zabbix 6.0 Certified Specialist ● Day 1 Event © 2022 by Zabbix. All rights reserved Trigger tag Theory 91 MACROS Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved 20 minutes 92 MACROS Macros resolve to a specific value depending on the context: Save time and make Zabbix configuration more flexible Can be used in item names, trigger expressions, operation steps, etc. Macros have specific syntax: The name must be written in UPPERCASE A macro must be included in curly brackets {MACRO} Zabbix supports four types of macros: Built-in macros {MACRO} User defined macros {$MACRO} Low-level discovery macros {#MACRO} Expression macros {?MACRO} Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 93 BUILT-IN MACROS Built-in macros are predefined names which will resolve to a value: Syntax: {MACRO} A full list of macros and supported locations is available in the documentation Must be used in a proper location, otherwise they will not resolve If a macro value is not resolved the macro name will be displayed instead Macro i Description Example output {HOST.HOST} Host name Zabbix server {HOST.IP} Host IP address 192.168.12.5 {ITEM.LASTVALUE} Last received metric for an item 3.523 {EVENT.NAME} Name of the problem event CPU load too high {MY.MACRO} Wrong macro name {MY.MACRO} https://www.zabbix.com/documentation/6.0/manual/appendix/macros/supported_by_location Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 94 USER MACROS User macros are user-defined variables: Syntax: {$USER.MACRO} If a macro value is not resolved the macro name will be displayed instead User macro values have three types: Text value is displayed in a plain text Secret text value is replaced with ******* in the frontend Vault secret only vault path is displayed in the frontend Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 95 USER MACRO LEVELS User macros can be defined on three levels: Global GLOBAL MACRO Administration > General > Macros Overridden by Template and Host macro values Template (recommended) TEMPLATE MACRO Configuration > Templates > {template} > Macros Overridden by Host macro values Host (change template macro value if needed) HOST MACRO Configuration > Hosts > {host} > Macros Macro {$CPU.LOAD.HIGH} Global Template Host Result 2 3 1 1 qwerty123! pa$$w0rd pa$$w0rd {$SSH.PASSWORD} {$SNMP.COMMUNITY} {$MEMORY.UTIL.MAX} public public 90 {$MY.MACRO} Zabbix 6.0 Certified Specialist ● Day 1 90 {$MY.MACRO} © 2022 by Zabbix. All rights reserved Theory 96 INHERITED MACROS Inherited macros are not displayed by default: Only macros defined (or redefined) on the host level are visible by default Switch to "Inherited and host macros" to display all macros in effect Template Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Global Theory 97 Host groups Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved 15 minutes 98 HOST GROUPS Hosts and Templates may have one or multiple Host groups: Host groups are used for logical grouping of hosts and templates A host group can contain both hosts and/or templates Each host or template must have at least one host group assigned Host or Template may belong to multiple host groups Each host group may have multiple hosts or templates assigned Host A Zabbix 6.0 Certified Specialist ● Day 1 Host B Host C Template X © 2022 by Zabbix. All rights reserved Template Y Template Z Theory 99 HOST GROUP PAGE Host group page contains: Host group name Number of hosts and templates in the group Clicking on Hosts or Templates will display the entries which belong to a group List of host group members Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 100 HOST GROUPS AND PERMISSIONS In Zabbix all access rights are based on user groups and host groups: read only User group Host group Users Hosts Even if a single user needs access to a single host it is done through groups: read write Admin Zabbix 6.0 Certified Specialist ● Day 1 User group Host group © 2022 by Zabbix. All rights reserved Host Theory 101 NESTED HOST GROUPS Zabbix supports nested host groups: Nested representation is accomplished by using the '/' forward slash It is not required to create the "parent" group to create nested groups Nested host groups can inherit access permissions from parent groups Logical representation Location Europe North America South America Representation in Zabbix London Paris New York Los Angeles Chicago Japan Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Location Location/Europe Location/Europe/London Location/Europe/Paris Location/North America/New York Location/North America/Los Angeles Location/North America/Chicago Location/South America Location/Japan Theory 102 CREATE HOST GROUP It is possible to create a Host group using one of the two options: Use Configuration > Host Groups > "Create Host group" button Create a host group on demand from the host configuration form Create a new host group Create a host group from the host configuration form ! Only Zabbix super admins can create host groups! Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 103 DELETE HOST GROUP Delete Host group button: Located at the bottom of Host groups configuration form Will delete an existing host group Deleting a host group will only delete the logical group, not hosts in the group Hosts included in this host group will be not deleted It is not possible to delete a host group if this is the only group for some hosts A warning will be displayed with names of hosts who have this as a last host group Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 104 ENABLING OR DISABLING HOSTS Enable/Disable hosts button: Located at the bottom of Host groups configuration form Will enable or disable all hosts belonging to the selected groups Hosts are disabled Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 105 Hosts Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved 20 minutes 106 HOST DEFINITION Typical Zabbix hosts are the devices you wish to monitor: Host can be a physical device (server, workstation, switch, etc.) Host can be a virtual services (virtual machine, docker container, etc.) Host can also be just an application (Database, Java application, Web page, etc.) A host contains multiple entities used for monitoring: Items data collectors Triggers problem conditions Graphs predefined graphs Web scenarios monitor web pages Discovery rules automatically create items, triggers and graphs from prototypes Host functions as a container for metrics and entities related to them Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 107 CONFIGURING HOSTS Configuration > Hosts is the section where users can create and configure hosts: Filter area allows to find specific hosts based on a host group, a host name, etc. The filter can be displayed or hidden by clicking on the filter sign Configuration area allows to change host properties Clicking on a host name will open the host configuration popup form Clicking on an entity type will open its configuration page Filter area Configuration area Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 108 HOST TAB The Host tab contains general host attributes: Host name and at least one host group must be defined All other fields are optional Configuration tabs General attributes Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 109 CONFIGURATION TABS Host configuration page has a multi tab interface: Information on more than one tab can be updated at once Host configuration tabs: Host general host properties (name, interfaces etc.) IPMI settings for IPMI authentication (algorithm, credentials) Tags host-level tags Macros inherited and host macros Inventory host inventory data (disabled, automatic or manual) Encryption encryption settings (unencrypted, PSK, certificates) Value mapping mapping between numeric/string values and string representations. Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 110 HOST OBJECT Host properties Host entities User macros Items Tags Value maps Triggers Graphs Web scenarios Low-level discovery rules Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 111 HOST NAME Host name must be defined: {HOST.HOST} Is case sensitive and must be unique This is the "technical" name of the host Alphanumeric characters, spaces, dots, dashes and underscores are allowed Leading and trailing spaces are not allowed Visible name is optional: {HOST.NAME} Can be empty – a "Host name" will be used instead If defined, it is also case-sensitive and must be unique If a "Visible name" is set, it will be used in all frontend sections for visualization Has UTF-8 MB4 support, local languages can be used Testing server in a local language Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 112 HOST GROUP A host must have at least one host group: Select button allows to select one or more existing groups Groups input field can also be used: Type a part of the existing group name and select the correct group name Type a non-existing group name and click on the name followed by (new) Create a new host group Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 113 HOST TAGS Tags can be created on a host level: Defined using the Tags tab interface Host tags will be inherited by all items and triggers on the host HOST ITEM TRIGGER Item Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 114 PRACTICAL SETUP 1) Create the following host groups: Training Training/Servers 2) Use filter to filter out only host groups with Training in the name 3) Create a new host training.lan: Name: training.lan Visible name: Training resources Host group: Training/Servers Description: Training server with common resources Do not create any interfaces 4) Specify tags for the new host: location environment <use your location> training 5) Use filter to display only hosts in the "Training/Servers" host group Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved 20 minutes Practical task No: 4 115 Interfaces Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved 20 minutes 116 HOST INTERFACES Zabbix needs host interfaces to monitor different network objects: Zabbix agents Network devices Hardware sensors Java applications Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 117 INTERFACE TYPES Interface defines an IP address or a DNS name of the monitoring target Four host interface types are supported: ZBX Zabbix agent in passive mode SNMP Simple Network Management Protocol IPMI Intelligent Platform Management Interface JMX Java Management Extensions IP address is ignored in this case SNMP interface type has additional settings: Bulk mode SNMP parameters Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 118 HOSTS WITHOUT INTERFACES No interfaces are defined on the host by default: Interfaces required for monitoring must be created manually Interface can use IP or DNS address Interfaces that are in use by any existing items cannot be removed Interface of the correct type is always required for following item types: Zabbix agent passive items SNMP agent items JMX agent items IPMI agent items Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 119 ITEMS WITHOUT INTERFACES Some item types do not require any interface: Zabbix agent in active mode Zabbix trapper Internal items Calculated checks Database monitor HTTP agent Script Some item types require any interface, but may use it or not: Simple check External check Telnet and SSH Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 120 HOST INTERFACES Each host may have multiple interfaces defined: Interfaces of different types on the same host is a typical scenario: Multiple interfaces of the same type also can be used on a single host: Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 121 INTERFACE MACROS Built-in macros can be used to get the interface parameters: {HOST.CONN} {HOST.IP} {HOST.DNS} {HOST.PORT} If multiple interfaces exist: Macros used in trigger expressions will address the interface used by the item Macros used in items without interfaces or scripts will use the following priority: Zabbix agent SNMP JMX IPMI Will resolve to *UNKNOWN* if no interfaces are defined at all on the host Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 122 AVAILABILITY ICON Each interface type has the availability icon: Green all interfaces of this type are available Yellow at least one interface of this type is available and at least one unavailable Red all interfaces of this type are unavailable Gray at least one interface has unknown status and none unavailable Clicking on the availability icon will display detailed status and error messages: Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 123 AVAILABILITY ICON If an interface type is not defined, availability icon not displayed at all: Zabbix agent interface is defined but not used (can be removed) Zabbix agent interface is not defined Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 124 UNKNOWN STATUS Zabbix server will set the interface availability to unknown if: The host is disabled There are no enabled items on the corresponding interface: All items for this interface are disabled This interface is not used in any item at all The host is monitored by a proxy that appears to be offline The host has been switched to be monitored by other server or proxy (until the config cache updates) Unknown availability Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 125 AVAILABILITY ICON For multiple interfaces, the status of every interface will be displayed: Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 126 UNREACHABLE POLLERS Unreachable pollers are responsible for the failed checks on interfaces: The data is gathered by the data collectors (usually some poller process) When a data collection error occurs, unreachable pollers will be utilized: ### Option: StartPollersUnreachable # Number of pre-forked instances of pollers for unreachable hosts (including IPMI and Java). # Range: 0-1000 StartPollersUnreachable=5 Unreachable pollers will start checking the interface based on Unreachable and Unavailable settings after the first data collection failure Start more unreachable pollers if a lot of network errors happen frequently ### Option: UnreachableDelay # How often host is checked for availability during the unreachability period, in seconds. UnreachableDelay=15 ### Option: UnreachablePeriod # After how many seconds of unreachability treat a host as unavailable. UnreachablePeriod=45 ### Option: UnavailableDelay # How often host is checked for availability during the unavailability period, in seconds. UnavailableDelay=60 Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 127 INTERFACE AVAILABILITY Interface availability is calculated by Zabbix server internally: First 45 seconds after the item check fails, the interface is unreachable: Different item keys are checked each time every 15 seconds The interface is still marked as available After 60 seconds, the interface is marked as not available From this moment, the item keys are checked only every 60 seconds Status changes can be tracked in Zabbix server log file 12:20220824:103500 Zabbix agent item "system.localtime" failed: first network error, wait for 15 sec 15:20220824:103515 Zabbix agent item "system.sw.arch" failed: another network error, wait for 15 sec 15:20220824:103530 Zabbix agent item "kernel.maxproc" failed: another network error, wait for 15 sec 15:20220824:103545 disabling Zabbix agent checks on host "PROD Server": interface unavailable 15:20220824:103745 enabling Zabbix agent checks on host "PROD Server": interface became available 15 sec 15 sec 15 sec Monitored Unreachability period 60 sec 60 sec Unavailability period Monitored Unreachable poller Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 128 PRACTICAL SETUP 1) Link template "Zabbix agent" to "Training resources" host: Add Zabbix agent interface with a DNS name training.lan Check interface status Link template "Zabbix agent" Check interface status Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved 10 minutes Practical task No: 5 129 Data collection Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved 40 minutes 130 DATA COLLECTION Zabbix supports multiple ways of data collection: Using Zabbix agent Zabbix agent is installed on monitoring targets Monitoring can be done in active or passive mode directly or using Zabbix proxies Agentless Monitoring is performed directly from Zabbix server or Zabbix proxy To monitor Java applications Zabbix Java gateway can be deployed Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 131 ITEM FORM Data is gathered from a host using Items: Create a new host or use an existing one Create a new item and set parameters: Name Type Key (must be unique on a host) Interface (if required) Data type Units Update interval History and trends storage period Other (based on item type) The form changes according to the selected Type ! Multiple items with the same name can be created on a host if their keys are unique Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 132 ITEM NAME Item name is used to display information in the other frontend sections: Naming guide: Choose a simple, descriptive name for each item Prefix item names (metric) with the object name (metric location): <metric location>: <metric name>, for example: Interface eth0: Bits in Interface eth0: Bits out You may use the # if the metric location is just a number or index: #0: CPU utilization #1: CPU utilization Consider adding suffixes such as “per second”, “per hour”, etc., to describe the metric better. i https://www.zabbix.com/documentation/guidelines/doc Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 133 ITEM TYPE Item type describes how the metric will be collected: Passive (Polling) Zabbix agent Simple check SNMP agent IPMI agent JMX agent HTTP agent SSH agent Telnet agent Database monitor External check Script Zabbix 6.0 Certified Specialist ● Day 1 Active (Trapping) Zabbix agent (active) SNMP trap Zabbix trapper HTTP agent if "Enable trapping" is set Other (processed internally) Zabbix internal Calculated Dependent item © 2022 by Zabbix. All rights reserved Theory 134 ITEM KEY Item key has the following syntax: key[parameters,<optional parameters>] Key Predefined for some item types (Zabbix agent, Internal item, Simple check, etc.) Free form string for other types (SNMP, Zabbix trapper, HTTP agent, etc.) Only item key must be unique per host/template Parameters If parameters are specified, they are mandatory. Some item keys do not have parameters Key parameters may be a quoted string, an unquoted string or a list Macros can be used in parameters <Optional parameters> The "<>" means optional parameter, which may be left empty Optional parameters will use default values if not specified i https://www.zabbix.com/documentation/6.0/manual/config/items/item/key Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 135 ITEM KEY EXAMPLES Without parameters: agent.ping agent.variant system.boottime With mandatory parameters: net.tcp.listen[port] net.tcp.listen[631] net.tcp.listen[22] With mandatory and optional parameters: net.if.in[interface,<mode>] net.if.in[eth0] net.if.in[eth0,] net.if.in[eth0,errors] net.tcp.port[<ip>,port] net.tcp.port[,80] net.tcp.port[www.example.com,80] The appropriate number of commas must be added if any further parameters are specified: proc.cpu.util[<name>,<user>,<type>,<cmdline>,<mode>,<zone>] proc.cpu.util[zabbix_agentd] proc.cpu.util[,root] proc.cpu.util[,,,nginx] Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 136 PARAMETER QUOTING Item key parameters may require quoting: To quote item key parameters, use double quotes only A key parameter must be quoted if it contains: Comma Square brackets Quotation mark (must be escaped with a backslash \ ) , [] \" Incorrect use of quoting may lead to unsupported or incorrectly working items If the key parameter is a quoted string, any Unicode character is allowed Correct Wrong net.if.in["eth0",errors] net.if.in["eth0,errors"] log[/var/log/messages,"Error: [A-Za-z,.]"] log[/var/log/messages,Error: [A-Za-z,.]] vfs.file.exists[/tmp/application.pid,"file,dir"] vfs.file.exists[/tmp/application.pid,file,dir] Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 137 TYPE OF INFORMATION Zabbix item may have one of the following data types: Numeric unsigned 64-bit unsigned integer Numeric float double precision floating point Character 255 characters of text Text 64 kilobytes of text Log data with log related properties: timestamp, source, etc. (64 KB) If a wrong data type is selected, the item may become unsupported Example: data 0.0125 (float) and type Numeric (unsigned) Text values will be truncated to match the database value type limit Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 138 AUTOMATIC SUGGESTION OF TYPE Frontend will suggest "Type of information" automatically for some item types: Zabbix agent Simple checks Correct type is automatically suggested If not suggested type is selected, a yellow warning sign will be displayed Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 139 HISTORY STORAGE History and trends are two ways of storing collected data in Zabbix: History keeps each collected value with a UNIX timestamp and nanoseconds Trends keep aggregated information on hourly basis and are less resource-hungry: Minimum Maximum Average Number of values You can set for how many days history or trends will be kept: Per item in the item configuration form (or by mass updating) Globally by using override in Housekeeping settings If History is set to "Do not keep history": Problems will not be detected for such items Trends still can be calculated Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 140 TRENDS Trends are very efficient to store data for longer time periods: Recalculated automatically with every new value and stored every hour Can be calculated only for numerical item types 1 Hour Timestamp Value 2020-04-21 18:00:02 2365456 2020-04-21 18:01:02 2889365 2020-04-21 18:03:02 3275493 2020-04-21 18:04:02 3923523 .................................... ............... 2020-04-21 18:58:02 8743255 2020-04-21 18:59:02 4223525 Timestamp MIN 2020-04-21 19:00:00 2365456 Zabbix 6.0 Certified Specialist ● Day 1 History 60 values 5400 bytes MAX AVG COUNT 8743255 5343356 60 © 2022 by Zabbix. All rights reserved Trends 4 values 80 bytes Theory 141 EXPIRED DATA Expired data is removed from Zabbix database by the housekeeper process: Only historical data is removed: Item history and trends Expired events and problems Expired audit logs, sessions, etc. Housekeeper can be executed manually zabbix_server -R housekeeper_execute Housekeeper is executed automatically every hour by default ### Option: HousekeepingFrequency # How often Zabbix will perform housekeeping procedure (in hours). # Housekeeping is removing outdated information from the database. # To lower load on server startup housekeeping is postponed for 30 minutes after Zabbix start # Range: 0-24 HousekeepingFrequency=1 Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 142 CLEAR HISTORY Collected values can be deleted on demand by pressing Clear history Both history and trends data will be cleared Command can be issued for one or multiple items Cleanup will start immediately using SQL DELETE statements (the housekeeper is not used) This can cause significant load on database if large amount of data must be erased Confirmation window will be displayed Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 143 TIMEOUT Data collection is affected by the timeout setting on Zabbix server or proxy: ### Option: Timeout # Specifies how long we wait for agent, SNMP device or external check (in seconds). # Range: 1-30 # Default: Timeout=10 I will wait only 10 seconds for an answer! This is a "general" timeout, which is not used in some situations: Active checks are not affected by the timeout on Zabbix server side HTTP checks have their own timeout Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 144 Simple checks Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved 20 minutes 145 SIMPLE CHECKS Simple checks are used for remote agent-less checks of services: They are executed from Zabbix server or proxy without any agent Timeout in Zabbix server / proxy configuration file affects simple checks Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 146 USING INTERFACE ADDRESS The address of any interface can be used as a destination address (optionally): If <target> is not specified in the item parameters, the address from the selected interface is used Interface address is used If <target> is specified in the item parameters, the interface address is not used at all Interface address is not used Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 147 SIMPLE CHECKS AND INTERFACES Simple checks may use interface address, but they do not rely on the availability: Interface availability status does not affect simple checks Simple checks are not affecting any of the interface availabilities Zabbix agent is not responding, but simple check will ping the host Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 148 ICMP PING It is possible to use ICMP packets to test host accessibility: fping utility is used for sending ICMP packets: fping binary file must have correct suid/permission settings (executable by Zabbix) fping defaults can differ depending on the platform and version ### Option: FpingLocation # Location of fping. # Make sure that fping binary has root ownership and SUID flag set. FpingLocation=/usr/sbin/fping Zabbix pinger process performs ICMP checks Start more pingers if lots of ICMP checks are used ### Option: StartPingers # Number of pre-forked instances of ICMP pingers. # Range: 0-1000 StartPingers=5 Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 149 ICMP ITEMS Zabbix supports the following ICMP items: icmpping[<target>,<packets>,<interval>,<size>,<timeout>] icmppingloss[<target>,<packets>,<interval>,<size>,<timeout>] icmppingsec[<target>,<packets>,<interval>,<size>,<timeout>,<mode>] fping -C5 -i10 dev-server.example.com dev-server.example.com : [0], 84 bytes, 0.06 ms (0.06 avg, 0% loss) dev-server.example.com : [1], 84 bytes, 0.08 ms (0.07 avg, 0% loss) dev-server.example.com : [2], 84 bytes, 0.06 ms (0.06 avg, 0% loss) dev-server.example.com : [3], 84 bytes, 0.06 ms (0.07 avg, 0% loss) dev-server.example.com : [4], 84 bytes, 0.06 ms (0.06 avg, 0% loss) Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 150 TCP AND UDP CHECKS Availability and performance of remote TCP or UDP services: Syntax net.tcp.service[service,<ip>,<port>] net.tcp.service.perf[service,<ip>,<port>] net.udp.service[service,<ip>,<port>] net.udp.service.perf[service,<ip>,<port>] It is possible to check any TCP service by using tcp as a service name and a port Only ntp is supported as a UDP service currently Examples: net.tcp.service[ssh] net.tcp.service.perf[http,,8080] Services that have predefined default ports and responses (except http) to check for: ssh, ldap, smtp, ftp, pop, nntp, imap, http, https, telnet i https://www.zabbix.com/documentation/6.0/manual/appendix/items/service_check_details Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 151 PRACTICAL SETUP 1) Create two simple check items on the "Training resources" host using the host interface: ICMP ping status with tag "component : icmp ping" Web service performance with tag "component : http" 2) Create a simple check without using the host interface: NTP service availability using DNS address: training.lan and with tag "component : ntp" 3) Check that all items are receiving data: Use Latest data view Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved 20 minutes Practical task No: 6 152 Zabbix agent Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved 20 minutes 153 ZABBIX AGENT Zabbix agent is a process deployed on monitoring targets: May run on various supported platforms (Linux, UNIX, macOS, Windows, etc.) Collects data from a device or an application Has very low memory footprint and resource usage Can work in the passive or active mode (simultaneously) Supports native communication encryption Passive mode Active mode Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 154 ZABBIX AGENT INSTALLATION Zabbix agent can be installed from packages for most Linux distributions: Use Zabbix repository and package manager to install Pre-compiled agents are available for other systems: Go to https://www.zabbix.com/download_agents Choose the correct OS distribution, hardware and encryption support Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 155 ZABBIX AGENT GENERATIONS Two agent types are currently developed in parallel: Zabbix agent The "classic" Zabbix agent Written in C language Can be executed on all platforms Still will be developed for future versions of Zabbix Zabbix agent 2 The next generation of Zabbix agent Written in Go Language Requires monitored platform to have Go language support Easily extendable with plugins Supports all the functionality of Zabbix agent Agent generation can be detected by using agent.variant item key Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 156 AGENT VS AGENT 2 Parameter i Agent Agent 2 Programming language C Golang Linux daemonization Yes By systemd only Run as Windows service Yes Yes Supported extensions Loadable modules Plugins Supported platforms All (Including BSD, HP-UX, Solaris, etc.) Linux, Windows Concurrency Active checks are executed sequentially All checks can be executed concurrently Scheduled/flexible intervals Passive checks only Active and passive checks Persistent storage No Yes (SQLite3) Timeout settings On agent level only Plugins can override timeout https://www.zabbix.com/documentation/6.0/en/manual/appendix/agent_comparison Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 157 ZABBIX AGENT ITEM TYPES Only two item types are using Zabbix agent for data collection: Zabbix agent Zabbix agent (active) The item key must be known to Zabbix agent: Built-in into the agent (older versions may not support all keys from Zabbix 6.0) Defined as a UserParameter in the agent configuration (discussed later) Some item keys are OS-specific The type of information must match the value type returned by the agent, for example: system.cpu.load numeric (float) system.uptime numeric (unsigned) agent.version character Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 158 FRONTEND REFERENCE Frontend has a quick reference to all built-in item keys: Press the Select button next to the Key field to bring up the list of supported item keys If an agent version differs from a frontend version, some keys may not work It is recommended to read Zabbix documentation to fully understand the potential of an item Read documentation i https://www.zabbix.com/documentation/6.0/manual/config/items/itemtypes/zabbix_agent Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 159 ZABBIX AGENT ON LINUX Zabbix agent on Linux / *NIX systems: Runs from a limited zabbix account by default User = zabbix; Group = zabbix Additional privileges may be required to launch scripts or monitor files Configuration file is located at /etc/zabbix/zabbix_agentd.conf by default /etc/zabbix/zabbix_agentd.d directory is included by default Log file is located at /var/log/zabbix/zabbix_agentd.log by default 20220103:095804.178 Starting Zabbix Agent [Production server]. Zabbix 6.0.0 (revision d3db14ccb5). 20220103:095804.178 **** Enabled features **** 20220103:095804.178 IPv6 support: YES 20220103:095804.178 TLS support: YES 20220103:095804.178 ************************** 20220103:095804.178 using configuration file: /etc/zabbix/zabbix_agentd.conf 20220103:095804.179 agent #0 started [main process] 20220103:095804.186 agent #2 started [listener #1] 20220103:095804.189 agent #3 started [listener #2] 20220103:095804.191 agent #1 started [collector] 20220103:095804.191 agent #4 started [listener #3] Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 160 RUNNING AGENT AS ROOT Running Zabbix agent as root is not recommended but possible: Systemd service settings must be modified: /etc/systemd/system/zabbix-agent.service.d/override.conf [Service] User=root Group=root Only for C-based Zabbix agent additional configuration settings are required: A user must be specified ### Option: User # Drop privileges to a specific, existing user on the system. # Only has effect if run as 'root' and AllowRoot is disabled. User=root AllowRoot agent configuration parameter must be enabled ### Option: AllowRoot # Allow the agent to run as 'root'. If disabled and the agent is started by 'root', the agent # will try to switch to the user specified by the User configuration option instead. AllowRoot=1 ! Run Zabbix agent as root only if there are no other options! Sudo is a much safer choice. Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 161 COMMAND-LINE PARAMETERS Zabbix agent supports command-line options and runtime commands: Get help information zabbix_agentd --help zabbix_agentd -h Change log level (increase/decrease, change for one process only, etc.) zabbix_agentd -R log_level_increase="listener" List all built-in items: zabbix_agentd --print zabbix_agentd -p Test an item key The test is performed using currently logged in user (not zabbix user by default) zabbix_agentd -t system.cpu.load system.cpu.load Zabbix 6.0 Certified Specialist ● Day 1 [d|0.75] © 2022 by Zabbix. All rights reserved Theory 162 SAME KEY, DIFFERENT RESULTS Different item types may have the same item key, which will perform differently: Simple check is executed from Zabbix server or proxy Simple check www.example.com Zabbix agent check is executed using Zabbix agent Zabbix agent Agent service item checks may be used to check connectivity between hosts Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 163 PRACTICAL SETUP 1) Install Zabbix agent (C-based) on your virtual machine: Use Zabbix official repository to install agent package Start Zabbix agent and enable auto-start Check Zabbix agent log file Make sure that host "Zabbix server" is available and collecting data 2) Check Zabbix agent version using frontend and command line 3) Test few agent keys using the command line: The agent generation Memory used by zabbix_server process 4) Test agent key using Zabbix frontend test functionality: Zabbix log files directory size Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved 15 minutes Practical task No: 7 164 Zabbix agent on Windows Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved 10 minutes 165 ZABBIX AGENT INSTALLATION On Windows, there are two ways to install Zabbix agent: Use precompiled binaries and command-line commands zabbix_agentd.exe --install --config C:\Program Files\Zabbix\zabbix_agentd.conf zabbix_agentd.exe --start Use MSI packages provided by Zabbix (recommended) Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 166 ZABBIX AGENT ON WINDOWS By default, Zabbix agent on MS Windows: Runs from the "Local system" account with full access to everything on the OS Is installed as a standard Windows service For all modern x64 systems 64-bit agent is recommended 64-bit agent cannot access 32-bit performance counters and vice versa Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 167 WINDOWS SPECIFIC AGENT KEYS Some Zabbix item keys are Windows specific: Windows service monitoring service.info[service,<param>] services[<type>,<state>,<exclude>] monitors Windows service state lists all services that match specified parameters Windows process monitoring proc_info[process,<attribute>,<type>] returns various information about specific process(es) Virtual memory monitoring vm.vmemory.size[<type>] Virtual memory size in bytes or in percentage from total Event log monitoring Performance counters monitoring WMI query monitoring Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 168 EVENT LOG MONITORING Windows event logs can be monitored by Zabbix eventlog[name,<regexp>,<severity>,<source>,<eventid>,<maxlines>,<mode>] name of the event log (System, Security etc) regexp - regular expression describing the required pattern severity - regular expression describing severity (“Information”, “Warning”, “Error”, “Critical”, etc.) source - regular expression describing source identifier eventid - regular expression describing the event identifier, for example (529|680) Additional settings for event log items: Item must be configured as an active check Type of information: Log (retrieves the timestamp of an original event) Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 169 PERFORMANCE COUNTER MONITORING Zabbix can gather information from Windows performance counters: perf_counter[counter,<interval>] performance counter values perf_counter_en[counter,<interval>] performance counter in English (recommended) typeperf -qx command returns all counters Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 170 WMI QUERY MONITORING WMI queries is another way how to collect metrics from Windows machines: wmi.get[<namespace>,<query>] Execute WMI query and return the object value Windows Management Instrumentation Query Language is used Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved Theory 171 QUESTIONS? Zabbix 5.0 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved 172 Time for a break :) Zabbix 6.0 Certified Specialist ● Day 1 © 2022 by Zabbix. All rights reserved