# Nagios and Request Tracker Integration

# Creating Tickets

# Introduction

## Goals

- Learn how to configure Nagios to raise tickets in RT

## Notes

- Commands preceded with “$” imply that you should execute the command as a general user - not as root.
- Commands preceded with “#” imply that you should be working as root.
- Commands with more specific command lines (e.g. “bdr1-campusY>” or “mysql>”) imply that you are executing commands on remote equipment, or within another program.

# Exercises

To configure RT and Nagios so that alerts from Nagios automatically create tickets requires a few steps:

- Create a proper contact entry for Nagios in /etc/nagios4/objects/contacts.cfg

- Update either /etc/nagios4/conf.d/services.cfg or an individual host entry to use the new contact group.

These next two items should already be done in RT if you have finished the RT exercises.

- Install the rt-mailgate software and configure it properly in your /etc/aliases file for your MTA in use.

- Configure the appropriate queues in RT to receive emails passed to it from Nagios via the rt-mailgate software.

# Exercises

Log in to your virtual machine (hostX-campusY) as the “sysadm” user.

# Configure a Contact in Nagios

Become root on your host:

    $ sudo -s

Edit the file /etc/nagios4/objects/contacts.cfg

    # editor /etc/nagios4/objects/contacts.cfg

In this file we will first add a new contact name under the default root contact entry.

WARNING: DO NOT remove the root contact entry.

REALLY - READ THIS! DO NOT remove the “nagiosadmin” contact entry.

ADD the entry below just AFTER the ROOT contact entry

The new contact should look like this (UNDERNEATH the nagiosadmin contact entry):

    define contact {
            contact_name                    net
            alias                           RT Alert Queue
            service_notification_period     24x7
            host_notification_period        24x7
            service_notification_options    c
            host_notification_options       d
            service_notification_commands   notify-service-by-email
            host_notification_commands      notify-host-by-email
            email                           net@srv1-campusY.ws.nsrc.org
    }

Be sure that you replace “campusY” with the number of your group. So, if you are in group 1, then you would use “net@srv1-campus1.ws.nsrc.org”.

At the end of the file add the following entry: (Note: do not delete or update other entries.)

DO THIS AT THE END OF THE FILE

    define contactgroup {
            contactgroup_name       tickets
            alias                   email to ticket system for RT
            members                 net
    }

Save and exit from the file.

## Notes

- the service_notification_option of “c” means only notify once a service is considered “critical” by Nagios (i.e. down). The host_notification_option of “d” means down. By specifying only “c” and “d” this means that notifications will not be sent for other states.

- Note the email address in use “net@srv1-campusY.ws.nsrc.org” - this is important as this was previously defined in the Request Tracker (RT) exercises.

# Choose a Service to Monitor that Creates Tickets in RT

To send email to generate tickets in RT if SSH goes down on a box you would edit the SSH service check:

    # editor /etc/nagios4/conf.d/services.cfg

Find the service with the hostgroup_name of “ssh-servers” and add the “contact_groups” entry at the end of the definition. When you are done your definition should look like this. Also change “max_check_attempts” from “10” to “4”.

    define service {
            hostgroup_name                  ssh-servers
            service_description             SSH
            check_command                   check_ssh
            use                             generic-service
            # ssh occasionally flaps. Let's be sure it's down.
            retry_interval                  2
            max_check_attempts              4
            notification_interval           0 ; set > 0 if you want to be renotified
            contact_groups                  tickets,admins
    }

Change max_check_attempts to “4” so that Nagios send an E-mail to RT much more quickly. If we leave max_check_attempts at “10” then the process is slower.

Save and exit from the file.

## Notes

- Note the additional item that we now have, “contact_groups.” You can do this for other entries as well if you wish.

- We, also, included the default contact group of admins. You could leave this off if you wish.

# Include noc as members of ssh-server

    # editor /etc/nagios4/conf.d/hostgroups.cfg

Find the group with the hostgroup_name of “ssh-servers” and add noc to the “members” entry at the end of the definition. When you are done your definition should looke like this:

    define hostgroup {
      hostgroup_name  ssh-servers
      alias           SSH Servers
      members         srv1-campusY, host1-campusY, host2-campusY, host3-campusY, \
                      host4-campusY, host5-campusY, host6-campusY, transit1-nren, \
                      bdr1-campusY, core1-campusY, dist1-b1-campusY, dist1-b2-campusY, \
                      noc
    }

Now save and exit from the file.

To create a "NOC entry add this to the file pcs.cfg at the bottom of the file:

    define host {
        use         linux-server
        host_name   noc
        alias       Main NOC server for NMM Workshop
        address     noc.ws.nsrc.org
    }

Save and exit from the file.

Now verify that your Nagios configuration looks good:

    # nagios4 -v /etc/nagios4/nagios.cfg

If there are no errors, then restart your nagios instance:

    # systemctl restart nagios4

If SSH goes down on any of the devices you are monitoring Nagios should generate a new ticket in Request Tracker. We will stop the SSH service on the classroom NOC server.

# Bring down the SSH service on NOC or another server (optional)

Right now you are monitoring noc.ws.nsrc.org for the ssh service. You can ask an instructor to take down the service.

But you can also work within your groups to make changes to the ssh servers in your campus to generate RT tickets. On any server which is monitored by your nagios server, for example hostX-campusY, you can do the following:

    $ ssh sysadm@hostX-campusY.ws.nsrc.org
    $ sudo vi /etc/ssh/sshd_config

Change the line:

    #Port 22

to:

    #Port 22
    Port 2222

Then restart the ssh service:

    $ sudo systemctl restart ssh

If you log out of the server you can log back in via ssh by doing:

    $ ssh -p 2222 sysadm@hostX-campus.Y.ws.nsrc.org

and to put it back you can remove the line

    Port 2222

from /etc/ssh/sshd_config and type.

    $ sudo systemctl restart ssh

If you are using the workshop web interface it is possible to reconnect to hostX-campusY using the “Console” option to the srv1 server, and once on there use `incus shell hostX-campusY` to get a shell on any of the hosts.

# See Nagios Tickets in RT

- It will take a bit (up to 5 minutes) for Nagios to report that SSH is “critical”. Then you must wait for a total of 4 checks before the SSH service is deemed to be down “hard”. At that point Nagios will send an email to net@srv1-campusY.ws.nsrc.org and a ticket will be created in RT.

- Remember to see this go to http://oob-srv1-campusY.ws.nsrc.org/rt/ and log in as Username “sysadm-campusY” with the password you used for your host. The new ticket should appear in the “10 newest unowned tickets” box in the main net queue on the main login page in RT.