1 Introduction

1.1 Goals

Install and configure Munin on nodes (hosts) to be monitored

1.2 Notes

Commands preceded with "$" imply that you should execute the command as a general user - not as root.
Commands preceded with "#" imply that you should be working as root.
Commands with more specific command lines (e.g. "rtrX>" or "mysql>") imply that you are executing commands on remote equipment, or within another program.

2 Exercises

2.1 Installing Munin on the hosts to be monitored

We are now going to install munin on our 3 other hosts. To do this, we'll be using Ansible :)

Let's start by defining a group called munin_nodes in our inventory.

This group will reuse the existing group definitions, app_web and app_db - that is, a "group of groups".

$ sudo editor /etc/ansible/hosts

At the end of the file, add the following lines:

[munin_nodes:children]
app_web
app_db

The :children keyword is required to tell ansible that this group's members are groups, and not nodes.

Now, we'll need to create a playbook for installing munin-node itself.

Now, here are some hints about what will go in this playbook:

The hosts on which munin-node will need to be installed are in the group munin-nodes
The package munin-node needs to be present (installed)
A statement needs to be added to /etc/munin/munin-node.conf, allowing hostN (where N is the number of your master) to "talk" to each munin node.
The service munin-node needs to be restarted after the configuration change

You already know how to perform steps 1, 2, and 4. For step 3, we'll give you the solution here as we haven't yet seen the module used by Ansible to add a line in a file:

- name: allow host221 to talk to munin-node
  lineinfile: dest=/etc/munin/munin-node.conf insertafter="^# cidr_deny  192.0.2.42/32" line="cidr_allow 10.10.0.0/24"
  tags: configure
  notify: restart munin-node

Now it's time to write the required playbook!

Go to your home directory, and verify you're in /home/sysadm:

$ cd
$ pwd

/home/sysadm

Create a new file, munin-node.yml:

$ editor munin-node.yml

Go ahead! If you need help, raise your hand. We'll help you write this playbook if you really can't figure it out, but by reusing previous playbooks, you should be able to figure it out!

2.2 Tell the munin master about the new hosts

One step remains...

You need to add the 3 hosts to the master munin configuration on hostN, in /etc/munin/munin.conf

Edit that file (using sudo) and find the section you edited earlier for [hostN.ws.nsrc.org].

Below it, make 3 new host definitions, similar to the existing one:

[hostY.ws.nsrc.org]
    address 10.10.0.Y
    use_node_name yes

... repeat for all three nodes.

Wait 5 minutes, then browse to http://hostN.ws.nsrc.org/munin/

2.3 Enabling plugins: Apache

You may have noticed from looking at the Plugins page on the Munin web site, that Munin is capable of collecting many types of statistics from servers.

One of them we mentioned is Apache - but if you browse your Munin installation at http://hostN.ws.nsrc.org/munin/, you'll notice that Apache statistics are missing!

To fix this, we'll need to install a package that allows Munin to collect data from Apache, and enable the apache plugins in munin-node's configuration, on each of your hosts.

The package that needs to be installed on the hosts is libwww-perl.

Modify the playbook (munin-nodes.yml) to enable installation of this package.

There is one more step required after adding the package libwww-perl, before Munin can enable the Apache plugin.

Note: The steps below are included to explain what we would need to do, if we didn't use ansible. Please don't execute them!

Normally, we'd need to run the following command on each host:

$ sudo munin-node-configure --shell 2> /dev/null

This would show you the commands required to enable the Apache plugin. You would see something like this:

ln -s '/usr/share/munin/plugins/apache_accesses' '/etc/munin/plugins/apache_accesses'
ln -s '/usr/share/munin/plugins/apache_processes' '/etc/munin/plugins/apache_processes'
ln -s '/usr/share/munin/plugins/apache_volume' '/etc/munin/plugins/apache_volume'

... these are the commands that need to be run, as root, on each host, before Munin will start monitoring Apache statistics.

So we need to find a way for our playbook to call munin-node-configure --shell and execute the commands returned.

Here's what we suggest you add this task your playbook:

- name: reconfigure munin-node after installing support packages
  tags: configure
  shell: "munin-node-configure --shell | sh"
  notify: restart munin-node

... the command munin-node-configure --shell | sh can be explained as:

run the command munin-node-configure --shell (this outputs the commands required to enable the Apache plugin)
pipe the output ("send it") to the command sh, which is a command shell

Try and integrate the above into your playbook. If you succeed, you will now be able to see Apache statistics in your Munin Web UI (http://hostN.ws.nsrc.org/munin).

2.4 Enabling more plugins: MySQL

You might remember that of your three hosts running services:

two of them are running Apache
one of them is running MySQL

We'll therefore need to enable the Munin MySQL plugin as well. This is similar to the above - we'll need to add a package to start with, called libcache-cache-perl.

Repeating the steps above, you will notice that we are now installing 3 different packages:

munin-node itself
libwww-perl
libcache-cache-perl

We could optimize this playbook at bit by installing all 3 packages in one task:

    - name: install munin-node, and supporting packages for plugins to work
      apt: pkg={{item}} state=present
      with_items:
      - munin-node
      - libwww-perl
      - libcache-cache-perl
      tags: install

2.5 Putting it all together

If you succeed in putting it all together, you'll now have a playbook that looks like this:

- hosts:
    - munin_nodes

  tasks:
    - name: ensure package cache is up to date
      apt: update_cache=yes cache_valid_time=3600
      tags: install

    - name: install munin-node, and supporting packages for plugins to work
      apt: pkg={{item}} state=present
      with_items:
      - munin-node
      - libwww-perl
      - libcache-cache-perl
      tags: install

    - name: reconfigure munin-node after installing support packages
      tags: configure
      shell: "munin-node-configure --shell | sh"
      notify: restart munin-node

    - name: allow host221 to talk to munin-node
      lineinfile: dest=/etc/munin/munin-node.conf insertafter="^# cidr_deny  192.0.2.42/32" line="cidr_allow 10.10.0.0/24"
      tags: configure
      notify: restart munin-node

  handlers:
    - name: restart munin-node
      service: name=munin-node state=restarted

If you need to, copy/paste the above playbook into your munin-node.yml file, and try to run it!

Note: like all things true about the Internet, someone has already written a munin role for Ansible. Consider taking a look at it:

https://github.com/espebra/ansible/tree/master/roles/munin-node

You'll notice that we install libwww-perl and libcache-cache-perl on both Apache and MySQL servers - if we really wanted to, we could optimize the playbook to only install libwww-perl on the Apache servers, and libcache-cache-perl only on the MySQL DB server.

But that's for another day!