Agenda: exercises-log-management-swatch.txt

Network Management & Monitoring

Log management, part II : Using swatch
--------------------------------------


Notes:
------
* Commands preceded with "$" imply that you should execute the command as
  a general user - not as root.
* Commands preceded with "#" imply that you should be working as root.
* Commands with more specific command lines (e.g. "RTR-GW>" or "mysql>") 
  imply that you are executing commands on remote equipment, or within 
  another program.

Exercises
---------

0. Log in to your PC or open a terminal window as the root user:

        $ sudo bash

1. Let's enable logging of everything to a single file:

        # vi /etc/rsyslog.conf

        - Find the line
        
        local5.*                        -?RouterLogs

        ... and add a new line below:

        local5.*            /var/log/network/everything

... this will enable logging of ALL messages to a single file, so that we
can run a monitoring script on the messages.

        - Now restart rsyslog:

        # service rsyslog restart

2. Enable a daily automated script to truncate the log file so it doesn't
grow too big:

        # vi /etc/logrotate.d/everything
        
        - In the file add the following:

/var/log/network/everything {
  daily
  copytruncate
  rotate 1
  postrotate
        /etc/init.d/swatch restart
  endscript
}


2. Install swatch

        # apt-get install swatch

3. Create the file /etc/swatch.conf and add the following rules in the file:

        # vi /etc/swatch.conf

watchfor /PRIV_AUTH_PASS/
        mail=sysadm,subject=Enable mode entered
        threshold type=limit,count=1,seconds=3600

watchfor /CONFIG_I/
        mail=sysadm,subject=Router configuration
        threshold type=limit,count=1,seconds=3600

watchfor /LINK/
        mail=sysadm,subject=Link state change
        threshold type=limit,count=1,seconds=3600

watchfor /SSH/
        mail=sysadm,subject=SSH connection
        threshold type=limit,count=1,seconds=3600

        Save the file and exit


4. Start swatch:

        # swatch -c /etc/swatch.conf --daemon -t /var/log/network/everything

        Check that it is running:

        # ps ax | grep swatch

5. Log in to your router, and run some "config" commands (example below):

        # telnet 10.10.X.254            [where "X" is your router number]
        rtrX.ws.nsrc.org> enable
        Password: <password>
        rtrX.ws.nsrc.org# config terminal
        rtrX.ws.nsrc.org(config)# int FastEthernet0/0
        rtrX.ws.nsrc.org(config-int)# description Description Change for FastEthernet0/0 for Swatch
        rtrX.ws.nsrc.org(config-int)# ctrl-z
        rtrX.ws.nsrc.org# write memory
        rtrX.ws.nsrc.org# exit

        Just as in the previous exercise, attempt to shutdown / no shutdown
        a loopback interface

6. Verify that you are receiving emails to the sysadmin user from Swatch

        $ su - sysadmn
        $ mutt -f /var/mail/sysadm


7.  Let's add some ACLs to the router

        rtrX# conf t
        rtrX(config)# access-list 123 deny tcp any host 10.10.X.254 eq 23 log
        rtrX(config)# access-list 123 permit ip any any
        rtrX(config)# interface fastEthernet 0/1
        rtrX(config)# ip access-group 123 in
        rtrX(config)# exit

        (remember, X is the number of your group)

        Explanation: we are now filtering Telnet to the router, on the inside
        interface, explicitly, but we allow anything else.  The "permit" statement
        is required or we will be disabling all IP access to the router!

8. Test that it works:

        From your PC:

        $ telnet 10.10.X.254
        Trying 10.10.X.254...
        telnet: Unable to connect to remote host: No route to host
        $

        Notice that it says "No route to host" instead of "Connection refused"

        This is because, although we have disabled Telnet already by enabling
        SSH on the routers, an active ACL will respond differently than a closed
        port (TCP RST vs. ICMP Host Unreachable)

        Now check out /var/log/network/everything:

        $ tail /var/log/network/everything
        Jun  2 13:46:14 rtrX 6133: *Jun  2 15:46:13.552: %SEC-6-IPACCESSLOGP: list 123 denied tcp 10.10.X.37(43523) -> 10.10.X.254(23), 1 packet 

        Hint: if your log is filled with "SSH-5-*" messages, ignore them like this:

        $ grep -v SSH-5 /var/log/network/everything | tail

        ... you should see SEC-6-IPACCESSLOGP messages

9. Add a new swatch rule to detect these events

        # vi /etc/swatch.conf, and add this:

        watchfor /SEC-6-IPACCESS/
            mail=sysadm,subject=Blocked connection
                threshold type=limit,count=1,seconds=3600

10. Kill swatch, and restart it:

        # ps ax |grep swatch | grep -v grep

12345 ?        Ss     0:00 /usr/bin/swatch -c /etc/swatch.conf --daemon -t /var/log/network/everything


        The number on the LEFT is the number you need to kill - here 12345

        # kill 12345    (the number YOU got!!)

11. Restart swatch

        # swatch -c /etc/swatch.conf --daemon -t /var/log/network/everything

12. Try to telnet to the router again, and check your mail!