Agenda: exercises-log-management-tenshi.htm

File exercises-log-management-tenshi.htm, 13.1 KB (added by Chris Elliott, 6 years ago)
Line 
1<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
2<html xmlns="http://www.w3.org/1999/xhtml">
3<head>
4  <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
5  <meta http-equiv="Content-Style-Type" content="text/css" />
6  <meta name="generator" content="pandoc" />
7  <title>Log Management Part 2: Using Tenshi</title>
8  <style type="text/css">code{white-space: pre;}</style>
9  <link href="data:text/css;charset=utf-8,%0A%0A%0A%0Adiv%23header%2C%20header%0A%7B%0A%0Aborder%2Dbottom%3A%201px%20solid%20%23aaa%3B%0Amargin%2Dbottom%3A%200%2E5em%3B%0A%7D%0A%2Etitle%20%0A%7B%0Atext%2Dalign%3A%20center%3B%0A%7D%0A%2Eauthor%2C%20%2Edate%20%0A%7B%0Atext%2Dalign%3A%20center%3B%0A%7D%0A%0Adiv%23TOC%2C%20nav%23TOC%0A%7B%0A%0Aborder%2Dbottom%3A%201px%20solid%20%23aaa%3B%0Amargin%2Dbottom%3A%200%2E5em%3B%0A%7D%0A%40media%20print%0A%7B%0Adiv%23TOC%2C%20nav%23TOC%0A%7B%0A%0Adisplay%3A%20none%3B%0A%7D%0A%7D%0A%0Ah1%2C%20h2%2C%20h3%2C%20h4%2C%20h5%2C%20h6%0A%7B%0Afont%2Dfamily%3A%20%22Helvetica%20Neue%22%2C%20Helvetica%2C%20%22Liberation%20Sans%22%2C%20Calibri%2C%20Arial%2C%20sans%2Dserif%3B%20%0A%0Apage%2Dbreak%2Dafter%3A%20avoid%3B%20%0A%7D%0A%0Adiv%20div%2C%20section%20section%20%0A%7B%0Amargin%2Dleft%3A%202em%3B%20%0A%7D%0Ap%20%7B%7D%0Ablockquote%0A%7B%20font%2Dstyle%3A%20italic%3B%0A%7D%0Ali%20%0A%7B%0A%7D%0Ali%20%3E%20p%20%0A%7B%0Amargin%2Dtop%3A%201em%3B%20%0A%7D%0Aul%20%0A%7B%0A%7D%0Aul%20li%20%0A%7B%0A%7D%0Aol%20%0A%7B%0A%7D%0Aol%20li%20%0A%7B%0A%7D%0Ahr%20%7B%7D%0A%0Asub%20%0A%7B%0A%7D%0Asup%20%0A%7B%0A%7D%0Aem%20%0A%7B%0A%7D%0Aem%20%3E%20em%20%0A%7B%0Afont%2Dstyle%3A%20normal%3B%0A%7D%0Astrong%20%0A%7B%0A%7D%0A%0Aa%20%0A%7B%0A%0Atext%2Ddecoration%3A%20none%3B%0A%7D%0A%40media%20screen%0A%7B%0Aa%3Ahover%0A%7B%0A%0Atext%2Ddecoration%3A%20underline%3B%0A%7D%0A%7D%0A%40media%20print%0A%7B%0Aa%20%7B%0A%0Acolor%3A%20black%3B%0Abackground%3A%20transparent%3B%0A%7D%0Aa%5Bhref%5E%3D%22http%3A%2F%2F%22%5D%3Aafter%2C%20a%5Bhref%5E%3D%22https%3A%2F%2F%22%5D%3Aafter%0A%7B%0A%0Acontent%3A%20%22%20%28%22%20attr%28href%29%20%22%29%20%22%3B%0Afont%2Dsize%3A%2090%25%3B%0A%7D%0A%7D%0A%0Aimg%0A%7B%0A%0Avertical%2Dalign%3A%20middle%3B%0A%7D%0Adiv%2Efigure%20%0A%7B%0A%0Amargin%2Dleft%3A%20auto%3B%0Amargin%2Dright%3A%20auto%3B%0Atext%2Dalign%3A%20center%3B%0Afont%2Dstyle%3A%20italic%3B%0A%7D%0Ap%2Ecaption%20%0A%7B%0A%0A%7D%0A%0Apre%2C%20code%20%7B%0Abackground%2Dcolor%3A%20%23fdf7ee%3B%0A%0A%0A%0Awhite%2Dspace%3A%20pre%2Dwrap%3B%20%0Awhite%2Dspace%3A%20%2Dmoz%2Dpre%2Dwrap%20%21important%3B%20%0Awhite%2Dspace%3A%20%2Dpre%2Dwrap%3B%20%0Awhite%2Dspace%3A%20%2Do%2Dpre%2Dwrap%3B%20%0Aword%2Dwrap%3A%20break%2Dword%3B%20%0A%0A%7D%0Apre%20%0A%7B%0A%0Apadding%3A%200%2E5em%3B%20%0Aborder%2Dradius%3A%205px%3B%20%0A%0Aborder%3A%201px%20solid%20%23aaa%3B%0A%0Amargin%2Dleft%3A%200%2E5em%3B%0Amargin%2Dright%3A%200%2E5em%3B%0A%7D%0A%40media%20screen%0A%7B%0Apre%0A%7B%0A%0Awhite%2Dspace%3A%20pre%3B%0Aoverflow%3A%20auto%3B%0A%0Aborder%3A%201px%20dotted%20%23777%3B%0A%7D%0A%7D%0Acode%20%0A%7B%0A%7D%0Ap%20%3E%20code%2C%20li%20%3E%20code%20%0A%7B%0A%0Apadding%2Dleft%3A%202px%3B%0Apadding%2Dright%3A%202px%3B%0A%7D%0Ali%20%3E%20p%20code%20%0A%7B%0A%0Apadding%3A%202px%3B%0A%7D%0A%0Aspan%2Emath%20%0A%7B%0A%0A%7D%0Adiv%2Emath%20%0A%7B%0A%7D%0Aspan%2ELaTeX%20%0A%7B%0A%7D%20eq%20%0A%7B%0A%7D%20%0A%0Atable%0A%7B%0Aborder%2Dcollapse%3A%20collapse%3B%0Aborder%2Dspacing%3A%200%3B%20%0Aborder%2Dbottom%3A%202pt%20solid%20%23000%3B%0Aborder%2Dtop%3A%202pt%20solid%20%23000%3B%20%0A%0Amargin%2Dleft%3A%20auto%3B%0Amargin%2Dright%3A%20auto%3B%0A%7D%0Athead%20%0A%7B%0Aborder%2Dbottom%3A%201pt%20solid%20%23000%3B%0Abackground%2Dcolor%3A%20%23eee%3B%20%0A%7D%0Atr%2Eheader%20%0A%7B%0A%7D%20tbody%20%0A%7B%0A%7D%0A%0Atr%20%7B%0A%7D%0Atr%2Eodd%3Ahover%2C%20tr%2Eeven%3Ahover%20%0A%7B%0Abackground%2Dcolor%3A%20%23eee%3B%0A%7D%0A%0Atr%2Eodd%20%7B%7D%0Atr%2Eeven%20%7B%7D%0Atd%2C%20th%20%0A%7B%20vertical%2Dalign%3A%20top%3B%20%0Avertical%2Dalign%3A%20baseline%3B%20%0Apadding%2Dleft%3A%200%2E5em%3B%0Apadding%2Dright%3A%200%2E5em%3B%0Apadding%2Dtop%3A%200%2E2em%3B%0Apadding%2Dbottom%3A%200%2E2em%3B%0A%7D%0A%0A%0Ath%20%0A%7B%0Afont%2Dweight%3A%20bold%3B%20%7D%0Atfoot%20%0A%7B%0A%7D%0Acaption%20%0A%7B%0Acaption%2Dside%3A%20top%3B%0Aborder%3A%20none%3B%0Afont%2Dsize%3A%200%2E9em%3B%0Afont%2Dstyle%3A%20italic%3B%0Atext%2Dalign%3A%20center%3B%0Amargin%2Dbottom%3A%200%2E3em%3B%20%0Apadding%2Dbottom%3A%200%2E2em%3B%0A%7D%0A%0Adl%20%0A%7B%0Aborder%2Dtop%3A%202pt%20solid%20black%3B%0Apadding%2Dtop%3A%200%2E5em%3B%0Aborder%2Dbottom%3A%202pt%20solid%20black%3B%0A%7D%0Adt%20%0A%7B%0Afont%2Dweight%3A%20bold%3B%0A%7D%0Add%2Bdt%20%0A%7B%0Aborder%2Dtop%3A%201pt%20solid%20black%3B%0Apadding%2Dtop%3A%200%2E5em%3B%0A%7D%0Add%20%0A%7B%0Amargin%2Dbottom%3A%200%2E5em%3B%0A%7D%0Add%2Bdd%20%0A%7B%0Aborder%2Dtop%3A%201px%20solid%20black%3B%20%0A%7D%0A%0Aa%2Efootnote%2C%20a%2EfootnoteRef%20%7B%20%0Afont%2Dsize%3A%20small%3B%20vertical%2Dalign%3A%20text%2Dtop%3B%0A%7D%0Aa%5Bhref%5E%3D%22%23fnref%22%5D%2C%20a%2Ereversefootnote%20%0A%7B%0A%7D%0A%40media%20print%0A%7B%0Aa%5Bhref%5E%3D%22%23fnref%22%5D%2C%20a%2Ereversefootnote%20%0A%7B%0A%0Adisplay%3A%20none%3B%0A%7D%0A%7D%0Adiv%2Efootnotes%20%0A%7B%0A%7D%0Adiv%2Efootnotes%20li%5Bid%5E%3D%22fn%22%5D%20%0A%7B%0A%7D%0A%0A%40media%20print%0A%7B%0A%2Enoprint%0A%7B%0Adisplay%3Anone%3B%0A%7D%0A%7D%0A" rel="stylesheet" type="text/css" />
10</head>
11<body>
12<div id="header">
13<h1 class="title">Log Management Part 2: Using Tenshi</h1>
14<h3 class="date">Network Monitoring &amp; Management</h3>
15</div>
16<div id="TOC">
17<ul>
18<li><a href="#notes"><span class="toc-section-number">1</span> Notes</a></li>
19<li><a href="#exercises"><span class="toc-section-number">2</span> Exercises</a><ul>
20<li><a href="#update-syslog-ng-configuration"><span class="toc-section-number">2.1</span> Update syslog-ng configuration</a></li>
21<li><a href="#log-rotation"><span class="toc-section-number">2.2</span> Log rotation</a></li>
22<li><a href="#install-tenshi"><span class="toc-section-number">2.3</span> Install tenshi</a></li>
23<li><a href="#configure-tenshi"><span class="toc-section-number">2.4</span> Configure tenshi</a></li>
24<li><a href="#testing-tenshi"><span class="toc-section-number">2.5</span> Testing Tenshi</a></li>
25<li><a href="#optional-add-a-new-tenshi-rule"><span class="toc-section-number">2.6</span> Optional: Add a new Tenshi rule</a></li>
26</ul></li>
27</ul>
28</div>
29<h1 id="notes"><span class="header-section-number">1</span> Notes</h1>
30<ul>
31<li>Commands preceded with &quot;$&quot; imply that you should execute the command as a general user - not as root.</li>
32<li>Commands preceded with &quot;#&quot; imply that you should be working as root.</li>
33<li>Commands with more specific command lines (e.g. &quot;rtrX&gt;&quot; or &quot;mysql&gt;&quot;) imply that you are executing commands on remote equipment, or within another program.</li>
34</ul>
35<h1 id="exercises"><span class="header-section-number">2</span> Exercises</h1>
36<p>First make sure that your routers are configured to send logs to your PC (this should have been done in the previous exercise).</p>
37<h2 id="update-syslog-ng-configuration"><span class="header-section-number">2.1</span> Update syslog-ng configuration</h2>
38<p>If you have not already done so, log in to your virtual machine and become the root user:</p>
39<pre><code>$ sudo -s
40#</code></pre>
41<p>Configure syslog-ng to save all router logs in one file for monitoring purposes.</p>
42<p>Edit <code>/etc/syslog-ng/conf.d/10-network.conf</code>,</p>
43<pre><code># cd /etc/syslog-ng/conf.d/
44# editor 10-network.conf</code></pre>
45<p>... and add this just below the line that starts with &quot;template&quot;:</p>
46<pre><code>file(&quot;/var/log/network/everything&quot;, owner(root) group(root) perm(0644));</code></pre>
47<p>In the end, the contents of the file should look like:</p>
48<pre><code>filter f_routers { facility(local0); };
49
50log {
51    source(s_src);
52    filter(f_routers);
53    destination(routers);
54};
55
56destination routers {
57  file(&quot;/var/log/network/$YEAR/$MONTH/$DAY/$HOST-$YEAR-$MONTH-$DAY-$HOUR.log&quot;
58  owner(root) group(root) perm(0644) dir_perm(0755) create_dirs(yes)
59  template(&quot;$YEAR $DATE $HOST $MSG\n&quot;));
60
61  file(&quot;/var/log/network/everything&quot;, owner(root) group(root) perm(0644));
62
63};</code></pre>
64<p>This will enable logging of ALL messages matching the local0 facility to a single file, so that we can run a monitoring script on the messages.</p>
65<p>Be sure to save and exit from the file.</p>
66<p>Now restart syslog-ng so that is sees the new configuration:</p>
67<pre><code># service syslog-ng restart</code></pre>
68<h2 id="log-rotation"><span class="header-section-number">2.2</span> Log rotation</h2>
69<p>Create a daily automated script to truncate the log file so it doesn't grow too big (COPY and PASTE):</p>
70<pre><code># editor /etc/logrotate.d/everything</code></pre>
71<p>Add the following to this file:</p>
72<pre><code>/var/log/network/everything {
73  daily
74  copytruncate
75  rotate 1
76  postrotate
77    /etc/init.d/tenshi restart
78  endscript
79}</code></pre>
80<p>Then save and exit from the file.</p>
81<h2 id="install-tenshi"><span class="header-section-number">2.3</span> Install tenshi</h2>
82<pre><code># apt-get install tenshi</code></pre>
83<h2 id="configure-tenshi"><span class="header-section-number">2.4</span> Configure tenshi</h2>
84<p>Configure Tenshi to send you alarms when the routers are configured (COPY and PASTE this text):</p>
85<pre><code># editor /etc/tenshi/includes-available/network</code></pre>
86<p>Add the following to this file:</p>
87<pre><code>set logfile /var/log/network/everything
88set queue network_alarms tenshi@localhost sysadm@localhost [*/1 * * * *] Log check
89
90group_host ^10\.10\.
91network_alarms SYS-5-CONFIG_I
92network_alarms PRIV_AUTH_PASS
93network_alarms LINK
94group_end</code></pre>
95<p>Then save and exit from the file.</p>
96<p>Create a symlink so that Tenshi loads your new file (COPY and PASTE):</p>
97<pre><code># ln -s /etc/tenshi/includes-available/network /etc/tenshi/includes-active/</code></pre>
98<p>Finally restart Tenshi:</p>
99<pre><code># service tenshi restart</code></pre>
100<p>You may see the following warning message:</p>
101<pre><code>&quot;[WARNING] /var/log/network/everything: no such file&quot;</code></pre>
102<p>don't worry, this is fine. The file &quot;everything&quot; will be created once an initial log message is received.</p>
103<h2 id="testing-tenshi"><span class="header-section-number">2.5</span> Testing Tenshi</h2>
104<p>Log in to your router, and run some &quot;config&quot; commands (example below):</p>
105<pre><code>$ ssh cisco@rtrX                        [where &quot;X&quot; is your router number]
106rtrX&gt; enable
107Password: &lt;password&gt;
108rtrX# config terminal
109rtrX(config)# int FastEthernet0/0
110rtrX(config-if)# description Description Change for FastEthernet0/0 for Tenshi
111rtrX(config-if)# ctrl-z                 (same as exit, exit twice)
112rtrX# write memory</code></pre>
113<p>Don't exit from the router yet. Just as in the previous syslog-ng exercises, attempt to shutdown / no shutdown loopback interface:</p>
114<pre><code>rtrX# conf t
115rtrX(config)# interface Loopback 999
116rtrX(config-if)# shutdown</code></pre>
117<p>wait a few seconds</p>
118<pre><code>rtrX(config-if)# no shutdown</code></pre>
119<p>Then exit, and save the config (&quot;write mem&quot;):</p>
120<pre><code>rtrX(config-if)# ctrl-z                 (same as exit, exit twice)
121rtrX# write memory
122rtrX# exit</code></pre>
123<p>Verify that you are receiving emails to the sysadm user from Tenshi. A quick check is to look in the mail directory:</p>
124<pre><code>$ ls -l /var/mail</code></pre>
125<ul>
126<li>Note: Tenshi checks /var/log/network/everything once a minute, so you may have to wait up to a minute for the email to arrive to the sysadm user.</li>
127</ul>
128<p>Make sure you are logged in as sysadm (not root). Either open a new session to your virtual machine, or exit from the root user (exit). Then do:</p>
129<pre><code>$ mutt</code></pre>
130<p>Scroll <code>up/down</code> to select a message from &quot;tenshi@localhost&quot;, then press <code>ENTER</code> to view it, and <code>q</code> to quit and 'q' again to quit mutt.</p>
131<p>If mails are not arriving, then check the following:</p>
132<ul>
133<li><p>Are logs arriving in the file <code>/var/log/network/everything</code>?</p>
134<pre><code>$ tail /var/log/network/everything</code></pre></li>
135<li><p>Do these logs show a hostname like 'rtr5', or possibly an IP like 10.10.5.254 ? Remember that the way we have configured tenshi, it only looks at hostnames or IP addresses matching the pattern 'rtr' or '10.10' (depending on how you configured tenshi).</p></li>
136<li><p>Check your tenshi configuration file. Restart tenshi if you change it.</p></li>
137<li><p>If you are still stuck ask an instructor or a neighbor for help.</p></li>
138</ul>
139<h2 id="optional-add-a-new-tenshi-rule"><span class="header-section-number">2.6</span> Optional: Add a new Tenshi rule</h2>
140<p>See if you can figure out how to add a rule to Tenshi so that an email is sent if someone enters an incorrect enable password on your router.</p>
141<p>Hints:</p>
142<ul>
143<li>&quot;PRIV_AUTH_FAIL&quot; is the Cisco IOS log message in such cases.</li>
144<li>To test your new rule log in to your router, type &quot;enable&quot; and then enter an incorrect enable password.</li>
145<li>You will need to restart the Tenshi service once you have made your configuration changes.</li>
146</ul>
147</body>
148</html>