Agenda: exercises-smokeping.txt

File exercises-smokeping.txt, 10.9 KB (added by admin, 8 years ago)
Line 
1Network Management & Monitoring
2
3Smokeping
4
5Notes:
6------
7* Commands preceded with "$" imply that you should execute the command as
8  a general user - not as root.
9* Commands preceded with "#" imply that you should be working as root.
10* Commands with more specific command lines (e.g. "RTR-GW>" or "mysql>")
11  imply that you are executing commands on remote equipment, or within
12  another program.
13
14Exercises
15----------
16
170. Log in to your PC or open a terminal window as the sysadmn user.
18
19Once you are logged in you can continue with these exercises.
20
211. Install Smokeping
22
23    $ sudo apt-get install smokeping
24
252. Initial Configuration
26
27    $ cd /etc/smokeping/config.d
28    $ ls -l
29
30    -rwxr-xr-x 1 root root  578 2010-02-26 01:55 Alerts
31    -rwxr-xr-x 1 root root  237 2010-02-26 01:55 Database
32    -rwxr-xr-x 1 root root  413 2010-02-26 05:40 General
33    -rwxr-xr-x 1 root root  271 2010-02-26 01:55 pathnames
34    -rwxr-xr-x 1 root root  859 2010-02-26 01:55 Presentation
35    -rwxr-xr-x 1 root root  116 2010-02-26 01:55 Probes
36    -rwxr-xr-x 1 root root  155 2010-02-26 01:55 Slaves
37    -rwxr-xr-x 1 root root 8990 2010-02-26 06:30 Targets
38 
39    $ sudo vi General
40
41        Change the following lines:
42
43        owner    = NOC
44        contact  = sysadm@localhost
45        cgiurl   = http://localhost/cgi-bin/smokeping.cgi
46        mailhost = localhost
47
48        Save the file and exit. Now let's restart the
49        Smokeping service to verify that no mistakes have been made
50        before going any further:
51
52        $ sudo service smokeping restart
53
542. Configure monitoring of devices
55
56        The majority of your time and work configuring Smokeping
57        will be done in the file /etc/smokeping/config.d/Targets.
58       
59        For this class please do the following:
60
61        Use the default FPing probe to check:
62
63      - all the student PCs
64      - classroom NOC
65      - switches
66      - routers
67     
68    You can use the classroom Network Diagram on the classroom wiki to
69    figure out addresses for each item, etc.
70
71    Please note - a complete set of configuration files is available on
72    your classroom wiki by going to the "Configuration Files" link on the
73    main page of the wiki (http://noc.ws.nsrc.org/wiki/).
74
75    Create some hierarchy to the Smokeping menu for your
76    checks. Such as:
77
78        + PCs
79
80        ++ pc1
81        menu = pc1
82        title = pc1
83        host = pc1
84
85        ++ pc2
86        menu = pc2
87        title = pc2
88        host = pc2
89       
90    Save the file and restart Smokeping:
91
92        $ sudo ser
93
94    Go to your browser and check the Smokeping page:
95
96        http://pcN.ws.nsrc.org/cgi-bin/smokeping.cgi
97
98    If everything is looking OK, continue adding:
99
100        + Routers
101
102        ++ gw
103        menu = gw
104        title = gw
105        menu = gw
106
107        ++ r1
108        menu = r1
109        title = r1
110        host = r1
111
112        + Classroom Switch
113
114        ++ sw
115        menu = sw
116        title = sw
117        menu = sw       
118 
119        ...
120
121    Save the file, restart smokeping, and check
122    your browser again.
123
1243. Add new probes
125
126    The current entry in Probes is fine, but if you wish to
127    use additional Smokeping checks you can add them in here
128    and you can specify their default behavior. You can do
129    this, as well, in the Targets file if you wish.
130
131    Here is an example of a Probes file that would specify
132    what to use to check for HTTP and DNS latency as well as
133    the FPing probe that is used for ping latency:
134
135       $ sudo vi Probes
136
137        *** Probes ***
138
139        + FPing
140
141        binary = /usr/bin/fping
142
143        + EchoPingHttp
144
145        + DNS
146        binary = /usr/bin/dig
147        pings = 5
148        step = 180
149        lookup = www.nsrc.org
150
151        Save the file.
152
1534. Add HTTP latency checks
154
155    Now edit your Targets again:
156
157        $ sudo vi Targets
158       
159    Add a check for HTTP latency for all the classroom PCs.
160    This will mean adding another category, such as:
161
162        + HTTP Servers
163        probe = EchoPingHttp
164
165        ++ PC1
166        host = pc1
167
168        ++ PC2
169        host = pc2
170       
171        ...
172
173    If you have time, consider checking some machines that are
174    external to our classroom and the conference (your organization's
175    website, a popular web page, etc...)
176
1775. Add DNS Latency Checks
178
179    You can check either or both internal or external names using
180    the DNS latency probe.
181
182    Add a menu hierarchy for DNS Latency. Check an external address
183    (nsrc.org) and an internal address (noc). This will look something
184    like this (in Targets):
185
186        + DNS
187        probe = DNS
188        menu = External DNS Check
189        title = DNS Latency
190
191        ++ nsrc
192        host = nsrc.org
193
194        ++ noc
195        host = noc.mgmt
196
197    Exit and save your changes to the file Targets.
198
199    Restart Smokeping to see the changes:
200
201    $ sudo service smokeping restart
202
203    Look at additional Smokeping probes and consider implementing
204    some of them:
205
206        http://oss.oetiker.ch/smokeping/probe/index.en.html
207
208    As trying to explain all syntactical details of how the file
209    /etc/smokeping/config.d/Targets is used would require several
210    pages we will go through some examples in class, and you can
211    refer to the Smokeping configuration files that are in use on
212    the classroom NOC box by going to:
213
214        http://noc/configs/etc/smokeping
215        http://noc/configs/etc/smokeping/config.d
216
217
2186. Send Smokeping alerts (SKIP THIS EXERCISE FOR TRACK 2)
219
220    $ sudo vi Alerts
221
222    Update the top of the file where it says:
223
224        *** Alerts ***
225        to = alertee@address.somewhere
226        from = smokealert@company.xy
227
228    to include a proper "to" and "from" field for your server.
229    Something like:
230
231        *** Alerts ***
232        to = sysadm@localhost
233        from = smokeping-alert@localhost
234
235    If you have installed RT, you can instead send your alerts
236    to an existing RT queue:
237
238        *** Alerts ***
239        to = net@localhost
240
241    At the end of the file, add another alert like this:
242
243    +anydelay
244    type = rtt
245    # in milliseconds
246    pattern = >1
247    comment = Just for testing
248
249    Notice the pattern in this alert. It means that an alert will be triggered
250    as soon as a sample measurement has "ANY" delay, that is, more than one
251    millisecond. This is just for testing. In reality, you will want to create
252    an alert based on your observed baseline. For example, if your DNS servers'
253    delay suddendly goes from under 10 ms to over 100ms.
254
255    Next, be sure you have this test alert defined for some of your Targets.
256    You can either turn on alerts by defining alerts for a probe in
257    the /etc/smokeping/config.d/Probes file, or by individual Targets
258    entries.
259
260    In our case let's edit the Targets file and turn on alerts for our
261    DNS Latency checks.
262
263    $ sudo vi /etc/smokeping/config.d/Targets
264
265    Find the following section in the file:
266
267        + DNS
268        probe = DNS
269        menu = External DNS Check
270        title = DNS Latency
271
272        ++ nsrc
273        host = nsrc.org
274
275    And add the following alerts line after "+++ nsrc"
276
277        +++ nsrc
278        host = nsrc
279        alerts = anydelay
280
281    Save and exit from the file, then restart smokeping:
282
283    $ sudo service smokeping restart
284
285    Check your e-mail with mutt
286
287    $ mutt
288
289    (or check your RT queues)
290
291    And see if you have received alerts after 5 minutes.
292
2936. MultiHost Graphs
294
295    Once you have defined a group of hosts under a single probe type in your
296    /etc/smokeping/config.d/Targets file, then you can create a single graph
297    that will show you the results of all smokeping tests for all hosts that
298    you define. This has the advantage of letting you quickly compare, for
299    example, a group of hosts that you are monitoring with the FPing probe.
300
301    The MultiHost graph function in Smokeping is extremely picky - pay close
302        attention.
303
304    To create a MultiHost graph first edit the file Targets:
305
306    $ sudo vi Targets
307
308    If you had a section for the FPing probe defined that looked like this
309    (this is an example only - your Targets file may look different):
310
311        + Local
312        menu = Local
313        title = Local Network
314
315        ++ LocalMachine
316        menu = Local Machine
317        title = This host
318        host = localhost
319
320        ++ pc1
321        menu = pc1
322        title = pc1
323        host = pc1
324
325        ++ pc2
326        menu = pc2
327        title = pc2
328        host = pc2
329
330        ++ pc3
331        menu = pc3
332        title = pc3
333        host = pc3
334
335    Right now smokeping displays the results of the FPing probe for each
336    host defined in separate graphs. If you wish to see the results in a
337    single graph with multiple lines, then you would do this after the last
338    FPing probe host definition:
339
340        + MultiHostPCs
341        menu = MultiHost Ping
342        title = Consolidated Ping Response Time
343        host = /Local/LocalMachine /Local/pc1 /Local/pc2 /Local/pc3
344
345    (Note: if the lines get too long, you can have multiple lines for the
346    "host" entry by using the "\" character to indicate another line - ask about
347    this if you are unsure!)
348
349    Now save and exit the file Targets and restart smokeping:
350
351    $ sudo service smokeping restart
352
353    You should see a new graph under the "MultiHost Ping" menu in your
354    smokeping web interface. This graph will have different color lines
355    for each host you have defined.
356
357
3587. Slave instances - only done if we have the time.
359
360    This is a description only for informational purposes in case you wish
361    to attempt this type of configuration once the workshop is over.
362
363    The idea behind this is that you can run multiple smokeping instances
364    at multiple locations that are monitoring the same hosts and/or services
365    as your master instance. The slaves will send their results to the
366    master server and you will see these results side-by-side with your
367    local results. This allows you to view how users outside your network
368    see your services and hosts.
369
370    This can be a powerful tool for resolving service and host issues that
371    may be difficult to troubleshoot if you only have local data.
372
373    Graphically this looks this:
374
375          [slave 1]     [slave 2]      [slave 3]
376                |             |              |
377                +-------+     |     +--------+
378                        |     |     |
379                        v     v     v
380                        +---------------+
381                        |    master     |
382                        +---------------+
383
384    You can see example of this data here:
385
386    http://oss.oetiker.ch/smokeping-demo/
387
388    Look at the various graph groups and notice that many of the graphs
389    have multiple lines with the color code chart listing items such as
390    "median RTT from mipsrv01" - These are not MultiHost graphs, but rather
391    graphs with data from external smokeping servers.
392
393    To configure a smokeping master/slave server you can see the documentation
394    here:
395
396    http://oss.oetiker.ch/smokeping/doc/smokeping_master_slave.en.html
397
398    In addition, a sample set of steps for configuring this is available in
399    the file sample-smokeping-master-slave.txt which is available under the
400    attachments listing on the agenda page of the class wiki.
401