Agenda: exercises-smokeping.2.txt

File exercises-smokeping.2.txt, 10.8 KB (added by admin, 8 years ago)
Line 
1Network Management & Monitoring
2
3Smokeping
4
5Notes:
6------
7* Commands preceded with "$" imply that you should execute the command as
8  a general user - not as root.
9* Commands preceded with "#" imply that you should be working as root.
10* Commands with more specific command lines (e.g. "RTR-GW>" or "mysql>")
11  imply that you are executing commands on remote equipment, or within
12  another program.
13
14Exercises
15----------
16
170. Log in to your PC or open a terminal window as the sysadmin user.
18
19Once you are logged in you can continue with these exercises.
20
211. Install Smokeping
22
23    $ sudo apt-get install smokeping
24
252. Initial Configuration
26
27    $ cd /etc/smokeping/config.d
28    $ ls -l
29
30    -rwxr-xr-x 1 root root  578 2010-02-26 01:55 Alerts
31    -rwxr-xr-x 1 root root  237 2010-02-26 01:55 Database
32    -rwxr-xr-x 1 root root  413 2010-02-26 05:40 General
33    -rwxr-xr-x 1 root root  271 2010-02-26 01:55 pathnames
34    -rwxr-xr-x 1 root root  859 2010-02-26 01:55 Presentation
35    -rwxr-xr-x 1 root root  116 2010-02-26 01:55 Probes
36    -rwxr-xr-x 1 root root  155 2010-02-26 01:55 Slaves
37    -rwxr-xr-x 1 root root 8990 2010-02-26 06:30 Targets
38 
39    $ sudo vi General
40
41        Change the following lines:
42
43        owner    = NOC
44        contact  = sysadmin@localhost
45        cgiurl   = http://localhost/cgi-bin/smokeping.cgi
46        mailhost = localhost
47
48        Save the file and exit. Now let's restart the
49        Smokeping service to verify that no mistakes have been made
50        before going any further:
51
52        $ sudo /etc/init.d/smokeping stop
53        $ sudo /etc/init.d/smokeping start
54
552. Configure monitoring of devices
56
57        The majority of your time and work configuring Smokeping
58        will be done in the file /etc/smokeping/config.d/Targets.
59       
60        For this class please do the following:
61
62        Use the default FPing probe to check:
63
64      - all the student NOC PCs
65      - classroom NOC
66      - switches
67      - routers
68     
69    You can use the classroom Network Diagram on the classroom wiki to
70    figure out addresses for each item, etc.
71
72    Create some hierarchy to the Smokeping menu for your
73    checks. Such as:
74
75        + PCs
76
77        ++ pc1
78        menu = pc1
79        title = pc1
80        host = pc1
81
82        ++ pc2
83        menu = pc2
84        title = pc2
85        host = pc2
86       
87    Save the file and restart Smokeping:
88
89        $ sudo /etc/init.d/smokeping stop
90        $ sudo /etc/init.d/smokeping start
91
92    Go to your browser and check the Smokeping page:
93
94        http://10.10.x.y/cgi-bin/smokeping.cgi
95
96    If everything is looking OK, continue adding:
97
98        + Routers
99
100        ++ bb-gw
101        menu = bb-gw
102        title = bb-gw
103        menu = bb-gw
104
105        ++ r1
106        menu = r1
107        title = r1
108        host = r1
109
110        + Switches
111
112        ++ bb-sw
113        menu = bb-sw
114        title = bb-sw
115        menu = bb-sw   
116 
117        ...
118
119    Save the file, restart smokeping, and check
120    your browser again.
121
1223. Add new probes
123
124    The current entry in Probes is fine, but if you wish to
125    use additional Smokeping checks you can add them in here
126    and you can specify their default behavior. You can do
127    this, as well, in the Targets file if you wish.
128
129    Here is an example of a Probes file that would specify
130    what to use to check for HTTP and DNS latency as well as
131    the FPing probe that is used for ping latency:
132
133       $ sudo vi Probes
134
135        *** Probes ***
136
137        + FPing
138
139        binary = /usr/bin/fping
140
141        + EchoPingHttp
142
143        + DNS
144        binary = /usr/bin/dig
145        pings = 5
146        step = 180
147        lookup = www.nsrc.org
148
149        Save the file.
150
1514. Add HTTP latency checks
152
153    Now edit your Targets again:
154
155        $ sudo vi Targets
156       
157    Add a check for HTTP latency for all the classroom PCs.
158    This will mean adding another category, such as:
159
160        + HTTP Servers
161        probe = EchoPingHttp
162
163        ++ PC1
164        host = pc1
165
166        ++ PC2
167        host = pc2
168       
169        ...
170
171    If you have time, consider checking some machines that are
172    external to our classroom and the conference (your organization's
173    website, a popular web page, etc...)
174
1755. Add DNS Latency Checks
176
177    You can check either or both internal or external names using
178    the DNS latency probe.
179
180    Add a menu hierarchy for DNS Latency. Check an external address
181    (nsrc.org) and an internal address (noc). This will look something
182    like this (in Targets):
183
184        + DNS
185        probe = DNS
186        menu = External DNS Check
187        title = DNS Latency
188
189        ++ nsrc
190        host = nsrc.org
191
192        ++ noc
193        host = noc.mgmt
194
195    Exit and save your changes to the file Targets.
196
197    Restart Smokeping to see the changes:
198
199    $ sudo /etc/init.d/smokeping stop
200    $ sudo /etc/init.d/smokeping start
201
202    Look at additional Smokeping probes and consider implementing
203    some of them:
204
205        http://oss.oetiker.ch/smokeping/probe/index.en.html
206
207    As trying to explain all syntactical details of how the file
208    /etc/smokeping/config.d/Targets is used would require several
209    pages we will go through some examples in class, and you can
210    refer to the Smokeping configuration files that are in use on
211    the classroom NOC box by going to:
212
213        http://noc/configs/etc/smokeping
214        http://noc/configs/etc/smokeping/config.d
215
216
2176. Send Smokeping alerts
218
219    $ sudo vi Alerts
220
221    Update the top of the file where it says:
222
223        *** Alerts ***
224        to = alertee@address.somewhere
225        from = smokealert@company.xy
226
227    to include a proper "to" and "from" field for your server.
228    Something like:
229
230        *** Alerts ***
231        to = sysadmin@localhost
232        from = smokeping-alert@localhost
233
234    If you have installed RT, you can instead send your alerts
235    to an existing RT queue:
236
237        *** Alerts ***
238        to = net@localhost
239
240    At the end of the file, add another alert like this:
241
242    +anydelay
243    type = rtt
244    # in milliseconds
245    pattern = >1
246    comment = Just for testing
247
248    Notice the pattern in this alert. It means that an alert will be triggered
249    as soon as a sample measurement has "ANY" delay, that is, more than one
250    millisecond. This is just for testing. In reality, you will want to create
251    an alert based on your observed baseline. For example, if your DNS servers'
252    delay suddendly goes from under 10 ms to over 100ms.
253
254    Next, be sure you have this test alert defined for some of your Targets.
255    You can either turn on alerts by defining alerts for a probe in
256    the /etc/smokeping/config.d/Probes file, or by individual Targets
257    entries.
258
259    In our case let's edit the Targets file and turn on alerts for our
260    DNS Latency checks.
261
262    $ sudo vi /etc/smokeping/config.d/Targets
263
264    Find the following section in the file:
265
266        + DNS
267        probe = DNS
268        menu = External DNS Check
269        title = DNS Latency
270
271        ++ nsrc
272        host = nsrc.org
273
274    And add the following alerts line after "+++ nsrc"
275
276        +++ nsrc
277        host = nsrc
278        alerts = anydelay
279
280    Save and exit from the file, then restart smokeping:
281
282    $ sudo /etc/init.d/smokeping stop
283    $ sudo /etc/init.d/smokeping start
284
285    Check your e-mail with mutt
286
287    $ mutt
288
289    (or check your RT queues)
290
291    And see if you have received alerts after 5 minutes.
292
2936. MultiHost Graphs
294
295    Once you have defined a group of hosts under a single probe type in your
296    /etc/smokeping/config.d/Targets file, then you can create a single graph
297    that will show you the results of all smokeping tests for all hosts that
298    you define. This has the advantage of letting you quickly compare, for
299    example, a group of hosts that you are monitoring with the FPing probe.
300
301    The MultiHost graph function in Smokeping is extremely picky - pay close
302        attention.
303
304    To create a MultiHost graph first edit the file Targets:
305
306    $ sudo vi Targets
307
308    If you had a section for the FPing probe defined that looked like this
309    (this is an example only - your Targets file may look different):
310
311        + Local
312        menu = Local
313        title = Local Network
314
315        ++ LocalMachine
316        menu = Local Machine
317        title = This host
318        host = localhost
319
320        ++ pc1
321        menu = pc1
322        title = pc1
323        host = pc1
324
325        ++ pc2
326        menu = pc2
327        title = pc2
328        host = pc2
329
330        ++ pc3
331        menu = pc3
332        title = pc3
333        host = pc3
334
335    Right now smokeping displays the results of the FPing probe for each
336    host defined in separate graphs. If you wish to see the results in a
337    single graph with multiple lines, then you would do this after the last
338    FPing probe host definition:
339
340        + MultiHostPCs
341        menu = MultiHost Ping
342        title = Consolidated Ping Response Time
343        host = /Local/LocalMachine /Local/pc1 /Local/pc2 /Local/pc3
344
345    (Note: if the lines get too long, you can have multiple lines for the
346    "host" entry by using the "\" character to indicate another line - ask about
347    this if you are unsure!)
348
349    Now save and exit the file Targets and restart smokeping:
350
351    $ sudo /etc/init.d/smokeping stop
352    $ sudo /etc/init.d/smokeping start
353
354    You should see a new graph under the "MultiHost Ping" menu in your
355    smokeping web interface. This graph will have different color lines
356    for each host you have defined.
357
358
3597. Slave instances - only done if we have the time.
360
361    This is a description only for informational purposes in case you wish
362    to attempt this type of configuration once the workshop is over.
363
364    The idea behind this is that you can run multiple smokeping instances
365    at multiple locations that are monitoring the same hosts and/or services
366    as your master instance. The slaves will send their results to the
367    master server and you will see these results side-by-side with your
368    local results. This allows you to view how users outside your network
369    see your services and hosts.
370
371    This can be a powerful tool for resolving service and host issues that
372    may be difficult to troubleshoot if you only have local data.
373
374    Graphically this looks this:
375
376          [slave 1]     [slave 2]      [slave 3]
377                |             |              |
378                +-------+     |     +--------+
379                        |     |     |
380                        v     v     v
381                        +---------------+
382                        |    master     |
383                        +---------------+
384
385    You can see example of this data here:
386
387    http://oss.oetiker.ch/smokeping-demo/
388
389    Look at the various graph groups and notice that many of the graphs
390    have multiple lines with the color code chart listing items such as
391    "median RTT from mipsrv01" - These are not MultiHost graphs, but rather
392    graphs with data from external smokeping servers.
393
394    To configure a smokeping master/slave server you can see the documentation
395    here:
396
397    http://oss.oetiker.ch/smokeping/doc/smokeping_master_slave.en.html
398
399    In addition, a sample set of steps for configuring this is available in
400    the file sample-smokeping-master-slave.txt.
401