1 | <html> |
---|
2 | <head> |
---|
3 | <title>Smokeping Exercises</title> |
---|
4 | </head> |
---|
5 | <body> |
---|
6 | <pre> |
---|
7 | |
---|
8 | Network Monitoring and Management |
---|
9 | |
---|
10 | Smokeping |
---|
11 | --------- |
---|
12 | |
---|
13 | Notes: |
---|
14 | ------ |
---|
15 | * Commands preceded with "$" imply that you should execute the command as |
---|
16 | a general user - not as root. |
---|
17 | * Commands preceded with "#" imply that you should be working as root. |
---|
18 | * Commands with more specific command lines (e.g. "RTR-GW>" or "mysql>") |
---|
19 | imply that you are executing commands on remote equipment, or within |
---|
20 | another program. |
---|
21 | |
---|
22 | Exercises |
---|
23 | ---------- |
---|
24 | |
---|
25 | 0. Log in to your PC or open a terminal window as the sysadmn user. |
---|
26 | |
---|
27 | Once you are logged in you can continue with these exercises. |
---|
28 | |
---|
29 | 1. Install Smokeping |
---|
30 | |
---|
31 | $ sudo apt-get install smokeping |
---|
32 | |
---|
33 | 2. Initial Configuration |
---|
34 | |
---|
35 | $ cd /etc/smokeping/config.d |
---|
36 | $ ls -l |
---|
37 | |
---|
38 | -rwxr-xr-x 1 root root 578 2010-02-26 01:55 Alerts |
---|
39 | -rwxr-xr-x 1 root root 237 2010-02-26 01:55 Database |
---|
40 | -rwxr-xr-x 1 root root 413 2010-02-26 05:40 General |
---|
41 | -rwxr-xr-x 1 root root 271 2010-02-26 01:55 pathnames |
---|
42 | -rwxr-xr-x 1 root root 859 2010-02-26 01:55 Presentation |
---|
43 | -rwxr-xr-x 1 root root 116 2010-02-26 01:55 Probes |
---|
44 | -rwxr-xr-x 1 root root 155 2010-02-26 01:55 Slaves |
---|
45 | -rwxr-xr-x 1 root root 8990 2010-02-26 06:30 Targets |
---|
46 | |
---|
47 | $ sudo vi General |
---|
48 | |
---|
49 | Change the following lines: |
---|
50 | |
---|
51 | owner = NOC |
---|
52 | contact = sysadm@localhost |
---|
53 | cgiurl = http://localhost/cgi-bin/smokeping.cgi |
---|
54 | mailhost = localhost |
---|
55 | |
---|
56 | Save the file and exit. Now let's restart the |
---|
57 | Smokeping service to verify that no mistakes have been made |
---|
58 | before going any further: |
---|
59 | |
---|
60 | $ sudo service smokeping restart |
---|
61 | |
---|
62 | 2. Configure monitoring of devices |
---|
63 | |
---|
64 | The majority of your time and work configuring Smokeping |
---|
65 | will be done in the file /etc/smokeping/config.d/Targets. |
---|
66 | |
---|
67 | For this class please do the following: |
---|
68 | |
---|
69 | Use the default FPing probe to check: |
---|
70 | |
---|
71 | - all the student PCs |
---|
72 | - classroom NOC |
---|
73 | - switches |
---|
74 | - routers |
---|
75 | |
---|
76 | You can use the classroom Network Diagram on the classroom wiki to |
---|
77 | figure out addresses for each item, etc. |
---|
78 | |
---|
79 | ############################################################################### |
---|
80 | Please note - a complete set of configuration files is available on the class |
---|
81 | web server at http://noc.ws.nsrc.org/configs |
---|
82 | |
---|
83 | We strongly suggest you review these files to see what a complete set of |
---|
84 | Smokeping configuration files looks like. To review the Targets file the direct |
---|
85 | link is: |
---|
86 | |
---|
87 | <a href="http://noc/configs/etc/smokeping/config.d/Targets">http://noc/configs/etc/smokeping/config.d/Targets</a> |
---|
88 | |
---|
89 | ############################################################################### |
---|
90 | |
---|
91 | Create some hierarchy to the Smokeping menu for your |
---|
92 | checks. Such as: |
---|
93 | |
---|
94 | +Local |
---|
95 | |
---|
96 | menu = Network Monitoring and Management |
---|
97 | title = NOC Server for Network Monitoring Class |
---|
98 | |
---|
99 | ++LocalMachine |
---|
100 | |
---|
101 | menu = The NOC@NetManage |
---|
102 | title = The NOC@NetManage |
---|
103 | host = localhost |
---|
104 | |
---|
105 | # |
---|
106 | # Classroom PCs |
---|
107 | # |
---|
108 | |
---|
109 | ++PCs |
---|
110 | |
---|
111 | +++pc1 |
---|
112 | menu = pc1 |
---|
113 | title = pc1 |
---|
114 | host = pc1 |
---|
115 | |
---|
116 | +++pc2 |
---|
117 | menu = pc2 |
---|
118 | title = pc2 |
---|
119 | host = pc2 |
---|
120 | |
---|
121 | Save the file and restart Smokeping: |
---|
122 | |
---|
123 | $ sudo service smokeping restart |
---|
124 | |
---|
125 | Go to your browser and check the Smokeping page: |
---|
126 | |
---|
127 | http://pcN.ws.nsrc.org/cgi-bin/smokeping.cgi |
---|
128 | |
---|
129 | If everything is looking OK, continue adding: |
---|
130 | |
---|
131 | ++Routers |
---|
132 | |
---|
133 | +++rtr |
---|
134 | menu = rtr |
---|
135 | title = Gateway Router |
---|
136 | host = rtr |
---|
137 | |
---|
138 | +++rtr1 |
---|
139 | menu = rtr1 |
---|
140 | title = Router 1, Group 1 |
---|
141 | host = rtr1 |
---|
142 | |
---|
143 | ++Switch |
---|
144 | |
---|
145 | ++sw |
---|
146 | menu = sw |
---|
147 | title = Backbone Switch |
---|
148 | host = sw |
---|
149 | |
---|
150 | ... |
---|
151 | |
---|
152 | Save the file, restart smokeping, and check |
---|
153 | your browser again. |
---|
154 | |
---|
155 | 3. Add new probes |
---|
156 | |
---|
157 | The current entry in Probes is fine, but if you wish to |
---|
158 | use additional Smokeping checks you can add them in here |
---|
159 | and you can specify their default behavior. You can do |
---|
160 | this, as well, in the Targets file if you wish. |
---|
161 | |
---|
162 | Here is an example of a Probes file that would specify |
---|
163 | what to use to check for HTTP and DNS latency as well as |
---|
164 | the FPing probe that is used for ping latency: |
---|
165 | |
---|
166 | $ sudo vi Probes |
---|
167 | |
---|
168 | *** Probes *** |
---|
169 | |
---|
170 | + FPing |
---|
171 | |
---|
172 | binary = /usr/bin/fping |
---|
173 | |
---|
174 | + EchoPingHttp |
---|
175 | |
---|
176 | + DNS |
---|
177 | binary = /usr/bin/dig |
---|
178 | pings = 5 |
---|
179 | step = 180 |
---|
180 | lookup = www.nsrc.org |
---|
181 | |
---|
182 | Save the file. |
---|
183 | |
---|
184 | 4. Add HTTP latency checks |
---|
185 | |
---|
186 | Now edit your Targets again: |
---|
187 | |
---|
188 | $ sudo vi Targets |
---|
189 | |
---|
190 | Add a check for HTTP latency for all the classroom PCs. |
---|
191 | This will mean adding another category, such as: |
---|
192 | |
---|
193 | ++HTTP Servers |
---|
194 | probe = EchoPingHttp |
---|
195 | menu = HTTP Response |
---|
196 | title = HTTP Response Student PCs |
---|
197 | |
---|
198 | +++pc1 |
---|
199 | menu = pc1 |
---|
200 | title = PC1 HTTP Response Time |
---|
201 | host = pc1 |
---|
202 | |
---|
203 | +++pc1 |
---|
204 | host = pc2 |
---|
205 | title = PC2 HTTP Response Time |
---|
206 | host = pc2 |
---|
207 | |
---|
208 | ... |
---|
209 | |
---|
210 | If you have time, consider checking some machines that are |
---|
211 | external to our classroom and the conference (your organization's |
---|
212 | website, a popular web page, etc...) |
---|
213 | |
---|
214 | 5. Add DNS Latency Checks |
---|
215 | |
---|
216 | You can check either or both internal or external names using |
---|
217 | the DNS latency probe. |
---|
218 | |
---|
219 | Add a menu hierarchy for DNS Latency. Check an external address |
---|
220 | (nsrc.org) and an internal address (noc). This will look something |
---|
221 | like this (in Targets): |
---|
222 | |
---|
223 | +DNS |
---|
224 | probe = DNS |
---|
225 | menu = External DNS Check |
---|
226 | title = DNS Latency |
---|
227 | |
---|
228 | ++nsrc |
---|
229 | host = nsrc.org |
---|
230 | |
---|
231 | ++noc |
---|
232 | host = noc.mgmt |
---|
233 | |
---|
234 | Exit and save your changes to the file Targets. |
---|
235 | |
---|
236 | Restart Smokeping to see the changes: |
---|
237 | |
---|
238 | $ sudo service smokeping restart |
---|
239 | |
---|
240 | Look at additional Smokeping probes and consider implementing |
---|
241 | some of them: |
---|
242 | |
---|
243 | http://oss.oetiker.ch/smokeping/probe/index.en.html |
---|
244 | |
---|
245 | As trying to explain all syntactical details of how the file |
---|
246 | /etc/smokeping/config.d/Targets is used would require several |
---|
247 | pages we will go through some examples in class, and you can |
---|
248 | refer to the Smokeping configuration files that are in use on |
---|
249 | the classroom NOC box by going to: |
---|
250 | |
---|
251 | http://noc/configs/etc/smokeping |
---|
252 | http://noc/configs/etc/smokeping/config.d |
---|
253 | |
---|
254 | |
---|
255 | 6. Send Smokeping alerts |
---|
256 | |
---|
257 | $ sudo vi Alerts |
---|
258 | |
---|
259 | Update the top of the file where it says: |
---|
260 | |
---|
261 | *** Alerts *** |
---|
262 | to = alertee@address.somewhere |
---|
263 | from = smokealert@company.xy |
---|
264 | |
---|
265 | to include a proper "to" and "from" field for your server. |
---|
266 | Something like: |
---|
267 | |
---|
268 | *** Alerts *** |
---|
269 | to = sysadm@localhost |
---|
270 | from = smokeping-alert@localhost |
---|
271 | |
---|
272 | If you have installed RT, you can instead send your alerts |
---|
273 | to an existing RT queue: |
---|
274 | |
---|
275 | *** Alerts *** |
---|
276 | to = net@localhost |
---|
277 | |
---|
278 | At the end of the file, add another alert like this: |
---|
279 | |
---|
280 | +anydelay |
---|
281 | type = rtt |
---|
282 | # in milliseconds |
---|
283 | pattern = >1 |
---|
284 | comment = Just for testing |
---|
285 | |
---|
286 | Notice the pattern in this alert. It means that an alert will be triggered |
---|
287 | as soon as a sample measurement has "ANY" delay, that is, more than one |
---|
288 | millisecond. This is just for testing. In reality, you will want to create |
---|
289 | an alert based on your observed baseline. For example, if your DNS servers' |
---|
290 | delay suddendly goes from under 10 ms to over 100ms. |
---|
291 | |
---|
292 | Next, be sure you have this test alert defined for some of your Targets. |
---|
293 | You can either turn on alerts by defining alerts for a probe in |
---|
294 | the /etc/smokeping/config.d/Probes file, or by individual Targets |
---|
295 | entries. |
---|
296 | |
---|
297 | In our case let's edit the Targets file and turn on alerts for our |
---|
298 | DNS Latency checks. |
---|
299 | |
---|
300 | $ sudo vi /etc/smokeping/config.d/Targets |
---|
301 | |
---|
302 | Find the following section in the file: |
---|
303 | |
---|
304 | +DNS |
---|
305 | probe = DNS |
---|
306 | menu = External DNS Check |
---|
307 | title = DNS Latency |
---|
308 | |
---|
309 | ++nsrc |
---|
310 | host = nsrc.org |
---|
311 | |
---|
312 | And add the following alerts line after "+++ nsrc" |
---|
313 | |
---|
314 | +++nsrc |
---|
315 | host = nsrc |
---|
316 | alerts = anydelay |
---|
317 | |
---|
318 | Save and exit from the file, then restart smokeping: |
---|
319 | |
---|
320 | $ sudo service smokeping restart |
---|
321 | |
---|
322 | Check your e-mail with mutt |
---|
323 | |
---|
324 | $ mutt |
---|
325 | |
---|
326 | (or check your RT queues) |
---|
327 | |
---|
328 | And see if you have received alerts after 5 minutes. |
---|
329 | |
---|
330 | 6. MultiHost Graphs |
---|
331 | |
---|
332 | Once you have defined a group of hosts under a single probe type in your |
---|
333 | /etc/smokeping/config.d/Targets file, then you can create a single graph |
---|
334 | that will show you the results of all smokeping tests for all hosts that |
---|
335 | you define. This has the advantage of letting you quickly compare, for |
---|
336 | example, a group of hosts that you are monitoring with the FPing probe. |
---|
337 | |
---|
338 | The MultiHost graph function in Smokeping is extremely picky - pay close |
---|
339 | attention. |
---|
340 | |
---|
341 | To create a MultiHost graph first edit the file Targets: |
---|
342 | |
---|
343 | $ sudo vi Targets |
---|
344 | |
---|
345 | If you had a section for the FPing probe defined that looked like this |
---|
346 | (this is an example only - your Targets file may look different): |
---|
347 | |
---|
348 | +Local |
---|
349 | menu = Local |
---|
350 | title = Local Network |
---|
351 | |
---|
352 | ++LocalMachine |
---|
353 | menu = Local Machine |
---|
354 | title = This host |
---|
355 | host = localhost |
---|
356 | |
---|
357 | ++pc1 |
---|
358 | menu = pc1 |
---|
359 | title = pc1 |
---|
360 | host = pc1 |
---|
361 | |
---|
362 | ++pc2 |
---|
363 | menu = pc2 |
---|
364 | title = pc2 |
---|
365 | host = pc2 |
---|
366 | |
---|
367 | ++pc3 |
---|
368 | menu = pc3 |
---|
369 | title = pc3 |
---|
370 | host = pc3 |
---|
371 | |
---|
372 | Right now smokeping displays the results of the FPing probe for each |
---|
373 | host defined in separate graphs. If you wish to see the results in a |
---|
374 | single graph with multiple lines, then you would do this after the last |
---|
375 | FPing probe host definition: |
---|
376 | |
---|
377 | +MultiHostPCs |
---|
378 | menu = MultiHost Ping |
---|
379 | title = Consolidated Ping Response Time |
---|
380 | host = /Local/LocalMachine /Local/pc1 /Local/pc2 /Local/pc3 |
---|
381 | |
---|
382 | (Note: if the lines get too long, you can have multiple lines for the |
---|
383 | "host" entry by using the "\" character to indicate another line - ask about |
---|
384 | this if you are unsure!) |
---|
385 | |
---|
386 | Now save and exit the file Targets and restart smokeping: |
---|
387 | |
---|
388 | $ sudo service smokeping restart |
---|
389 | |
---|
390 | You should see a new graph under the "MultiHost Ping" menu in your |
---|
391 | smokeping web interface. This graph will have different color lines |
---|
392 | for each host you have defined. |
---|
393 | |
---|
394 | |
---|
395 | 7. Slave instances - only done if we have the time. |
---|
396 | |
---|
397 | This is a description only for informational purposes in case you wish |
---|
398 | to attempt this type of configuration once the workshop is over. |
---|
399 | |
---|
400 | The idea behind this is that you can run multiple smokeping instances |
---|
401 | at multiple locations that are monitoring the same hosts and/or services |
---|
402 | as your master instance. The slaves will send their results to the |
---|
403 | master server and you will see these results side-by-side with your |
---|
404 | local results. This allows you to view how users outside your network |
---|
405 | see your services and hosts. |
---|
406 | |
---|
407 | This can be a powerful tool for resolving service and host issues that |
---|
408 | may be difficult to troubleshoot if you only have local data. |
---|
409 | |
---|
410 | Graphically this looks this: |
---|
411 | |
---|
412 | [slave 1] [slave 2] [slave 3] |
---|
413 | | | | |
---|
414 | +-------+ | +--------+ |
---|
415 | | | | |
---|
416 | v v v |
---|
417 | +---------------+ |
---|
418 | | master | |
---|
419 | +---------------+ |
---|
420 | |
---|
421 | You can see example of this data here: |
---|
422 | |
---|
423 | http://oss.oetiker.ch/smokeping-demo/ |
---|
424 | |
---|
425 | Look at the various graph groups and notice that many of the graphs |
---|
426 | have multiple lines with the color code chart listing items such as |
---|
427 | "median RTT from mipsrv01" - These are not MultiHost graphs, but rather |
---|
428 | graphs with data from external smokeping servers. |
---|
429 | |
---|
430 | To configure a smokeping master/slave server you can see the documentation |
---|
431 | here: |
---|
432 | |
---|
433 | http://oss.oetiker.ch/smokeping/doc/smokeping_master_slave.en.html |
---|
434 | |
---|
435 | In addition, a sample set of steps for configuring this is available in |
---|
436 | the file sample-smokeping-master-slave.txt which is available under the |
---|
437 | attachments listing on the agenda page of the class wiki. |
---|
438 | </pre> |
---|
439 | </body> |
---|
440 | </html> |
---|