1 | Network Management & Monitoring |
---|
2 | |
---|
3 | Smokeping |
---|
4 | |
---|
5 | Notes: |
---|
6 | ------ |
---|
7 | * Commands preceded with "$" imply that you should execute the command as |
---|
8 | a general user - not as root. |
---|
9 | * Commands preceded with "#" imply that you should be working as root. |
---|
10 | * Commands with more specific command lines (e.g. "RTR-GW>" or "mysql>") |
---|
11 | imply that you are executing commands on remote equipment, or within |
---|
12 | another program. |
---|
13 | |
---|
14 | Exercises |
---|
15 | ---------- |
---|
16 | |
---|
17 | 0. Log in to your PC or open a terminal window as the sysadmin user. |
---|
18 | |
---|
19 | Once you are logged in you can continue with these exercises. |
---|
20 | |
---|
21 | 1. Install Smokeping |
---|
22 | |
---|
23 | $ sudo apt-get install smokeping |
---|
24 | |
---|
25 | 2. Initial Configuration |
---|
26 | |
---|
27 | $ cd /etc/smokeping/config.d |
---|
28 | $ ls -l |
---|
29 | |
---|
30 | -rwxr-xr-x 1 root root 578 2010-02-26 01:55 Alerts |
---|
31 | -rwxr-xr-x 1 root root 237 2010-02-26 01:55 Database |
---|
32 | -rwxr-xr-x 1 root root 413 2010-02-26 05:40 General |
---|
33 | -rwxr-xr-x 1 root root 271 2010-02-26 01:55 pathnames |
---|
34 | -rwxr-xr-x 1 root root 859 2010-02-26 01:55 Presentation |
---|
35 | -rwxr-xr-x 1 root root 116 2010-02-26 01:55 Probes |
---|
36 | -rwxr-xr-x 1 root root 155 2010-02-26 01:55 Slaves |
---|
37 | -rwxr-xr-x 1 root root 8990 2010-02-26 06:30 Targets |
---|
38 | |
---|
39 | $ sudo vi General |
---|
40 | |
---|
41 | Change the following lines: |
---|
42 | |
---|
43 | owner = NOC |
---|
44 | contact = sysadmin@localhost |
---|
45 | cgiurl = http://localhost/cgi-bin/smokeping.cgi |
---|
46 | mailhost = localhost |
---|
47 | |
---|
48 | Save the file and exit. Now let's restart the |
---|
49 | Smokeping service to verify that no mistakes have been made |
---|
50 | before going any further: |
---|
51 | |
---|
52 | $ sudo /etc/init.d/smokeping stop |
---|
53 | $ sudo /etc/init.d/smokeping start |
---|
54 | |
---|
55 | 2. Configure monitoring of devices |
---|
56 | |
---|
57 | The majority of your time and work configuring Smokeping |
---|
58 | will be done in the file /etc/smokeping/config.d/Targets. |
---|
59 | |
---|
60 | For this class please do the following: |
---|
61 | |
---|
62 | Use the default FPing probe to check: |
---|
63 | |
---|
64 | - all the student NOC PCs |
---|
65 | - classroom NOC |
---|
66 | - switches |
---|
67 | - routers |
---|
68 | |
---|
69 | You can use the classroom Network Diagram on the classroom wiki to |
---|
70 | figure out addresses for each item, etc. |
---|
71 | |
---|
72 | Create some hierarchy to the Smokeping menu for your |
---|
73 | checks. Such as: |
---|
74 | |
---|
75 | + PCs |
---|
76 | |
---|
77 | ++ pc1 |
---|
78 | menu = pc1 |
---|
79 | title = pc1 |
---|
80 | host = pc1 |
---|
81 | |
---|
82 | ++ pc2 |
---|
83 | menu = pc2 |
---|
84 | title = pc2 |
---|
85 | host = pc2 |
---|
86 | |
---|
87 | Save the file and restart Smokeping: |
---|
88 | |
---|
89 | $ sudo /etc/init.d/smokeping stop |
---|
90 | $ sudo /etc/init.d/smokeping start |
---|
91 | |
---|
92 | Go to your browser and check the Smokeping page: |
---|
93 | |
---|
94 | http://10.10.x.y/cgi-bin/smokeping.cgi |
---|
95 | |
---|
96 | If everything is looking OK, continue adding: |
---|
97 | |
---|
98 | + Routers |
---|
99 | |
---|
100 | ++ bb-gw |
---|
101 | menu = bb-gw |
---|
102 | title = bb-gw |
---|
103 | menu = bb-gw |
---|
104 | |
---|
105 | ++ r1 |
---|
106 | menu = r1 |
---|
107 | title = r1 |
---|
108 | host = r1 |
---|
109 | |
---|
110 | + Switches |
---|
111 | |
---|
112 | ++ bb-sw |
---|
113 | menu = bb-sw |
---|
114 | title = bb-sw |
---|
115 | menu = bb-sw |
---|
116 | |
---|
117 | ... |
---|
118 | |
---|
119 | Save the file, restart smokeping, and check |
---|
120 | your browser again. |
---|
121 | |
---|
122 | 3. Add new probes |
---|
123 | |
---|
124 | The current entry in Probes is fine, but if you wish to |
---|
125 | use additional Smokeping checks you can add them in here |
---|
126 | and you can specify their default behavior. You can do |
---|
127 | this, as well, in the Targets file if you wish. |
---|
128 | |
---|
129 | Here is an example of a Probes file that would specify |
---|
130 | what to use to check for HTTP and DNS latency as well as |
---|
131 | the FPing probe that is used for ping latency: |
---|
132 | |
---|
133 | $ sudo vi Probes |
---|
134 | |
---|
135 | *** Probes *** |
---|
136 | |
---|
137 | + FPing |
---|
138 | |
---|
139 | binary = /usr/bin/fping |
---|
140 | |
---|
141 | + EchoPingHttp |
---|
142 | |
---|
143 | + DNS |
---|
144 | binary = /usr/bin/dig |
---|
145 | pings = 5 |
---|
146 | step = 180 |
---|
147 | lookup = www.nsrc.org |
---|
148 | |
---|
149 | Save the file. |
---|
150 | |
---|
151 | 4. Add HTTP latency checks |
---|
152 | |
---|
153 | Now edit your Targets again: |
---|
154 | |
---|
155 | $ sudo vi Targets |
---|
156 | |
---|
157 | Add a check for HTTP latency for all the classroom PCs. |
---|
158 | This will mean adding another category, such as: |
---|
159 | |
---|
160 | + HTTP Servers |
---|
161 | probe = EchoPingHttp |
---|
162 | |
---|
163 | ++ PC1 |
---|
164 | host = pc1 |
---|
165 | |
---|
166 | ++ PC2 |
---|
167 | host = pc2 |
---|
168 | |
---|
169 | ... |
---|
170 | |
---|
171 | If you have time, consider checking some machines that are |
---|
172 | external to our classroom and the conference (your organization's |
---|
173 | website, a popular web page, etc...) |
---|
174 | |
---|
175 | 5. Add DNS Latency Checks |
---|
176 | |
---|
177 | You can check either or both internal or external names using |
---|
178 | the DNS latency probe. |
---|
179 | |
---|
180 | Add a menu hierarchy for DNS Latency. Check an external address |
---|
181 | (nsrc.org) and an internal address (noc). This will look something |
---|
182 | like this (in Targets): |
---|
183 | |
---|
184 | + DNS |
---|
185 | probe = DNS |
---|
186 | menu = External DNS Check |
---|
187 | title = DNS Latency |
---|
188 | |
---|
189 | ++ nsrc |
---|
190 | host = nsrc.org |
---|
191 | |
---|
192 | ++ noc |
---|
193 | host = noc.mgmt |
---|
194 | |
---|
195 | Exit and save your changes to the file Targets. |
---|
196 | |
---|
197 | Restart Smokeping to see the changes: |
---|
198 | |
---|
199 | $ sudo /etc/init.d/smokeping stop |
---|
200 | $ sudo /etc/init.d/smokeping start |
---|
201 | |
---|
202 | Look at additional Smokeping probes and consider implementing |
---|
203 | some of them: |
---|
204 | |
---|
205 | http://oss.oetiker.ch/smokeping/probe/index.en.html |
---|
206 | |
---|
207 | As trying to explain all syntactical details of how the file |
---|
208 | /etc/smokeping/config.d/Targets is used would require several |
---|
209 | pages we will go through some examples in class, and you can |
---|
210 | refer to the Smokeping configuration files that are in use on |
---|
211 | the classroom NOC box by going to: |
---|
212 | |
---|
213 | http://noc/configs/etc/smokeping |
---|
214 | http://noc/configs/etc/smokeping/config.d |
---|
215 | |
---|
216 | |
---|
217 | 6. Send Smokeping alerts |
---|
218 | |
---|
219 | $ sudo vi Alerts |
---|
220 | |
---|
221 | Update the top of the file where it says: |
---|
222 | |
---|
223 | *** Alerts *** |
---|
224 | to = alertee@address.somewhere |
---|
225 | from = smokealert@company.xy |
---|
226 | |
---|
227 | to include a proper "to" and "from" field for your server. |
---|
228 | Something like: |
---|
229 | |
---|
230 | *** Alerts *** |
---|
231 | to = sysadmin@localhost |
---|
232 | from = smokeping-alert@localhost |
---|
233 | |
---|
234 | If you have installed RT, you can instead send your alerts |
---|
235 | to an existing RT queue: |
---|
236 | |
---|
237 | *** Alerts *** |
---|
238 | to = net@localhost |
---|
239 | |
---|
240 | At the end of the file, add another alert like this: |
---|
241 | |
---|
242 | +anydelay |
---|
243 | type = rtt |
---|
244 | # in milliseconds |
---|
245 | pattern = >1 |
---|
246 | comment = Just for testing |
---|
247 | |
---|
248 | Notice the pattern in this alert. It means that an alert will be triggered |
---|
249 | as soon as a sample measurement has "ANY" delay, that is, more than one |
---|
250 | millisecond. This is just for testing. In reality, you will want to create |
---|
251 | an alert based on your observed baseline. For example, if your DNS servers' |
---|
252 | delay suddendly goes from under 10 ms to over 100ms. |
---|
253 | |
---|
254 | Next, be sure you have this test alert defined for some of your Targets. |
---|
255 | You can either turn on alerts by defining alerts for a probe in |
---|
256 | the /etc/smokeping/config.d/Probes file, or by individual Targets |
---|
257 | entries. |
---|
258 | |
---|
259 | In our case let's edit the Targets file and turn on alerts for our |
---|
260 | DNS Latency checks. |
---|
261 | |
---|
262 | $ sudo vi /etc/smokeping/config.d/Targets |
---|
263 | |
---|
264 | Find the following section in the file: |
---|
265 | |
---|
266 | + DNS |
---|
267 | probe = DNS |
---|
268 | menu = External DNS Check |
---|
269 | title = DNS Latency |
---|
270 | |
---|
271 | ++ nsrc |
---|
272 | host = nsrc.org |
---|
273 | |
---|
274 | And add the following alerts line after "+++ nsrc" |
---|
275 | |
---|
276 | +++ nsrc |
---|
277 | host = nsrc |
---|
278 | alerts = anydelay |
---|
279 | |
---|
280 | Save and exit from the file, then restart smokeping: |
---|
281 | |
---|
282 | $ sudo /etc/init.d/smokeping stop |
---|
283 | $ sudo /etc/init.d/smokeping start |
---|
284 | |
---|
285 | Check your e-mail with mutt |
---|
286 | |
---|
287 | $ mutt |
---|
288 | |
---|
289 | (or check your RT queues) |
---|
290 | |
---|
291 | And see if you have received alerts after 5 minutes. |
---|
292 | |
---|
293 | 6. MultiHost Graphs |
---|
294 | |
---|
295 | Once you have defined a group of hosts under a single probe type in your |
---|
296 | /etc/smokeping/config.d/Targets file, then you can create a single graph |
---|
297 | that will show you the results of all smokeping tests for all hosts that |
---|
298 | you define. This has the advantage of letting you quickly compare, for |
---|
299 | example, a group of hosts that you are monitoring with the FPing probe. |
---|
300 | |
---|
301 | The MultiHost graph function in Smokeping is extremely picky - pay close |
---|
302 | attention. |
---|
303 | |
---|
304 | To create a MultiHost graph first edit the file Targets: |
---|
305 | |
---|
306 | $ sudo vi Targets |
---|
307 | |
---|
308 | If you had a section for the FPing probe defined that looked like this |
---|
309 | (this is an example only - your Targets file may look different): |
---|
310 | |
---|
311 | + Local |
---|
312 | menu = Local |
---|
313 | title = Local Network |
---|
314 | |
---|
315 | ++ LocalMachine |
---|
316 | menu = Local Machine |
---|
317 | title = This host |
---|
318 | host = localhost |
---|
319 | |
---|
320 | ++ pc1 |
---|
321 | menu = pc1 |
---|
322 | title = pc1 |
---|
323 | host = pc1 |
---|
324 | |
---|
325 | ++ pc2 |
---|
326 | menu = pc2 |
---|
327 | title = pc2 |
---|
328 | host = pc2 |
---|
329 | |
---|
330 | ++ pc3 |
---|
331 | menu = pc3 |
---|
332 | title = pc3 |
---|
333 | host = pc3 |
---|
334 | |
---|
335 | Right now smokeping displays the results of the FPing probe for each |
---|
336 | host defined in separate graphs. If you wish to see the results in a |
---|
337 | single graph with multiple lines, then you would do this after the last |
---|
338 | FPing probe host definition: |
---|
339 | |
---|
340 | + MultiHostPCs |
---|
341 | menu = MultiHost Ping |
---|
342 | title = Consolidated Ping Response Time |
---|
343 | host = /Local/LocalMachine /Local/pc1 /Local/pc2 /Local/pc3 |
---|
344 | |
---|
345 | (Note: if the lines get too long, you can have multiple lines for the |
---|
346 | "host" entry by using the "\" character to indicate another line - ask about |
---|
347 | this if you are unsure!) |
---|
348 | |
---|
349 | Now save and exit the file Targets and restart smokeping: |
---|
350 | |
---|
351 | $ sudo /etc/init.d/smokeping stop |
---|
352 | $ sudo /etc/init.d/smokeping start |
---|
353 | |
---|
354 | You should see a new graph under the "MultiHost Ping" menu in your |
---|
355 | smokeping web interface. This graph will have different color lines |
---|
356 | for each host you have defined. |
---|
357 | |
---|
358 | |
---|
359 | 7. Slave instances - only done if we have the time. |
---|
360 | |
---|
361 | This is a description only for informational purposes in case you wish |
---|
362 | to attempt this type of configuration once the workshop is over. |
---|
363 | |
---|
364 | The idea behind this is that you can run multiple smokeping instances |
---|
365 | at multiple locations that are monitoring the same hosts and/or services |
---|
366 | as your master instance. The slaves will send their results to the |
---|
367 | master server and you will see these results side-by-side with your |
---|
368 | local results. This allows you to view how users outside your network |
---|
369 | see your services and hosts. |
---|
370 | |
---|
371 | This can be a powerful tool for resolving service and host issues that |
---|
372 | may be difficult to troubleshoot if you only have local data. |
---|
373 | |
---|
374 | Graphically this looks this: |
---|
375 | |
---|
376 | [slave 1] [slave 2] [slave 3] |
---|
377 | | | | |
---|
378 | +-------+ | +--------+ |
---|
379 | | | | |
---|
380 | v v v |
---|
381 | +---------------+ |
---|
382 | | master | |
---|
383 | +---------------+ |
---|
384 | |
---|
385 | You can see example of this data here: |
---|
386 | |
---|
387 | http://oss.oetiker.ch/smokeping-demo/ |
---|
388 | |
---|
389 | Look at the various graph groups and notice that many of the graphs |
---|
390 | have multiple lines with the color code chart listing items such as |
---|
391 | "median RTT from mipsrv01" - These are not MultiHost graphs, but rather |
---|
392 | graphs with data from external smokeping servers. |
---|
393 | |
---|
394 | To configure a smokeping master/slave server you can see the documentation |
---|
395 | here: |
---|
396 | |
---|
397 | http://oss.oetiker.ch/smokeping/doc/smokeping_master_slave.en.html |
---|
398 | |
---|
399 | In addition, a sample set of steps for configuring this is available in |
---|
400 | the file sample-smokeping-master-slave.txt. |
---|
401 | |
---|