| 1 | <html> |
|---|
| 2 | <head> |
|---|
| 3 | <title>Smokeping Exercises</title> |
|---|
| 4 | </head> |
|---|
| 5 | <body> |
|---|
| 6 | <pre> |
|---|
| 7 | |
|---|
| 8 | Network Monitoring and Management |
|---|
| 9 | |
|---|
| 10 | Smokeping |
|---|
| 11 | --------- |
|---|
| 12 | |
|---|
| 13 | Notes: |
|---|
| 14 | ------ |
|---|
| 15 | * Commands preceded with "$" imply that you should execute the command as |
|---|
| 16 | a general user - not as root. |
|---|
| 17 | * Commands preceded with "#" imply that you should be working as root. |
|---|
| 18 | * Commands with more specific command lines (e.g. "RTR-GW>" or "mysql>") |
|---|
| 19 | imply that you are executing commands on remote equipment, or within |
|---|
| 20 | another program. |
|---|
| 21 | |
|---|
| 22 | Exercises |
|---|
| 23 | ---------- |
|---|
| 24 | |
|---|
| 25 | 0. Log in to your PC or open a terminal window as the sysadmn user. |
|---|
| 26 | |
|---|
| 27 | Once you are logged in you can continue with these exercises. |
|---|
| 28 | |
|---|
| 29 | 1. Install Smokeping |
|---|
| 30 | |
|---|
| 31 | $ sudo apt-get install smokeping |
|---|
| 32 | |
|---|
| 33 | 2. Initial Configuration |
|---|
| 34 | |
|---|
| 35 | $ cd /etc/smokeping/config.d |
|---|
| 36 | $ ls -l |
|---|
| 37 | |
|---|
| 38 | -rwxr-xr-x 1 root root 578 2010-02-26 01:55 Alerts |
|---|
| 39 | -rwxr-xr-x 1 root root 237 2010-02-26 01:55 Database |
|---|
| 40 | -rwxr-xr-x 1 root root 413 2010-02-26 05:40 General |
|---|
| 41 | -rwxr-xr-x 1 root root 271 2010-02-26 01:55 pathnames |
|---|
| 42 | -rwxr-xr-x 1 root root 859 2010-02-26 01:55 Presentation |
|---|
| 43 | -rwxr-xr-x 1 root root 116 2010-02-26 01:55 Probes |
|---|
| 44 | -rwxr-xr-x 1 root root 155 2010-02-26 01:55 Slaves |
|---|
| 45 | -rwxr-xr-x 1 root root 8990 2010-02-26 06:30 Targets |
|---|
| 46 | |
|---|
| 47 | $ sudo vi General |
|---|
| 48 | |
|---|
| 49 | Change the following lines: |
|---|
| 50 | |
|---|
| 51 | owner = NOC |
|---|
| 52 | contact = sysadm@localhost |
|---|
| 53 | cgiurl = http://localhost/cgi-bin/smokeping.cgi |
|---|
| 54 | mailhost = localhost |
|---|
| 55 | |
|---|
| 56 | Save the file and exit. Now let's restart the |
|---|
| 57 | Smokeping service to verify that no mistakes have been made |
|---|
| 58 | before going any further: |
|---|
| 59 | |
|---|
| 60 | $ sudo service smokeping restart |
|---|
| 61 | |
|---|
| 62 | 2. Configure monitoring of devices |
|---|
| 63 | |
|---|
| 64 | The majority of your time and work configuring Smokeping |
|---|
| 65 | will be done in the file /etc/smokeping/config.d/Targets. |
|---|
| 66 | |
|---|
| 67 | For this class please do the following: |
|---|
| 68 | |
|---|
| 69 | Use the default FPing probe to check: |
|---|
| 70 | |
|---|
| 71 | - all the student PCs |
|---|
| 72 | - classroom NOC |
|---|
| 73 | - switches |
|---|
| 74 | - routers |
|---|
| 75 | |
|---|
| 76 | You can use the classroom Network Diagram on the classroom wiki to |
|---|
| 77 | figure out addresses for each item, etc. |
|---|
| 78 | |
|---|
| 79 | ############################################################################### |
|---|
| 80 | Please note - a complete set of configuration files is available on the class |
|---|
| 81 | web server at http://noc.ws.nsrc.org/configs |
|---|
| 82 | |
|---|
| 83 | We strongly suggest you review these files to see what a complete set of |
|---|
| 84 | Smokeping configuration files looks like. To review the Targets file the direct |
|---|
| 85 | link is: |
|---|
| 86 | |
|---|
| 87 | <a href="http://noc/configs/etc/smokeping/config.d/Targets">http://noc/configs/etc/smokeping/config.d/Targets</a> |
|---|
| 88 | |
|---|
| 89 | ############################################################################### |
|---|
| 90 | |
|---|
| 91 | Create some hierarchy to the Smokeping menu for your |
|---|
| 92 | checks. Such as: |
|---|
| 93 | |
|---|
| 94 | +Local |
|---|
| 95 | |
|---|
| 96 | menu = Network Monitoring and Management |
|---|
| 97 | title = NOC Server for Network Monitoring Class |
|---|
| 98 | |
|---|
| 99 | ++LocalMachine |
|---|
| 100 | |
|---|
| 101 | menu = The NOC@NetManage |
|---|
| 102 | title = The NOC@NetManage |
|---|
| 103 | host = localhost |
|---|
| 104 | |
|---|
| 105 | # |
|---|
| 106 | # Classroom PCs |
|---|
| 107 | # |
|---|
| 108 | |
|---|
| 109 | ++PCs |
|---|
| 110 | |
|---|
| 111 | +++pc1 |
|---|
| 112 | menu = pc1 |
|---|
| 113 | title = pc1 |
|---|
| 114 | host = pc1 |
|---|
| 115 | |
|---|
| 116 | +++pc2 |
|---|
| 117 | menu = pc2 |
|---|
| 118 | title = pc2 |
|---|
| 119 | host = pc2 |
|---|
| 120 | |
|---|
| 121 | Save the file and restart Smokeping: |
|---|
| 122 | |
|---|
| 123 | $ sudo service smokeping restart |
|---|
| 124 | |
|---|
| 125 | Go to your browser and check the Smokeping page: |
|---|
| 126 | |
|---|
| 127 | http://pcN.ws.nsrc.org/cgi-bin/smokeping.cgi |
|---|
| 128 | |
|---|
| 129 | If everything is looking OK, continue adding: |
|---|
| 130 | |
|---|
| 131 | ++Routers |
|---|
| 132 | |
|---|
| 133 | +++rtr |
|---|
| 134 | menu = rtr |
|---|
| 135 | title = Gateway Router |
|---|
| 136 | host = rtr |
|---|
| 137 | |
|---|
| 138 | +++rtr1 |
|---|
| 139 | menu = rtr1 |
|---|
| 140 | title = Router 1, Group 1 |
|---|
| 141 | host = rtr1 |
|---|
| 142 | |
|---|
| 143 | ++Switch |
|---|
| 144 | |
|---|
| 145 | ++sw |
|---|
| 146 | menu = sw |
|---|
| 147 | title = Backbone Switch |
|---|
| 148 | host = sw |
|---|
| 149 | |
|---|
| 150 | ... |
|---|
| 151 | |
|---|
| 152 | Save the file, restart smokeping, and check |
|---|
| 153 | your browser again. |
|---|
| 154 | |
|---|
| 155 | 3. Add new probes |
|---|
| 156 | |
|---|
| 157 | The current entry in Probes is fine, but if you wish to |
|---|
| 158 | use additional Smokeping checks you can add them in here |
|---|
| 159 | and you can specify their default behavior. You can do |
|---|
| 160 | this, as well, in the Targets file if you wish. |
|---|
| 161 | |
|---|
| 162 | Here is an example of a Probes file that would specify |
|---|
| 163 | what to use to check for HTTP and DNS latency as well as |
|---|
| 164 | the FPing probe that is used for ping latency: |
|---|
| 165 | |
|---|
| 166 | $ sudo vi Probes |
|---|
| 167 | |
|---|
| 168 | *** Probes *** |
|---|
| 169 | |
|---|
| 170 | + FPing |
|---|
| 171 | |
|---|
| 172 | binary = /usr/bin/fping |
|---|
| 173 | |
|---|
| 174 | + EchoPingHttp |
|---|
| 175 | |
|---|
| 176 | + DNS |
|---|
| 177 | binary = /usr/bin/dig |
|---|
| 178 | pings = 5 |
|---|
| 179 | step = 180 |
|---|
| 180 | lookup = www.nsrc.org |
|---|
| 181 | |
|---|
| 182 | Save the file. |
|---|
| 183 | |
|---|
| 184 | 4. Add HTTP latency checks |
|---|
| 185 | |
|---|
| 186 | Now edit your Targets again: |
|---|
| 187 | |
|---|
| 188 | $ sudo vi Targets |
|---|
| 189 | |
|---|
| 190 | Add a check for HTTP latency for all the classroom PCs. |
|---|
| 191 | This will mean adding another category, such as: |
|---|
| 192 | |
|---|
| 193 | ++HTTP Servers |
|---|
| 194 | probe = EchoPingHttp |
|---|
| 195 | menu = HTTP Response |
|---|
| 196 | title = HTTP Response Student PCs |
|---|
| 197 | |
|---|
| 198 | +++pc1 |
|---|
| 199 | menu = pc1 |
|---|
| 200 | title = PC1 HTTP Response Time |
|---|
| 201 | host = pc1 |
|---|
| 202 | |
|---|
| 203 | +++pc1 |
|---|
| 204 | host = pc2 |
|---|
| 205 | title = PC2 HTTP Response Time |
|---|
| 206 | host = pc2 |
|---|
| 207 | |
|---|
| 208 | ... |
|---|
| 209 | |
|---|
| 210 | If you have time, consider checking some machines that are |
|---|
| 211 | external to our classroom and the conference (your organization's |
|---|
| 212 | website, a popular web page, etc...) |
|---|
| 213 | |
|---|
| 214 | 5. Add DNS Latency Checks |
|---|
| 215 | |
|---|
| 216 | You can check either or both internal or external names using |
|---|
| 217 | the DNS latency probe. |
|---|
| 218 | |
|---|
| 219 | Add a menu hierarchy for DNS Latency. Check an external address |
|---|
| 220 | (nsrc.org) and an internal address (noc). This will look something |
|---|
| 221 | like this (in Targets): |
|---|
| 222 | |
|---|
| 223 | +DNS |
|---|
| 224 | probe = DNS |
|---|
| 225 | menu = External DNS Check |
|---|
| 226 | title = DNS Latency |
|---|
| 227 | |
|---|
| 228 | ++nsrc |
|---|
| 229 | host = nsrc.org |
|---|
| 230 | |
|---|
| 231 | ++noc |
|---|
| 232 | host = noc.mgmt |
|---|
| 233 | |
|---|
| 234 | Exit and save your changes to the file Targets. |
|---|
| 235 | |
|---|
| 236 | Restart Smokeping to see the changes: |
|---|
| 237 | |
|---|
| 238 | $ sudo service smokeping restart |
|---|
| 239 | |
|---|
| 240 | Look at additional Smokeping probes and consider implementing |
|---|
| 241 | some of them: |
|---|
| 242 | |
|---|
| 243 | http://oss.oetiker.ch/smokeping/probe/index.en.html |
|---|
| 244 | |
|---|
| 245 | As trying to explain all syntactical details of how the file |
|---|
| 246 | /etc/smokeping/config.d/Targets is used would require several |
|---|
| 247 | pages we will go through some examples in class, and you can |
|---|
| 248 | refer to the Smokeping configuration files that are in use on |
|---|
| 249 | the classroom NOC box by going to: |
|---|
| 250 | |
|---|
| 251 | http://noc/configs/etc/smokeping |
|---|
| 252 | http://noc/configs/etc/smokeping/config.d |
|---|
| 253 | |
|---|
| 254 | |
|---|
| 255 | 6. Send Smokeping alerts |
|---|
| 256 | |
|---|
| 257 | $ sudo vi Alerts |
|---|
| 258 | |
|---|
| 259 | Update the top of the file where it says: |
|---|
| 260 | |
|---|
| 261 | *** Alerts *** |
|---|
| 262 | to = alertee@address.somewhere |
|---|
| 263 | from = smokealert@company.xy |
|---|
| 264 | |
|---|
| 265 | to include a proper "to" and "from" field for your server. |
|---|
| 266 | Something like: |
|---|
| 267 | |
|---|
| 268 | *** Alerts *** |
|---|
| 269 | to = sysadm@localhost |
|---|
| 270 | from = smokeping-alert@localhost |
|---|
| 271 | |
|---|
| 272 | If you have installed RT, you can instead send your alerts |
|---|
| 273 | to an existing RT queue: |
|---|
| 274 | |
|---|
| 275 | *** Alerts *** |
|---|
| 276 | to = net@localhost |
|---|
| 277 | |
|---|
| 278 | At the end of the file, add another alert like this: |
|---|
| 279 | |
|---|
| 280 | +anydelay |
|---|
| 281 | type = rtt |
|---|
| 282 | # in milliseconds |
|---|
| 283 | pattern = >1 |
|---|
| 284 | comment = Just for testing |
|---|
| 285 | |
|---|
| 286 | Notice the pattern in this alert. It means that an alert will be triggered |
|---|
| 287 | as soon as a sample measurement has "ANY" delay, that is, more than one |
|---|
| 288 | millisecond. This is just for testing. In reality, you will want to create |
|---|
| 289 | an alert based on your observed baseline. For example, if your DNS servers' |
|---|
| 290 | delay suddendly goes from under 10 ms to over 100ms. |
|---|
| 291 | |
|---|
| 292 | Next, be sure you have this test alert defined for some of your Targets. |
|---|
| 293 | You can either turn on alerts by defining alerts for a probe in |
|---|
| 294 | the /etc/smokeping/config.d/Probes file, or by individual Targets |
|---|
| 295 | entries. |
|---|
| 296 | |
|---|
| 297 | In our case let's edit the Targets file and turn on alerts for our |
|---|
| 298 | DNS Latency checks. |
|---|
| 299 | |
|---|
| 300 | $ sudo vi /etc/smokeping/config.d/Targets |
|---|
| 301 | |
|---|
| 302 | Find the following section in the file: |
|---|
| 303 | |
|---|
| 304 | +DNS |
|---|
| 305 | probe = DNS |
|---|
| 306 | menu = External DNS Check |
|---|
| 307 | title = DNS Latency |
|---|
| 308 | |
|---|
| 309 | ++nsrc |
|---|
| 310 | host = nsrc.org |
|---|
| 311 | |
|---|
| 312 | And add the following alerts line after "+++ nsrc" |
|---|
| 313 | |
|---|
| 314 | +++nsrc |
|---|
| 315 | host = nsrc |
|---|
| 316 | alerts = anydelay |
|---|
| 317 | |
|---|
| 318 | Save and exit from the file, then restart smokeping: |
|---|
| 319 | |
|---|
| 320 | $ sudo service smokeping restart |
|---|
| 321 | |
|---|
| 322 | Check your e-mail with mutt |
|---|
| 323 | |
|---|
| 324 | $ mutt |
|---|
| 325 | |
|---|
| 326 | (or check your RT queues) |
|---|
| 327 | |
|---|
| 328 | And see if you have received alerts after 5 minutes. |
|---|
| 329 | |
|---|
| 330 | 6. MultiHost Graphs |
|---|
| 331 | |
|---|
| 332 | Once you have defined a group of hosts under a single probe type in your |
|---|
| 333 | /etc/smokeping/config.d/Targets file, then you can create a single graph |
|---|
| 334 | that will show you the results of all smokeping tests for all hosts that |
|---|
| 335 | you define. This has the advantage of letting you quickly compare, for |
|---|
| 336 | example, a group of hosts that you are monitoring with the FPing probe. |
|---|
| 337 | |
|---|
| 338 | The MultiHost graph function in Smokeping is extremely picky - pay close |
|---|
| 339 | attention. |
|---|
| 340 | |
|---|
| 341 | To create a MultiHost graph first edit the file Targets: |
|---|
| 342 | |
|---|
| 343 | $ sudo vi Targets |
|---|
| 344 | |
|---|
| 345 | If you had a section for the FPing probe defined that looked like this |
|---|
| 346 | (this is an example only - your Targets file may look different): |
|---|
| 347 | |
|---|
| 348 | +Local |
|---|
| 349 | menu = Local |
|---|
| 350 | title = Local Network |
|---|
| 351 | |
|---|
| 352 | ++LocalMachine |
|---|
| 353 | menu = Local Machine |
|---|
| 354 | title = This host |
|---|
| 355 | host = localhost |
|---|
| 356 | |
|---|
| 357 | ++pc1 |
|---|
| 358 | menu = pc1 |
|---|
| 359 | title = pc1 |
|---|
| 360 | host = pc1 |
|---|
| 361 | |
|---|
| 362 | ++pc2 |
|---|
| 363 | menu = pc2 |
|---|
| 364 | title = pc2 |
|---|
| 365 | host = pc2 |
|---|
| 366 | |
|---|
| 367 | ++pc3 |
|---|
| 368 | menu = pc3 |
|---|
| 369 | title = pc3 |
|---|
| 370 | host = pc3 |
|---|
| 371 | |
|---|
| 372 | Right now smokeping displays the results of the FPing probe for each |
|---|
| 373 | host defined in separate graphs. If you wish to see the results in a |
|---|
| 374 | single graph with multiple lines, then you would do this after the last |
|---|
| 375 | FPing probe host definition: |
|---|
| 376 | |
|---|
| 377 | +MultiHostPCs |
|---|
| 378 | menu = MultiHost Ping |
|---|
| 379 | title = Consolidated Ping Response Time |
|---|
| 380 | host = /Local/LocalMachine /Local/pc1 /Local/pc2 /Local/pc3 |
|---|
| 381 | |
|---|
| 382 | (Note: if the lines get too long, you can have multiple lines for the |
|---|
| 383 | "host" entry by using the "\" character to indicate another line - ask about |
|---|
| 384 | this if you are unsure!) |
|---|
| 385 | |
|---|
| 386 | Now save and exit the file Targets and restart smokeping: |
|---|
| 387 | |
|---|
| 388 | $ sudo service smokeping restart |
|---|
| 389 | |
|---|
| 390 | You should see a new graph under the "MultiHost Ping" menu in your |
|---|
| 391 | smokeping web interface. This graph will have different color lines |
|---|
| 392 | for each host you have defined. |
|---|
| 393 | |
|---|
| 394 | |
|---|
| 395 | 7. Slave instances - only done if we have the time. |
|---|
| 396 | |
|---|
| 397 | This is a description only for informational purposes in case you wish |
|---|
| 398 | to attempt this type of configuration once the workshop is over. |
|---|
| 399 | |
|---|
| 400 | The idea behind this is that you can run multiple smokeping instances |
|---|
| 401 | at multiple locations that are monitoring the same hosts and/or services |
|---|
| 402 | as your master instance. The slaves will send their results to the |
|---|
| 403 | master server and you will see these results side-by-side with your |
|---|
| 404 | local results. This allows you to view how users outside your network |
|---|
| 405 | see your services and hosts. |
|---|
| 406 | |
|---|
| 407 | This can be a powerful tool for resolving service and host issues that |
|---|
| 408 | may be difficult to troubleshoot if you only have local data. |
|---|
| 409 | |
|---|
| 410 | Graphically this looks this: |
|---|
| 411 | |
|---|
| 412 | [slave 1] [slave 2] [slave 3] |
|---|
| 413 | | | | |
|---|
| 414 | +-------+ | +--------+ |
|---|
| 415 | | | | |
|---|
| 416 | v v v |
|---|
| 417 | +---------------+ |
|---|
| 418 | | master | |
|---|
| 419 | +---------------+ |
|---|
| 420 | |
|---|
| 421 | You can see example of this data here: |
|---|
| 422 | |
|---|
| 423 | http://oss.oetiker.ch/smokeping-demo/ |
|---|
| 424 | |
|---|
| 425 | Look at the various graph groups and notice that many of the graphs |
|---|
| 426 | have multiple lines with the color code chart listing items such as |
|---|
| 427 | "median RTT from mipsrv01" - These are not MultiHost graphs, but rather |
|---|
| 428 | graphs with data from external smokeping servers. |
|---|
| 429 | |
|---|
| 430 | To configure a smokeping master/slave server you can see the documentation |
|---|
| 431 | here: |
|---|
| 432 | |
|---|
| 433 | http://oss.oetiker.ch/smokeping/doc/smokeping_master_slave.en.html |
|---|
| 434 | |
|---|
| 435 | In addition, a sample set of steps for configuring this is available in |
|---|
| 436 | the file sample-smokeping-master-slave.txt which is available under the |
|---|
| 437 | attachments listing on the agenda page of the class wiki. |
|---|
| 438 | </pre> |
|---|
| 439 | </body> |
|---|
| 440 | </html> |
|---|