| 1 | Network Management & Monitoring |
|---|
| 2 | |
|---|
| 3 | Smokeping |
|---|
| 4 | |
|---|
| 5 | Notes: |
|---|
| 6 | ------ |
|---|
| 7 | * Commands preceded with "$" imply that you should execute the command as |
|---|
| 8 | a general user - not as root. |
|---|
| 9 | * Commands preceded with "#" imply that you should be working as root. |
|---|
| 10 | * Commands with more specific command lines (e.g. "RTR-GW>" or "mysql>") |
|---|
| 11 | imply that you are executing commands on remote equipment, or within |
|---|
| 12 | another program. |
|---|
| 13 | |
|---|
| 14 | Exercises |
|---|
| 15 | ---------- |
|---|
| 16 | |
|---|
| 17 | 0. Log in to your PC or open a terminal window as the sysadmin user. |
|---|
| 18 | |
|---|
| 19 | Once you are logged in you can continue with these exercises. |
|---|
| 20 | |
|---|
| 21 | 1. Install Smokeping |
|---|
| 22 | |
|---|
| 23 | $ sudo apt-get install smokeping |
|---|
| 24 | |
|---|
| 25 | 2. Initial Configuration |
|---|
| 26 | |
|---|
| 27 | $ cd /etc/smokeping/config.d |
|---|
| 28 | $ ls -l |
|---|
| 29 | |
|---|
| 30 | -rwxr-xr-x 1 root root 578 2010-02-26 01:55 Alerts |
|---|
| 31 | -rwxr-xr-x 1 root root 237 2010-02-26 01:55 Database |
|---|
| 32 | -rwxr-xr-x 1 root root 413 2010-02-26 05:40 General |
|---|
| 33 | -rwxr-xr-x 1 root root 271 2010-02-26 01:55 pathnames |
|---|
| 34 | -rwxr-xr-x 1 root root 859 2010-02-26 01:55 Presentation |
|---|
| 35 | -rwxr-xr-x 1 root root 116 2010-02-26 01:55 Probes |
|---|
| 36 | -rwxr-xr-x 1 root root 155 2010-02-26 01:55 Slaves |
|---|
| 37 | -rwxr-xr-x 1 root root 8990 2010-02-26 06:30 Targets |
|---|
| 38 | |
|---|
| 39 | $ sudo vi General |
|---|
| 40 | |
|---|
| 41 | Change the following lines: |
|---|
| 42 | |
|---|
| 43 | owner = NOC |
|---|
| 44 | contact = sysadmin@localhost |
|---|
| 45 | cgiurl = http://localhost/cgi-bin/smokeping.cgi |
|---|
| 46 | mailhost = localhost |
|---|
| 47 | |
|---|
| 48 | Save the file and exit. Now let's restart the |
|---|
| 49 | Smokeping service to verify that no mistakes have been made |
|---|
| 50 | before going any further: |
|---|
| 51 | |
|---|
| 52 | $ sudo /etc/init.d/smokeping stop |
|---|
| 53 | $ sudo /etc/init.d/smokeping start |
|---|
| 54 | |
|---|
| 55 | 2. Configure monitoring of devices |
|---|
| 56 | |
|---|
| 57 | The majority of your time and work configuring Smokeping |
|---|
| 58 | will be done in the file /etc/smokeping/config.d/Targets. |
|---|
| 59 | |
|---|
| 60 | For this class please do the following: |
|---|
| 61 | |
|---|
| 62 | Use the default FPing probe to check: |
|---|
| 63 | |
|---|
| 64 | - all the student NOC PCs |
|---|
| 65 | - classroom NOC |
|---|
| 66 | - switches |
|---|
| 67 | - routers |
|---|
| 68 | |
|---|
| 69 | You can use the classroom Network Diagram on the classroom wiki to |
|---|
| 70 | figure out addresses for each item, etc. |
|---|
| 71 | |
|---|
| 72 | Create some hierarchy to the Smokeping menu for your |
|---|
| 73 | checks. Such as: |
|---|
| 74 | |
|---|
| 75 | + PCs |
|---|
| 76 | |
|---|
| 77 | ++ pc1 |
|---|
| 78 | menu = pc1 |
|---|
| 79 | title = pc1 |
|---|
| 80 | host = pc1 |
|---|
| 81 | |
|---|
| 82 | ++ pc2 |
|---|
| 83 | menu = pc2 |
|---|
| 84 | title = pc2 |
|---|
| 85 | host = pc2 |
|---|
| 86 | |
|---|
| 87 | Save the file and restart Smokeping: |
|---|
| 88 | |
|---|
| 89 | $ sudo /etc/init.d/smokeping stop |
|---|
| 90 | $ sudo /etc/init.d/smokeping start |
|---|
| 91 | |
|---|
| 92 | Go to your browser and check the Smokeping page: |
|---|
| 93 | |
|---|
| 94 | http://10.10.x.y/cgi-bin/smokeping.cgi |
|---|
| 95 | |
|---|
| 96 | If everything is looking OK, continue adding: |
|---|
| 97 | |
|---|
| 98 | + Routers |
|---|
| 99 | |
|---|
| 100 | ++ bb-gw |
|---|
| 101 | menu = bb-gw |
|---|
| 102 | title = bb-gw |
|---|
| 103 | menu = bb-gw |
|---|
| 104 | |
|---|
| 105 | ++ r1 |
|---|
| 106 | menu = r1 |
|---|
| 107 | title = r1 |
|---|
| 108 | host = r1 |
|---|
| 109 | |
|---|
| 110 | + Switches |
|---|
| 111 | |
|---|
| 112 | ++ bb-sw |
|---|
| 113 | menu = bb-sw |
|---|
| 114 | title = bb-sw |
|---|
| 115 | menu = bb-sw |
|---|
| 116 | |
|---|
| 117 | ... |
|---|
| 118 | |
|---|
| 119 | Save the file, restart smokeping, and check |
|---|
| 120 | your browser again. |
|---|
| 121 | |
|---|
| 122 | 3. Add new probes |
|---|
| 123 | |
|---|
| 124 | The current entry in Probes is fine, but if you wish to |
|---|
| 125 | use additional Smokeping checks you can add them in here |
|---|
| 126 | and you can specify their default behavior. You can do |
|---|
| 127 | this, as well, in the Targets file if you wish. |
|---|
| 128 | |
|---|
| 129 | Here is an example of a Probes file that would specify |
|---|
| 130 | what to use to check for HTTP and DNS latency as well as |
|---|
| 131 | the FPing probe that is used for ping latency: |
|---|
| 132 | |
|---|
| 133 | $ sudo vi Probes |
|---|
| 134 | |
|---|
| 135 | *** Probes *** |
|---|
| 136 | |
|---|
| 137 | + FPing |
|---|
| 138 | |
|---|
| 139 | binary = /usr/bin/fping |
|---|
| 140 | |
|---|
| 141 | + EchoPingHttp |
|---|
| 142 | |
|---|
| 143 | + DNS |
|---|
| 144 | binary = /usr/bin/dig |
|---|
| 145 | pings = 5 |
|---|
| 146 | step = 180 |
|---|
| 147 | lookup = www.nsrc.org |
|---|
| 148 | |
|---|
| 149 | Save the file. |
|---|
| 150 | |
|---|
| 151 | 4. Add HTTP latency checks |
|---|
| 152 | |
|---|
| 153 | Now edit your Targets again: |
|---|
| 154 | |
|---|
| 155 | $ sudo vi Targets |
|---|
| 156 | |
|---|
| 157 | Add a check for HTTP latency for all the classroom PCs. |
|---|
| 158 | This will mean adding another category, such as: |
|---|
| 159 | |
|---|
| 160 | + HTTP Servers |
|---|
| 161 | probe = EchoPingHttp |
|---|
| 162 | |
|---|
| 163 | ++ PC1 |
|---|
| 164 | host = pc1 |
|---|
| 165 | |
|---|
| 166 | ++ PC2 |
|---|
| 167 | host = pc2 |
|---|
| 168 | |
|---|
| 169 | ... |
|---|
| 170 | |
|---|
| 171 | If you have time, consider checking some machines that are |
|---|
| 172 | external to our classroom and the conference (your organization's |
|---|
| 173 | website, a popular web page, etc...) |
|---|
| 174 | |
|---|
| 175 | 5. Add DNS Latency Checks |
|---|
| 176 | |
|---|
| 177 | You can check either or both internal or external names using |
|---|
| 178 | the DNS latency probe. |
|---|
| 179 | |
|---|
| 180 | Add a menu hierarchy for DNS Latency. Check an external address |
|---|
| 181 | (nsrc.org) and an internal address (noc). This will look something |
|---|
| 182 | like this (in Targets): |
|---|
| 183 | |
|---|
| 184 | + DNS |
|---|
| 185 | probe = DNS |
|---|
| 186 | menu = External DNS Check |
|---|
| 187 | title = DNS Latency |
|---|
| 188 | |
|---|
| 189 | ++ nsrc |
|---|
| 190 | host = nsrc.org |
|---|
| 191 | |
|---|
| 192 | ++ noc |
|---|
| 193 | host = noc.mgmt |
|---|
| 194 | |
|---|
| 195 | Exit and save your changes to the file Targets. |
|---|
| 196 | |
|---|
| 197 | Restart Smokeping to see the changes: |
|---|
| 198 | |
|---|
| 199 | $ sudo /etc/init.d/smokeping stop |
|---|
| 200 | $ sudo /etc/init.d/smokeping start |
|---|
| 201 | |
|---|
| 202 | Look at additional Smokeping probes and consider implementing |
|---|
| 203 | some of them: |
|---|
| 204 | |
|---|
| 205 | http://oss.oetiker.ch/smokeping/probe/index.en.html |
|---|
| 206 | |
|---|
| 207 | As trying to explain all syntactical details of how the file |
|---|
| 208 | /etc/smokeping/config.d/Targets is used would require several |
|---|
| 209 | pages we will go through some examples in class, and you can |
|---|
| 210 | refer to the Smokeping configuration files that are in use on |
|---|
| 211 | the classroom NOC box by going to: |
|---|
| 212 | |
|---|
| 213 | http://noc/configs/etc/smokeping |
|---|
| 214 | http://noc/configs/etc/smokeping/config.d |
|---|
| 215 | |
|---|
| 216 | |
|---|
| 217 | 6. Send Smokeping alerts |
|---|
| 218 | |
|---|
| 219 | $ sudo vi Alerts |
|---|
| 220 | |
|---|
| 221 | Update the top of the file where it says: |
|---|
| 222 | |
|---|
| 223 | *** Alerts *** |
|---|
| 224 | to = alertee@address.somewhere |
|---|
| 225 | from = smokealert@company.xy |
|---|
| 226 | |
|---|
| 227 | to include a proper "to" and "from" field for your server. |
|---|
| 228 | Something like: |
|---|
| 229 | |
|---|
| 230 | *** Alerts *** |
|---|
| 231 | to = sysadmin@localhost |
|---|
| 232 | from = smokeping-alert@localhost |
|---|
| 233 | |
|---|
| 234 | If you have installed RT, you can instead send your alerts |
|---|
| 235 | to an existing RT queue: |
|---|
| 236 | |
|---|
| 237 | *** Alerts *** |
|---|
| 238 | to = net@localhost |
|---|
| 239 | |
|---|
| 240 | At the end of the file, add another alert like this: |
|---|
| 241 | |
|---|
| 242 | +anydelay |
|---|
| 243 | type = rtt |
|---|
| 244 | # in milliseconds |
|---|
| 245 | pattern = >1 |
|---|
| 246 | comment = Just for testing |
|---|
| 247 | |
|---|
| 248 | Notice the pattern in this alert. It means that an alert will be triggered |
|---|
| 249 | as soon as a sample measurement has "ANY" delay, that is, more than one |
|---|
| 250 | millisecond. This is just for testing. In reality, you will want to create |
|---|
| 251 | an alert based on your observed baseline. For example, if your DNS servers' |
|---|
| 252 | delay suddendly goes from under 10 ms to over 100ms. |
|---|
| 253 | |
|---|
| 254 | Next, be sure you have this test alert defined for some of your Targets. |
|---|
| 255 | You can either turn on alerts by defining alerts for a probe in |
|---|
| 256 | the /etc/smokeping/config.d/Probes file, or by individual Targets |
|---|
| 257 | entries. |
|---|
| 258 | |
|---|
| 259 | In our case let's edit the Targets file and turn on alerts for our |
|---|
| 260 | DNS Latency checks. |
|---|
| 261 | |
|---|
| 262 | $ sudo vi /etc/smokeping/config.d/Targets |
|---|
| 263 | |
|---|
| 264 | Find the following section in the file: |
|---|
| 265 | |
|---|
| 266 | + DNS |
|---|
| 267 | probe = DNS |
|---|
| 268 | menu = External DNS Check |
|---|
| 269 | title = DNS Latency |
|---|
| 270 | |
|---|
| 271 | ++ nsrc |
|---|
| 272 | host = nsrc.org |
|---|
| 273 | |
|---|
| 274 | And add the following alerts line after "+++ nsrc" |
|---|
| 275 | |
|---|
| 276 | +++ nsrc |
|---|
| 277 | host = nsrc |
|---|
| 278 | alerts = anydelay |
|---|
| 279 | |
|---|
| 280 | Save and exit from the file, then restart smokeping: |
|---|
| 281 | |
|---|
| 282 | $ sudo /etc/init.d/smokeping stop |
|---|
| 283 | $ sudo /etc/init.d/smokeping start |
|---|
| 284 | |
|---|
| 285 | Check your e-mail with mutt |
|---|
| 286 | |
|---|
| 287 | $ mutt |
|---|
| 288 | |
|---|
| 289 | (or check your RT queues) |
|---|
| 290 | |
|---|
| 291 | And see if you have received alerts after 5 minutes. |
|---|
| 292 | |
|---|
| 293 | 6. MultiHost Graphs |
|---|
| 294 | |
|---|
| 295 | Once you have defined a group of hosts under a single probe type in your |
|---|
| 296 | /etc/smokeping/config.d/Targets file, then you can create a single graph |
|---|
| 297 | that will show you the results of all smokeping tests for all hosts that |
|---|
| 298 | you define. This has the advantage of letting you quickly compare, for |
|---|
| 299 | example, a group of hosts that you are monitoring with the FPing probe. |
|---|
| 300 | |
|---|
| 301 | The MultiHost graph function in Smokeping is extremely picky - pay close |
|---|
| 302 | attention. |
|---|
| 303 | |
|---|
| 304 | To create a MultiHost graph first edit the file Targets: |
|---|
| 305 | |
|---|
| 306 | $ sudo vi Targets |
|---|
| 307 | |
|---|
| 308 | If you had a section for the FPing probe defined that looked like this |
|---|
| 309 | (this is an example only - your Targets file may look different): |
|---|
| 310 | |
|---|
| 311 | + Local |
|---|
| 312 | menu = Local |
|---|
| 313 | title = Local Network |
|---|
| 314 | |
|---|
| 315 | ++ LocalMachine |
|---|
| 316 | menu = Local Machine |
|---|
| 317 | title = This host |
|---|
| 318 | host = localhost |
|---|
| 319 | |
|---|
| 320 | ++ pc1 |
|---|
| 321 | menu = pc1 |
|---|
| 322 | title = pc1 |
|---|
| 323 | host = pc1 |
|---|
| 324 | |
|---|
| 325 | ++ pc2 |
|---|
| 326 | menu = pc2 |
|---|
| 327 | title = pc2 |
|---|
| 328 | host = pc2 |
|---|
| 329 | |
|---|
| 330 | ++ pc3 |
|---|
| 331 | menu = pc3 |
|---|
| 332 | title = pc3 |
|---|
| 333 | host = pc3 |
|---|
| 334 | |
|---|
| 335 | Right now smokeping displays the results of the FPing probe for each |
|---|
| 336 | host defined in separate graphs. If you wish to see the results in a |
|---|
| 337 | single graph with multiple lines, then you would do this after the last |
|---|
| 338 | FPing probe host definition: |
|---|
| 339 | |
|---|
| 340 | + MultiHostPCs |
|---|
| 341 | menu = MultiHost Ping |
|---|
| 342 | title = Consolidated Ping Response Time |
|---|
| 343 | host = /Local/LocalMachine /Local/pc1 /Local/pc2 /Local/pc3 |
|---|
| 344 | |
|---|
| 345 | (Note: if the lines get too long, you can have multiple lines for the |
|---|
| 346 | "host" entry by using the "\" character to indicate another line - ask about |
|---|
| 347 | this if you are unsure!) |
|---|
| 348 | |
|---|
| 349 | Now save and exit the file Targets and restart smokeping: |
|---|
| 350 | |
|---|
| 351 | $ sudo /etc/init.d/smokeping stop |
|---|
| 352 | $ sudo /etc/init.d/smokeping start |
|---|
| 353 | |
|---|
| 354 | You should see a new graph under the "MultiHost Ping" menu in your |
|---|
| 355 | smokeping web interface. This graph will have different color lines |
|---|
| 356 | for each host you have defined. |
|---|
| 357 | |
|---|
| 358 | |
|---|
| 359 | 7. Slave instances - only done if we have the time. |
|---|
| 360 | |
|---|
| 361 | This is a description only for informational purposes in case you wish |
|---|
| 362 | to attempt this type of configuration once the workshop is over. |
|---|
| 363 | |
|---|
| 364 | The idea behind this is that you can run multiple smokeping instances |
|---|
| 365 | at multiple locations that are monitoring the same hosts and/or services |
|---|
| 366 | as your master instance. The slaves will send their results to the |
|---|
| 367 | master server and you will see these results side-by-side with your |
|---|
| 368 | local results. This allows you to view how users outside your network |
|---|
| 369 | see your services and hosts. |
|---|
| 370 | |
|---|
| 371 | This can be a powerful tool for resolving service and host issues that |
|---|
| 372 | may be difficult to troubleshoot if you only have local data. |
|---|
| 373 | |
|---|
| 374 | Graphically this looks this: |
|---|
| 375 | |
|---|
| 376 | [slave 1] [slave 2] [slave 3] |
|---|
| 377 | | | | |
|---|
| 378 | +-------+ | +--------+ |
|---|
| 379 | | | | |
|---|
| 380 | v v v |
|---|
| 381 | +---------------+ |
|---|
| 382 | | master | |
|---|
| 383 | +---------------+ |
|---|
| 384 | |
|---|
| 385 | You can see example of this data here: |
|---|
| 386 | |
|---|
| 387 | http://oss.oetiker.ch/smokeping-demo/ |
|---|
| 388 | |
|---|
| 389 | Look at the various graph groups and notice that many of the graphs |
|---|
| 390 | have multiple lines with the color code chart listing items such as |
|---|
| 391 | "median RTT from mipsrv01" - These are not MultiHost graphs, but rather |
|---|
| 392 | graphs with data from external smokeping servers. |
|---|
| 393 | |
|---|
| 394 | To configure a smokeping master/slave server you can see the documentation |
|---|
| 395 | here: |
|---|
| 396 | |
|---|
| 397 | http://oss.oetiker.ch/smokeping/doc/smokeping_master_slave.en.html |
|---|
| 398 | |
|---|
| 399 | In addition, a sample set of steps for configuring this is available in |
|---|
| 400 | the file sample-smokeping-master-slave.txt. |
|---|
| 401 | |
|---|