| 1 | 1. Creating the master host |
|---|
| 2 | |
|---|
| 3 | The FreeBSD master host can be installed natively on a server, or it can |
|---|
| 4 | run as a VM under KVM / VirtualBox. |
|---|
| 5 | |
|---|
| 6 | That part is not documented here, it's fairly trivial to set up a new VM |
|---|
| 7 | using virt-manager under KVM, or VirtualBox's GUI. |
|---|
| 8 | |
|---|
| 9 | Basically, the FreeBSD instance should have at least 6 GB of RAM, and |
|---|
| 10 | maybe 30 GB of disk. |
|---|
| 11 | |
|---|
| 12 | If using KVM, make sure you: |
|---|
| 13 | |
|---|
| 14 | - are member of the libvirtd group |
|---|
| 15 | |
|---|
| 16 | - set the emulated NIC to e1000 (may still be issues with virtio net) |
|---|
| 17 | |
|---|
| 18 | - set disk cache to "none" using virsh edit (or use virt-manager's GUI |
|---|
| 19 | to do this under disk, advanced, performance) |
|---|
| 20 | |
|---|
| 21 | - set disk bus to Virtio |
|---|
| 22 | |
|---|
| 23 | - instead of running virt-manager remotely over SSH/X, consider using |
|---|
| 24 | vncserver (apt-get install vnc4server) and connecting to that remotely |
|---|
| 25 | to access the virt-manager GUI. |
|---|
| 26 | |
|---|
| 27 | If using VirtualBox 4+: |
|---|
| 28 | |
|---|
| 29 | # VBoxManage setextradata "My VM NAME" "VBoxInternal/Devices/e1000/0/LUN#0/Config/IfPolicyPromisc" "allow-all" |
|---|
| 30 | |
|---|
| 31 | (otherwise promisc. bridging won't work from within the host) |
|---|
| 32 | |
|---|
| 33 | |
|---|
| 34 | 2. Install FreeBSD using the ZFS root pool. |
|---|
| 35 | |
|---|
| 36 | After you are done, you should be able to reboot into the host as root. |
|---|
| 37 | |
|---|
| 38 | What still needs to be done: |
|---|
| 39 | |
|---|
| 40 | - Create the zfs jails partition |
|---|
| 41 | |
|---|
| 42 | # zfs create zroot/jails |
|---|
| 43 | # zfs set mountpoint=/jails zroot/jails |
|---|
| 44 | |
|---|
| 45 | - Set a few options in /etc/rc.conf: |
|---|
| 46 | |
|---|
| 47 | sshd_enable="YES" |
|---|
| 48 | hostname="dnshost.dns.nsrc.org" |
|---|
| 49 | ifconfig_em0="inet 10.XX.0.248/16" |
|---|
| 50 | defaultrouter="10.XX.0.254" |
|---|
| 51 | |
|---|
| 52 | (XX = whatever prefix you picked) |
|---|
| 53 | |
|---|
| 54 | - Create /etc/resolv.conf |
|---|
| 55 | |
|---|
| 56 | nameserver X.X.X.X |
|---|
| 57 | |
|---|
| 58 | - Add an nsrc user, if you haven't done so during the installation: |
|---|
| 59 | |
|---|
| 60 | # pw user add nsrc -c "NSRC user" -d /home/nsrc -m -G wheel |
|---|
| 61 | # passwd nsrc |
|---|
| 62 | |
|---|
| 63 | - You may want to allow root login to the host, by editing |
|---|
| 64 | `/etc/ssh/sshd_config` |
|---|
| 65 | |
|---|
| 66 | PermitRootLogin yes |
|---|
| 67 | |
|---|
| 68 | 3. Jails & VIMAGE |
|---|
| 69 | |
|---|
| 70 | The virtual setup is built around FreeBSD "next generation" jails and |
|---|
| 71 | the VIMAGE (virtualized network stack) framework. This allows us to |
|---|
| 72 | have machines with their own network stacks, while still being very |
|---|
| 73 | low overhead in terms of resources (each jail is a "super chroot"). |
|---|
| 74 | |
|---|
| 75 | Each jail is connected to the network using NG_BRIDGE, which is in turn |
|---|
| 76 | connected to whatever the "real" network device is (remember, the whole thing |
|---|
| 77 | could be running inside a VM). |
|---|
| 78 | |
|---|
| 79 | VNET/VIMAGE requires a kernel recompile: |
|---|
| 80 | |
|---|
| 81 | # cd /usr/src |
|---|
| 82 | # fetch ftp://ftp.freebsd.org/pub/FreeBSD/releases/amd64/amd64/9.3-RELEASE/src.txz |
|---|
| 83 | # tar -C / -xpf src.txz |
|---|
| 84 | |
|---|
| 85 | # cd /sys/amd64/conf |
|---|
| 86 | # cp GENERIC VIMAGE |
|---|
| 87 | # vi VIMAGE |
|---|
| 88 | (add "options VIMAGE") |
|---|
| 89 | # cd /usr/src |
|---|
| 90 | # make -j2 buildkernel KERNCONF=VIMAGE |
|---|
| 91 | (this will take a while) |
|---|
| 92 | # make installkernel KERNCONF=VIMAGE |
|---|
| 93 | |
|---|
| 94 | # cat >>/boot/loader.conf <<EOF |
|---|
| 95 | ng_bridge_load="YES" |
|---|
| 96 | ng_eiface_load="YES" |
|---|
| 97 | ng_ether_load="YES" |
|---|
| 98 | ng_socket_load="YES" |
|---|
| 99 | EOF |
|---|
| 100 | |
|---|
| 101 | The section below will allow participants to run tcpdump on their |
|---|
| 102 | ethernet device within their own jail, as well as access any USB |
|---|
| 103 | devices that are attached to the main host (this is mostly useful |
|---|
| 104 | when doing signing using a smartcard/token). |
|---|
| 105 | |
|---|
| 106 | (XXX The next step shouldn't be necessary - investigate) |
|---|
| 107 | |
|---|
| 108 | # cp /etc/defaults/devfs.rules /etc/ |
|---|
| 109 | |
|---|
| 110 | # cat >>/etc/devfs.rules <<EOF |
|---|
| 111 | [devfsrules_jail_bpf=5] |
|---|
| 112 | add include $devfsrules_jail |
|---|
| 113 | add path 'bpf*' unhide |
|---|
| 114 | add path 'ugen0.*' unhide |
|---|
| 115 | EOF |
|---|
| 116 | |
|---|
| 117 | Additionnally, it is recommended to set kern.maxfiles to at least 32768: |
|---|
| 118 | |
|---|
| 119 | # echo kern.maxfiles=32768 >>/etc/sysctl.conf |
|---|
| 120 | |
|---|
| 121 | XXX jail_example_parameters="allow.sysvipc=1" doesn't work with the vimage |
|---|
| 122 | start script. Workaround: jail -m jid=$JID allow.sysvipc=1 for each jail |
|---|
| 123 | in a post start script. for each jail |
|---|
| 124 | in a post start script. |
|---|
| 125 | |
|---|
| 126 | 4. Bridge setup |
|---|
| 127 | |
|---|
| 128 | To be able to support the many instances we will be running, the default |
|---|
| 129 | number of bridge links needs to be changed. Unfortunately, this requires |
|---|
| 130 | a kernel recompile at this state. |
|---|
| 131 | |
|---|
| 132 | |
|---|
| 133 | Change /sys/netgraph/ng_bridge.h, from: |
|---|
| 134 | |
|---|
| 135 | /* Maximum number of supported links */ |
|---|
| 136 | #define NG_BRIDGE_MAX_LINKS 32 |
|---|
| 137 | |
|---|
| 138 | to |
|---|
| 139 | |
|---|
| 140 | /* Maximum number of supported links */ |
|---|
| 141 | #define NG_BRIDGE_MAX_LINKS 128 |
|---|
| 142 | |
|---|
| 143 | Recompile the module if necessary: |
|---|
| 144 | |
|---|
| 145 | # cd /sys/modules/netgraph/bridge |
|---|
| 146 | # make clean; make; make install |
|---|
| 147 | |
|---|
| 148 | Reboot to test that everything loads properly |
|---|
| 149 | |
|---|
| 150 | # reboot |
|---|
| 151 | |
|---|
| 152 | 5. Jail management |
|---|
| 153 | |
|---|
| 154 | To make management of the jails easier, we use: |
|---|
| 155 | |
|---|
| 156 | http://druidbsd.sourceforge.net/vimage.shtml |
|---|
| 157 | |
|---|
| 158 | ... which implements jail-like controls to VNET instances. |
|---|
| 159 | |
|---|
| 160 | # pkg_add http://druidbsd.sourceforge.net/download/vimage-1.4.tbz |
|---|
| 161 | |
|---|
| 162 | If this doesn't work: |
|---|
| 163 | |
|---|
| 164 | # cd /tmp |
|---|
| 165 | # fetch http://druidbsd.sourceforge.net/download/vimage-1.4.tbz |
|---|
| 166 | # tar jxf vimage-1.4.tbz |
|---|
| 167 | # cp -p rc.d/vimage /etc/rc.d/ |
|---|
| 168 | |
|---|
| 169 | 6. Packages |
|---|
| 170 | |
|---|
| 171 | We're going to use mostly binary packages, but we may need to build |
|---|
| 172 | a few as well, so we'll need the ports collection: |
|---|
| 173 | |
|---|
| 174 | # portsnap fetch (takes a while) |
|---|
| 175 | # portsnap extract (takes a while) |
|---|
| 176 | |
|---|
| 177 | Let's get the pkg environment up and running: |
|---|
| 178 | |
|---|
| 179 | # pkg (only necessary on FreeBSD 9.x) |
|---|
| 180 | # echo 'WITH_PKGNG=yes' >>/etc/make.conf |
|---|
| 181 | # pkg2ng (XXX shouldn't be necessary with FreeBSD 10) |
|---|
| 182 | |
|---|
| 183 | We'll install a couple of smart package management tools |
|---|
| 184 | |
|---|
| 185 | # cd /usr/ports/ports-mgmt/portmaster/ && make all install clean |
|---|
| 186 | # portmaster -g shells/bash |
|---|
| 187 | |
|---|
| 188 | Now, make sure you have all the packages listed in packages.txt built/available. |
|---|
| 189 | |
|---|
| 190 | * Using the new (pkg) system which we setup earlier: |
|---|
| 191 | |
|---|
| 192 | # portmaster -g category/package |
|---|
| 193 | |
|---|
| 194 | Or, pkg install the packages from binary, then once done: |
|---|
| 195 | |
|---|
| 196 | # cd /usr/ports/packages/All && pkg create -a |
|---|
| 197 | |
|---|
| 198 | Some packages will need to be custom-built, such as |
|---|
| 199 | |
|---|
| 200 | * PHP (module support - but we could run with CGI in principle) |
|---|
| 201 | * opendnssec (softhsm must be enabled) |
|---|
| 202 | * bind910 (-DWITH_REPLACE_BASE) |
|---|
| 203 | |
|---|
| 204 | ... see the package list for more info |
|---|
| 205 | |
|---|
| 206 | For these, extract the ports collection (portsnap fetch && portsnap extract |
|---|
| 207 | as explained above), then: |
|---|
| 208 | |
|---|
| 209 | # mkdir -p /usr/ports/packages/All |
|---|
| 210 | |
|---|
| 211 | Then for each package: |
|---|
| 212 | |
|---|
| 213 | # portmaster -g <category>/<pkg_name> |
|---|
| 214 | |
|---|
| 215 | This will create a package of each port + the dependencies in |
|---|
| 216 | /usr/ports/packages/All/* |
|---|
| 217 | |
|---|
| 218 | Note: add this to /etc/make.conf: |
|---|
| 219 | |
|---|
| 220 | OPTIONS_SET=REPLACE_BASE BATCH SIGCHASE |
|---|
| 221 | |
|---|
| 222 | |
|---|
| 223 | 7. Prep master jail |
|---|
| 224 | |
|---|
| 225 | - Create ZFS container |
|---|
| 226 | |
|---|
| 227 | # zfs create zroot/jails/master |
|---|
| 228 | |
|---|
| 229 | - Create a master host distrib - you can do this with a stock FreeBSD |
|---|
| 230 | install CDROM mounted. |
|---|
| 231 | |
|---|
| 232 | # export BSDINSTALL_DISTSITE=file:///mnt/usr/freebsd-dists/ |
|---|
| 233 | # bsdinstall jail /jails/master |
|---|
| 234 | |
|---|
| 235 | NOTE: if you have good Internet connectivity at this point, you |
|---|
| 236 | can leave out "export BSDINSTALL_DISTSITE" and the installer will |
|---|
| 237 | fetch the distribution from the network, in which case you will be |
|---|
| 238 | asked to select the mirror you want to install from. |
|---|
| 239 | |
|---|
| 240 | Once the installer has run: |
|---|
| 241 | |
|---|
| 242 | - The root password is unimportant, we'll override it later. |
|---|
| 243 | - Accept all defaults |
|---|
| 244 | - Choose No when asked about enabling crash dumps, or adding users now. |
|---|
| 245 | |
|---|
| 246 | - Create package dir in master |
|---|
| 247 | |
|---|
| 248 | # mkdir /jails/master/packages |
|---|
| 249 | # mount -t devfs devfs /jails/master/dev |
|---|
| 250 | # mkdir -p /usr/ports/packages/All |
|---|
| 251 | # mount_nullfs /usr/ports/packages/All /jails/master/packages |
|---|
| 252 | |
|---|
| 253 | - Setup the pkg environment in the jail: |
|---|
| 254 | |
|---|
| 255 | NOTE: packages are preinstalled in this way; they can also be |
|---|
| 256 | installed by the participants, but there's litlle value in that, |
|---|
| 257 | as they will still have to configure them anyway. |
|---|
| 258 | |
|---|
| 259 | # jail -c path=/jails/master mount.devfs host.hostname=master ip4.addr=X.X.0.248 command=/bin/tcsh |
|---|
| 260 | |
|---|
| 261 | (replace X.X with the prefix, for example X.X if you are using X.X/16) |
|---|
| 262 | |
|---|
| 263 | Here the prompt should say: |
|---|
| 264 | |
|---|
| 265 | root@master:/ # |
|---|
| 266 | |
|---|
| 267 | Define a DNS: |
|---|
| 268 | |
|---|
| 269 | # echo 'nameserver X.X.0.254' >/etc/resolv.conf |
|---|
| 270 | |
|---|
| 271 | Now setup the pkg env: |
|---|
| 272 | |
|---|
| 273 | # pkg |
|---|
| 274 | (answer y) |
|---|
| 275 | |
|---|
| 276 | # rm /etc/resolv.conf |
|---|
| 277 | # exit |
|---|
| 278 | |
|---|
| 279 | - Re start jail up and install the packages |
|---|
| 280 | |
|---|
| 281 | (Note: could also do this in the previous step - room for improvement here) |
|---|
| 282 | |
|---|
| 283 | NOTE: packages are preinstalled in this way; they can also be |
|---|
| 284 | installed by the participants, but there's litlle value in that, |
|---|
| 285 | as they will still have to configure them anyway. |
|---|
| 286 | |
|---|
| 287 | # id=`jail -i -c vnet name=master host.hostname=master path=/jails/master persist` |
|---|
| 288 | # jexec $id /bin/tcsh |
|---|
| 289 | |
|---|
| 290 | # echo 'WITH_PKGNG=yes' >>/etc/make.conf |
|---|
| 291 | |
|---|
| 292 | Install the packages |
|---|
| 293 | |
|---|
| 294 | # cd /packages |
|---|
| 295 | # pkg add ./* |
|---|
| 296 | (if this causes issues, add the packages one at a time) |
|---|
| 297 | |
|---|
| 298 | If you get asked if you want to activate Postfix, says y |
|---|
| 299 | |
|---|
| 300 | 8. Exit & terminate the jail |
|---|
| 301 | |
|---|
| 302 | # exit |
|---|
| 303 | # jail -r $id |
|---|
| 304 | |
|---|
| 305 | 9. Snapshot the system |
|---|
| 306 | |
|---|
| 307 | zfs snapshot zroot/jails/master@initial |
|---|
| 308 | |
|---|
| 309 | Deploying the hosts |
|---|
| 310 | -------------------- |
|---|
| 311 | |
|---|
| 312 | First, decide on the IP prefix (/16) you will want to use for this workshop |
|---|
| 313 | instance. |
|---|
| 314 | |
|---|
| 315 | 1. You may want to add your keys to |
|---|
| 316 | setup/configs/common/root/.ssh/authorized_keys |
|---|
| 317 | |
|---|
| 318 | 2. Adjust setup-*/build/hostlist.tmpl to match the hosts you'll be needing. |
|---|
| 319 | |
|---|
| 320 | 3. Cloning hosts and distribute the configuration files |
|---|
| 321 | |
|---|
| 322 | Inspect 1-setup-zfs.sh, and make sure the src= parameter |
|---|
| 323 | is set to the zfs clone of the master (verify with zfs list -t snapshot) |
|---|
| 324 | and the dst= parameter is set as well. |
|---|
| 325 | |
|---|
| 326 | Inspect 2-setup-basic.sh, and check the parameters at the top for: |
|---|
| 327 | |
|---|
| 328 | - workshop username (sysadm) |
|---|
| 329 | - password (nsrc+ws) |
|---|
| 330 | - timezone (Region/Town) |
|---|
| 331 | |
|---|
| 332 | Now run the scripts (as root) |
|---|
| 333 | |
|---|
| 334 | # ./0-replace-prefix.sh 10.10 # default is 10.10, or pick another |
|---|
| 335 | # prefix |
|---|
| 336 | # ./1-setup-zfs.sh |
|---|
| 337 | # ./2-setup-basic.sh pushconf # this pushes the predefined files |
|---|
| 338 | # found under setup/configs |
|---|
| 339 | # ./2-setup-basic.sh localchanges # this carries out localchanges/tweaks |
|---|
| 340 | # ./3-fixes.sh # HACK - this needs to be merged into |
|---|
| 341 | # one of the other scripts |
|---|
| 342 | |
|---|
| 343 | 3. Create vnet jail definitions in /etc/rc.conf (adjust nhosts) |
|---|
| 344 | Make sure the interface name (if=) is set to the proper interface |
|---|
| 345 | (em0, vtnet0, etc...) in the script before you run it: |
|---|
| 346 | |
|---|
| 347 | # ./4-setup-vnet-jails-defs.sh |
|---|
| 348 | |
|---|
| 349 | 4. Start jails |
|---|
| 350 | |
|---|
| 351 | # service vimage onestart |
|---|
| 352 | |
|---|
| 353 | 5. Enable shared memory on every host (hack) |
|---|
| 354 | |
|---|
| 355 | for jail in `jls | awk '{ print $1 }' | grep -v JID` |
|---|
| 356 | do |
|---|
| 357 | jail -m jid=$jail allow.sysvipc=1 |
|---|
| 358 | done |
|---|
| 359 | |
|---|
| 360 | 6. Test ssh to a host |
|---|
| 361 | |
|---|
| 362 | ssh root@... |
|---|
| 363 | ssh |
|---|
| 364 | |
|---|
| 365 | Setting up the class-wide resolver, auth servers and root servers |
|---|
| 366 | ------------------------------------------------------------------ |
|---|
| 367 | |
|---|
| 368 | Configuration are pre-distributed via the config/ subdir, but the steps |
|---|
| 369 | below may be required prior to starting the services. |
|---|
| 370 | |
|---|
| 371 | 1. Setting up the resolver (X.X.0.230) |
|---|
| 372 | |
|---|
| 373 | # unbound-control-setup |
|---|
| 374 | # service unbound start |
|---|
| 375 | |
|---|
| 376 | Note: that at this point the server will not be doing validation until |
|---|
| 377 | a trust anchor has been configured - the root needs to be signed for |
|---|
| 378 | this. |
|---|
| 379 | |
|---|
| 380 | 2. Setting up the auth servers |
|---|
| 381 | |
|---|
| 382 | By default: |
|---|
| 383 | |
|---|
| 384 | auth1 is running NSD and is the master for "dns.nsrc.org" and X.X.arpa |
|---|
| 385 | auth2 is running BIND and is the slave for "dns.nsrc.org" and X.X.arpa |
|---|
| 386 | |
|---|
| 387 | Note: maybe this should be the other way around ? |
|---|
| 388 | |
|---|
| 389 | When the jails are started, everything should work |
|---|
| 390 | (no intervention needed) |
|---|
| 391 | |
|---|
| 392 | 3. Setting up the root servers |
|---|
| 393 | |
|---|
| 394 | a.root is running BIND and is the master for "." |
|---|
| 395 | b.root is running NSD and is the master for "." |
|---|
| 396 | |
|---|
| 397 | a.root should just start loading the zone on boot |
|---|
| 398 | b.root as well, but you can force a dump of the zone to a file by issuing |
|---|
| 399 | |
|---|
| 400 | # nsdc patch |
|---|
| 401 | |
|---|
| 402 | Note: a key "root_zone_xfer_key" is defined on nsd.conf for b.root, |
|---|
| 403 | but not used at the moment. There is no reason to restrict zone transfer |
|---|
| 404 | of the root. |
|---|
| 405 | |
|---|
| 406 | DHCP setup and resolution path |
|---|
| 407 | ------------------------------ |
|---|
| 408 | |
|---|
| 409 | DHCP should give out 254 as the nameserver - or whatever device is |
|---|
| 410 | providing DNS for the "real" root. |
|---|
| 411 | |
|---|
| 412 | The DNS handed out by the DHCP should be configured to forward requests |
|---|
| 413 | for dns.nsrc.org to the two auth servers. If using BIND, this would be: |
|---|
| 414 | |
|---|
| 415 | zone "dns.nsrc.org" { |
|---|
| 416 | type forward; |
|---|
| 417 | forwarders { X.X.0.230; }; |
|---|
| 418 | }; |
|---|
| 419 | |
|---|
| 420 | zone "X.X.in-addr.arpa" { |
|---|
| 421 | type forward; |
|---|
| 422 | forwarders { X.X.0.230; }; |
|---|
| 423 | }; |
|---|
| 424 | |
|---|
| 425 | In this way: |
|---|
| 426 | |
|---|
| 427 | - participant laptops get DHCP and use .254 (or whichever device can resolve |
|---|
| 428 | the "real" root), but can still resolve dns.nsrc.org using the forward |
|---|
| 429 | (or equivalent) statement above |
|---|
| 430 | |
|---|
| 431 | - class VMs are using .230 by default which means they can't see the "real" |
|---|
| 432 | root |
|---|
| 433 | |
|---|
| 434 | TIPS AND TRICKS |
|---|
| 435 | --------------- |
|---|
| 436 | |
|---|
| 437 | To quickly flush the cache in unbound: |
|---|
| 438 | |
|---|
| 439 | # unbound-control flush_zone . |
|---|
| 440 | |
|---|
| 441 | (flushes all RRs at and below .) |
|---|
| 442 | |
|---|
| 443 | NOTES: |
|---|
| 444 | |
|---|
| 445 | - periodic scripts are disabled by the crontab included under |
|---|
| 446 | configs/common/etc. |
|---|
| 447 | |
|---|
| 448 | This is to avoid killing the system with I/O when the periodic |
|---|
| 449 | maintenance scripts are run. |
|---|
| 450 | |
|---|
| 451 | - you may want to enable bsnmpd on the VMs. To do this, modify /etc/snmpd.conf |
|---|
| 452 | as follows: |
|---|
| 453 | |
|---|
| 454 | --- snmpd.config.orig 2013-08-12 07:32:38.255328350 +0000 |
|---|
| 455 | +++ snmpd.config 2013-08-12 07:36:35.125447049 +0000 |
|---|
| 456 | -location := "Room 200" |
|---|
| 457 | -contact := "sysmeister@example.com" |
|---|
| 458 | +location := "NSRC Workshop" |
|---|
| 459 | +contact := "me@example.com" |
|---|
| 460 | -read := "public" |
|---|
| 461 | +read := "NetManage" |
|---|
| 462 | -write := "geheim" |
|---|
| 463 | -trap := "mytrap" |
|---|
| 464 | +# write := "geheim" |
|---|
| 465 | +trap := "NetManage" |
|---|
| 466 | |
|---|
| 467 | ... enable bsnmpd in /etc/rc.conf |
|---|
| 468 | |
|---|
| 469 | bsnmpd_enable="YES" |
|---|
| 470 | |
|---|
| 471 | Start snmpd: |
|---|
| 472 | |
|---|
| 473 | $ sudo service snmpd start |
|---|