Agenda: workshop-deployment-guide.txt

1. Creating the master host

The FreeBSD master host can be installed natively on a server, or it can
run as a VM under KVM / VirtualBox.

That part is not documented here, it's fairly trivial to set up a new VM
using virt-manager under KVM, or VirtualBox's GUI.

Basically, the FreeBSD instance should have at least 6 GB of RAM, and
maybe 30 GB of disk.

If using KVM, make sure you:

- are member of the libvirtd group

- set the emulated NIC to e1000 (may still be issues with virtio net)

- set disk cache to "none" using virsh edit (or use virt-manager's GUI
  to do this under disk, advanced, performance) 

- set disk bus to Virtio

- instead of running virt-manager remotely over SSH/X, consider using
  vncserver (apt-get install vnc4server) and connecting to that remotely
  to access the virt-manager GUI.

If using VirtualBox 4+:

# VBoxManage setextradata "My VM NAME" "VBoxInternal/Devices/e1000/0/LUN#0/Config/IfPolicyPromisc" "allow-all"

(otherwise promisc. bridging won't work from within the host)


2. Install FreeBSD using the ZFS root pool.

After you are done, you should be able to reboot into the host as root.

What still needs to be done:

- Create the zfs jails partition

        # zfs create zroot/jails
        # zfs set mountpoint=/jails zroot/jails

- Set a few options in /etc/rc.conf:

        sshd_enable="YES"
        hostname="dnshost.dns.nsrc.org"
        ifconfig_em0="inet 10.XX.0.248/16"
        defaultrouter="10.XX.0.254"

  (XX = whatever prefix you picked)

- Create /etc/resolv.conf

        nameserver X.X.X.X

- Add an nsrc user, if you haven't done so during the installation:

        # pw user add nsrc -c "NSRC user" -d /home/nsrc -m -G wheel
        # passwd nsrc

- You may want to allow root login to the host, by editing
  `/etc/ssh/sshd_config`

        PermitRootLogin yes

3. Jails & VIMAGE

The virtual setup is built around FreeBSD "next generation" jails and
the VIMAGE (virtualized network stack) framework. This allows us to
have machines with their own network stacks, while still being very
low overhead in terms of resources (each jail is a "super chroot").

Each jail is connected to the network using NG_BRIDGE, which is in turn
connected to whatever the "real" network device is (remember, the whole thing
could be running inside a VM).

VNET/VIMAGE requires a kernel recompile:

# cd /usr/src 
# fetch ftp://ftp.freebsd.org/pub/FreeBSD/releases/amd64/amd64/9.3-RELEASE/src.txz
# tar -C / -xpf src.txz 

# cd /sys/amd64/conf
# cp GENERIC VIMAGE
# vi VIMAGE
        (add "options           VIMAGE")
# cd /usr/src
# make -j2 buildkernel KERNCONF=VIMAGE
  (this will take a while)
# make installkernel KERNCONF=VIMAGE

# cat >>/boot/loader.conf <<EOF
ng_bridge_load="YES"
ng_eiface_load="YES"
ng_ether_load="YES"
ng_socket_load="YES"
EOF

The section below will allow participants to run tcpdump on their
ethernet device within their own jail, as well as access any USB
devices that are attached to the main host (this is mostly useful
when doing signing using a smartcard/token).

(XXX The next step shouldn't be necessary - investigate)

# cp /etc/defaults/devfs.rules /etc/

# cat >>/etc/devfs.rules <<EOF
[devfsrules_jail_bpf=5]
add include $devfsrules_jail
add path 'bpf*' unhide
add path 'ugen0.*' unhide
EOF

Additionnally, it is recommended to set kern.maxfiles to at least 32768:

# echo kern.maxfiles=32768 >>/etc/sysctl.conf

XXX jail_example_parameters="allow.sysvipc=1" doesn't work with the vimage
start script.  Workaround: jail -m jid=$JID allow.sysvipc=1 for each jail
in a post start script. for each jail
in a post start script.

4. Bridge setup

To be able to support the many instances we will be running, the default
number of bridge links needs to be changed. Unfortunately, this requires
a kernel recompile at this state.


Change /sys/netgraph/ng_bridge.h, from:

/* Maximum number of supported links */
#define NG_BRIDGE_MAX_LINKS             32

to

/* Maximum number of supported links */
#define NG_BRIDGE_MAX_LINKS             128

Recompile the module if necessary:

# cd /sys/modules/netgraph/bridge
# make clean; make; make install

Reboot to test that everything loads properly

# reboot

5. Jail management

To make management of the jails easier, we use:

http://druidbsd.sourceforge.net/vimage.shtml

... which implements jail-like controls to VNET instances.

# pkg_add http://druidbsd.sourceforge.net/download/vimage-1.4.tbz

If this doesn't work:

# cd /tmp
# fetch http://druidbsd.sourceforge.net/download/vimage-1.4.tbz
# tar jxf vimage-1.4.tbz
# cp -p rc.d/vimage /etc/rc.d/

6. Packages

We're going to use mostly binary packages, but we may need to build
a few as well, so we'll need the ports collection:

        # portsnap fetch        (takes a while)
        # portsnap extract      (takes a while)

Let's get the pkg environment up and running:

        # pkg (only necessary on FreeBSD 9.x)
        # echo 'WITH_PKGNG=yes' >>/etc/make.conf
        # pkg2ng (XXX shouldn't be necessary with FreeBSD 10)

We'll install a couple of smart package management tools

        # cd /usr/ports/ports-mgmt/portmaster/ && make all install clean
        # portmaster -g shells/bash

Now, make sure you have all the packages listed in packages.txt built/available.

* Using the new (pkg) system which we setup earlier:

# portmaster -g category/package

Or, pkg install the packages from binary, then once done:

    # cd /usr/ports/packages/All && pkg create -a

Some packages will need to be custom-built, such as

* PHP (module support - but we could run with CGI in principle)
* opendnssec (softhsm must be enabled)
* bind910 (-DWITH_REPLACE_BASE)

... see the package list for more info

For these, extract the ports collection (portsnap fetch && portsnap extract
as explained above), then:

  # mkdir -p /usr/ports/packages/All
  
Then for each package:

  # portmaster -g <category>/<pkg_name>
  
  This will create a package of each port + the dependencies in
  /usr/ports/packages/All/*

Note: add this to /etc/make.conf:

OPTIONS_SET=REPLACE_BASE BATCH SIGCHASE


7. Prep master jail

- Create ZFS container

# zfs create zroot/jails/master

- Create a master host distrib - you can do this with a stock FreeBSD
  install CDROM mounted.

  # export BSDINSTALL_DISTSITE=file:///mnt/usr/freebsd-dists/
  # bsdinstall jail /jails/master

NOTE: if you have good Internet connectivity at this point, you
can leave out "export BSDINSTALL_DISTSITE" and the installer will
fetch the distribution from the network, in which case you will be
asked to select the mirror you want to install from.

   Once the installer has run:

  - The root password is unimportant, we'll override it later.
  - Accept all defaults
  - Choose No when asked about enabling crash dumps, or adding users now.

- Create package dir in master

        # mkdir /jails/master/packages
        # mount -t devfs devfs /jails/master/dev
        # mkdir -p /usr/ports/packages/All
        # mount_nullfs /usr/ports/packages/All /jails/master/packages

- Setup the pkg environment in the jail:

        NOTE: packages are preinstalled in this way; they can also be
    installed by the participants, but there's litlle value in that,
    as they will still have to configure them anyway.

        # jail -c path=/jails/master mount.devfs host.hostname=master ip4.addr=X.X.0.248 command=/bin/tcsh

        (replace X.X with the prefix, for example X.X if you are using X.X/16)

        Here the prompt should say:

        root@master:/ #
        
        Define a DNS:
        
        # echo 'nameserver X.X.0.254' >/etc/resolv.conf

        Now setup the pkg env:

        # pkg
        (answer y)

        # rm /etc/resolv.conf
        # exit

- Re start jail up and install the packages

  (Note: could also do this in the previous step - room for improvement here)

        NOTE: packages are preinstalled in this way; they can also be
        installed by the participants, but there's litlle value in that,
        as they will still have to configure them anyway.

        # id=`jail -i -c vnet name=master host.hostname=master path=/jails/master persist`
        # jexec $id /bin/tcsh

        # echo 'WITH_PKGNG=yes' >>/etc/make.conf

  Install the packages

        # cd /packages
        # pkg add ./*
        (if this causes issues, add the packages one at a time)

        If you get asked if you want to activate Postfix, says y

8. Exit & terminate the jail

        # exit
        # jail -r $id

9. Snapshot the system

        zfs snapshot zroot/jails/master@initial

Deploying the hosts
--------------------

First, decide on the IP prefix (/16) you will want to use for this workshop
instance.

1. You may want to add your keys to
   setup/configs/common/root/.ssh/authorized_keys

2. Adjust setup-*/build/hostlist.tmpl to match the hosts you'll be needing.

3. Cloning hosts and distribute the configuration files

Inspect 1-setup-zfs.sh, and make sure the src= parameter
is set to the zfs clone of the master (verify with zfs list -t snapshot)
and the dst= parameter is set as well.

Inspect 2-setup-basic.sh, and check the parameters at the top for:

- workshop username (sysadm)
- password (nsrc+ws)
- timezone (Region/Town)

Now run the scripts (as root)

        # ./0-replace-prefix.sh 10.10           # default is 10.10, or pick another
                                                                                # prefix
        # ./1-setup-zfs.sh
        # ./2-setup-basic.sh pushconf           # this pushes the predefined files
                                                                                # found under setup/configs
        # ./2-setup-basic.sh localchanges       # this carries out localchanges/tweaks
        # ./3-fixes.sh                                          # HACK - this needs to be merged into
                                                                                # one of the other scripts

3. Create vnet jail definitions in /etc/rc.conf (adjust nhosts)
   Make sure the interface name (if=) is set to the proper interface
   (em0, vtnet0, etc...) in the script before you run it:

        # ./4-setup-vnet-jails-defs.sh

4. Start jails 

        # service vimage onestart

5. Enable shared memory on every host (hack)

        for jail in `jls | awk '{ print $1 }' | grep -v JID`
        do
          jail -m jid=$jail allow.sysvipc=1
        done

6. Test ssh to a host

        ssh root@...
        ssh 

Setting up the class-wide resolver, auth servers and root servers
------------------------------------------------------------------

Configuration are pre-distributed via the config/ subdir, but the steps
below may be required prior to starting the services.

1. Setting up the resolver (X.X.0.230)

        # unbound-control-setup
        # service unbound start

        Note: that at this point the server will not be doing validation until
        a trust anchor has been configured - the root needs to be signed for
        this.

2. Setting up the auth servers

        By default:

        auth1 is running NSD and is the master for "dns.nsrc.org" and X.X.arpa
        auth2 is running BIND and is the slave for "dns.nsrc.org" and X.X.arpa

        Note: maybe this should be the other way around ?

        When the jails are started, everything should work
        (no intervention needed)

3. Setting up the root servers

        a.root is running BIND and is the master for "."
        b.root is running NSD and is the master for "."

        a.root should just start loading the zone on boot
        b.root as well, but you can force a dump of the zone to a file by issuing

        # nsdc patch

        Note: a key "root_zone_xfer_key" is defined on nsd.conf for b.root,
        but not used at the moment. There is no reason to restrict zone transfer
        of the root.

DHCP setup and resolution path
------------------------------

DHCP should give out 254 as the nameserver - or whatever device is
providing DNS for the "real" root.

The DNS handed out by the DHCP should be configured to forward requests
for dns.nsrc.org to the two auth servers. If using BIND, this would be:

zone "dns.nsrc.org" {
        type forward;
        forwarders { X.X.0.230; };
};

zone "X.X.in-addr.arpa" {
        type forward;
        forwarders { X.X.0.230; };
};

In this way:

- participant laptops get DHCP and use .254 (or whichever device can resolve
  the "real" root), but can still resolve dns.nsrc.org using the forward
  (or equivalent) statement above

- class VMs are using .230 by default which means they can't see the "real"
  root

TIPS AND TRICKS
---------------

To quickly flush the cache in unbound:

# unbound-control flush_zone .

(flushes all RRs at and below .)

NOTES:

- periodic scripts are disabled by the crontab included under
  configs/common/etc.

  This is to avoid killing the system with I/O when the periodic
  maintenance scripts are run.

- you may want to enable bsnmpd on the VMs. To do this, modify /etc/snmpd.conf
  as follows:

--- snmpd.config.orig   2013-08-12 07:32:38.255328350 +0000
+++ snmpd.config        2013-08-12 07:36:35.125447049 +0000
-location := "Room 200"
-contact := "sysmeister@example.com"
+location := "NSRC Workshop"
+contact := "me@example.com"
-read := "public"
+read := "NetManage"
-write := "geheim"
-trap := "mytrap"
+# write := "geheim"
+trap := "NetManage"

  ... enable bsnmpd in /etc/rc.conf

bsnmpd_enable="YES"

  Start snmpd:

$ sudo service snmpd start