1 Overview

1.1 Objectives

In this lab, you are going to convert a flat network into a full routed network, by using the layer 3 features of the switch you have.

Rather than building a new network from scratch, we will convert the existing network step-by-step, so that each step can be rolled back individually if it doesn't work, aiming to keep any disruption as localised as possible.

Work together closely in your group, to make sure you all know what's happening at each step. Since you all have your laptops on the wired network, you will be able to tell if you have broken your campus!

1.2 Planned network

The network we are aiming to build looks like this:

Routed campus network

Notice how the core switch has turned into a core router. We will do this by using its layer 3 features.

1.3 Kit access instructions

• The Cisco console port is accessed via telnet to 10.10.0.241 port 210X, where X is your group number
• The DHCP VM is accessed via ssh to 10.10.0.X or hostX.ws.nsrc.org. The username and password are given out in class
• The core switch/router is a real device and accessed using a real serial console cable!

1.4 Master addressing plan

Each group has a block of "public" IP addresses 100.64.X.0/24 (where X is your group number), and will be using private addresses from the 10.0.0.0/8 range.

We have designed the following VLAN and addressing plan for you to use:

Extend this plan in the obvious way if you have additional "buildings".

Rationale:

• VLAN IDs run from 1 to 4094, but we want to avoid vlan ID 1 since the default VLAN has special significance and may be difficult to trunk
• We know that we might need multiple subnets in a building (e.g. one for staff wired network, one for computer lab, one for wireless). So we allocate blocks of 10 VLANs to each building, and use the first one for the staff wired network.
• We take the first half of the limited public IP space and use it for a server network. From the top half we take a /30 for the point-to-point link between the border and core routers, and reserve two more /30's for other P2P uses. Then the rest is available for NAT, so each building can have its own public IP address.
• Private addresses are plentiful so we allocate 10.<bldg>.0.0/16 for each building; this makes it easy to tell which building an addres belong to, just by looking at the second number.

This plan allows for 255 buildings before you have to subdivide IP blocks, and 409 buildings before you have to subdivide the VLAN ranges; it should suit a moderately large campus.

2 Prepare new NAT configuration

Before we start renumbering buildings, it is worth getting the NAT rules ready. Our plan has a separate public IP address for each building subnet. This has two big advantages:

• It scales better, because there are limited TCP/UDP ports available. With multiple public addresses, each one has its own set of ports.
• If there is abuse coming from one of your public IP addresses, you can immediately tell which network it came from. Furthermore, we want this from our own address space, not the P2P link to our provider.

We are also going to be using public addresses in parts of our network, and it's important that NAT is not used for those.

Apply the following configuration on the border router:

ip nat pool NAT140 100.64.X.140 100.64.X.140 prefix-length 25
ip nat pool NAT141 100.64.X.141 100.64.X.141 prefix-length 25
ip nat pool NAT142 100.64.X.142 100.64.X.142 prefix-length 25
ip nat pool NAT143 100.64.X.143 100.64.X.143 prefix-length 25
ip nat pool NAT144 100.64.X.144 100.64.X.144 prefix-length 25
ip nat pool NAT145 100.64.X.145 100.64.X.145 prefix-length 25
ip nat pool NAT146 100.64.X.146 100.64.X.146 prefix-length 25
ip nat inside source list 141 pool NAT141 overload
ip nat inside source list 142 pool NAT142 overload
ip nat inside source list 143 pool NAT143 overload
ip nat inside source list 144 pool NAT144 overload
ip nat inside source list 145 pool NAT145 overload
ip nat inside source list 146 pool NAT146 overload
ip nat inside source list 199 pool NAT140 overload
access-list 141 permit ip 10.1.0.0 0.0.0.255 any
access-list 142 permit ip 10.2.0.0 0.0.0.255 any
access-list 143 permit ip 10.3.0.0 0.0.0.255 any
access-list 144 permit ip 10.4.0.0 0.0.0.255 any
access-list 145 permit ip 10.5.0.0 0.0.0.255 any
access-list 146 permit ip 10.6.0.0 0.0.0.255 any
! Lab workaround: do not NAT anything coming from 10.10.0.X address
access-list 199 deny   ip 10.10.0.0 0.0.0.255 any
! NAT anything else with a private source IP
access-list 199 permit ip 10.0.0.0 0.255.255.255 any
access-list 199 permit ip 172.16.0.0 0.15.255.255 any
access-list 199 permit ip 192.168.0.0 0.0.255.255 any

Do a "show run". The configuration probably still includes your existing NAT config:

access-list 101 permit ip 192.168.0.0 0.0.0.255 any
ip nat inside source list 101 interface FastEthernet0/0 overload

(Optionally you could remove this; if so, these users would start using the new "fallback address" .140 for their NAT public address)

CHECKPOINT: Ensure that your clients on 192.168.0 are still able to access the Internet. To check what IP address they are being NAT'd to, make an ssh connection from a client to your DHCP VM. Then inside the VM, use the commands "w" and "netstat -nt" to see the source IP address of the connection.

In a real network, you could just point a browser to a site such as "whatsmyip.org". In the lab this doesn't work because we have a second layer of NAT.

Save your changes on the border router (write); do this after every successful change.

Note that all of our NAT rules explicitly match some private IP address. This means that public IP addresses should not be NAT'd. Note also that the NAT rule for NAT140 covers a wider range than the other rules, so it should only be used for addresses which don't have their own specific rules.

Changing NAT can be tricky to perform on a running router. You may not be able to remove a NAT rule while it is being used. You can clear the NAT state table, but it will be immediately re-populated if there are users on the network. You may need to temporarily disconnect the border while you make the change

3 Introduce the first routed subnet: NOC

Considering layer 3 (routing) only, our starting point looks like this:

    +--------+
    | Border |
    +--------+
        |192.168.0.1
        |
        |       192.168.0.0/24
--+---+-+-+---+---------------
  |   |   |   |
 .. buildings ..

Note that the core switch doesn't appear in this diagram, because it is not doing any routing.

3.1 Create new subnet

So the first step we can make is to introduce one new routed subnet. Let's add the new NOC Server network (vlan 2), routed via the core switch, which is about to become the core router.

    +--------+
    | Border |
    +--------+
        |192.168.0.1
        |
        +--- ..
        +--- buildings
        +--- ..
        |
        |192.168.0.2 (vlan 1)
     +------+
     | Core |
     +------+
        |100.64.X.1 (vlan 2)
        |
        |        100.64.X.0/25
------+-+---------------------
      |
   NEW SERVER

So what do we have to do? On the core switch, firstly you need to create vlan 2. "show vlan" will show what VLANs you already have.

! On NEW cisco switches this is done in configuration mode:
conf t
vlan 2

! On OLD Cisco switches there is a separate vlan config mode:
vlan database
  vlan 2
  exit

Now in configuration mode (conf t), turn on IP routing globally, and assign routing addresses on Vlan1 and Vlan2 (you may already have done this for Vlan1 as a management address)

ip routing
interface Vlan1
  description Legacy network
  ip address 192.168.0.2 255.255.255.0
  no ip redirects
  no ip proxy-arp
interface Vlan2
  description NOC Server network
  ip address 100.64.X.1 255.255.255.128
  no ip redirects
  no ip proxy-arp

Now select a port on the switch which you're not currently using (say port 30), and bring out this new vlan on that port.

interface Fa0/30
  switchport access vlan 2

Plug a laptop into this port, and give it a static IP address 100.64.X.2 with netmask 255.255.255.128 and gateway 100.64.X.1

From this laptop:

• Can you ping 100.64.X.1? The answer should be yes (if not, debug)
• Can you ping 192.168.0.2? The answer should be yes (if not, debug)
• Can you ping 192.168.0.1? No!

Why not? Think about it, and look at the output of show ip route on both the core router and the border router. What's missing?

Answer: the core router knows how to reach the 192.168.0.0/24 network (because it's connected to it), but the border router doesn't know how to reach the 100.64.X.0/25 network. So let's fix that:

! On the BORDER router
ip route 100.64.X.0 255.255.255.128 192.168.0.2

While we're at it, the core router also needs to know how to reach other networks, and the old "ip default-gateway" doesn't work for this when routing is enabled.

! On the CORE router
ip route 0.0.0.0 0.0.0.0 192.168.0.1
no ip default-gateway

CHECKPOINT: check that your test laptop on the new NOC Server network is able to ping 10.10.0.241 (which is on the upstream network). If not, debug.

Aside: we haven't set up DHCP on this network, but a server network wouldn't normally have DHCP anyway

Meanwhile, confirm that all the other clients are still working on the old 192.168.0 network and blissfully unaware of the changes which are taking place.

3.2 Check no NAT

IMPORTANT: this new network is on public IP addresses and we need to make sure that its outgoing connections are not being subject to NAT!

This should be the case, because our NAT rules explicitly match only private addresses.

To test this, SSH from your test laptop on the new server network to the DHCP server, then type "w" and "netstat -nt" to see the IP address that you are logging in from. This should be the 100.64.X.2 address, not the outside address of the router.

3.3 Renumber servers

In a real network, you would now renumber your servers one by one, moving them from the 192.168.0.0/24 network onto the new 100.64.X.0/25 network. We will not concern ourself with server renumbering here.

3.4 Examine the config

Look at your network now, and check you understand how it is working. Realise that the core switch is now performing two different functions:

• For people on the 192.168.0 network, it is still acting as a switch
• For the server network, it is acting as a router

A layer 3 switch can be configured to do both roles simultaneously, which is useful while we migrate the network, but in the end we want to end up with a simpler configuration where it is only routing.

4 Renumber first building: Admin

Let's now do our first real building. (Maybe in reality you would do the building with the president's office last, not first :-)

          +--------+
          | Border |
          +--------+
              |192.168.0.1
              |
              +--- ..
              +--- other buildings
              +--- ..
              |
              |192.168.0.2 (vlan1)
   (vlan10)+------+
   10.1.0.1|      |100.64.X.1 (vlan2)
-----+-----| Core |-----+------
     |     |      |     |
NEW ADMIN  +------+  NEW SERVER

We can prepare for this change without affecting the users, by creating the new VLAN and assigning it to a spare port, and testing until we're happy.

4.1 Create the new VLAN

! On the CORE router
vlan 10
interface Vlan10
  description Admin building staff wired network
  ip address 10.1.0.1 255.255.255.0
  ip helper-address 10.10.0.X
  no ip redirects
  no ip proxy-arp

4.2 Update the DHCP server

You need to tell your DHCP server about the new subnet so it will allocate addresses for it. This needs to be done for every building you migrate.

Edit /etc/dhcp/dhcpd.conf and add:

subnet 10.1.0.0 netmask 255.255.255.0 {
  option routers 10.1.0.1;
  range 10.1.0.10 10.1.0.246;
}

Do service isc-dhcp-server restart, and tail /var/log/syslog to check there are no errors reported.

In our lab you also need to add a static route on the DHCP server. It should be sufficient to add a single static route covering all your private address space in one go:

# route add -net 10.0.0.0/8 gw 10.10.0.22X

You can make this persistent in /etc/network/interfaces if you wish.

4.3 Add static route

The border will need a static route to reach the new network.

! On the BORDER router
ip route 10.1.0.0 255.255.255.0 192.168.0.2

4.4 Assign new VLAN to a test port

Pick a spare port, let's say port 31.

interface FastEthernet0/31
  switchport access Vlan10

This allows you to connect a test laptop without affecting the live building.

CHECKPOINT: Give it a complete test. Does the laptop pick up an IP address from the new range? Is it being routed correctly, including to the outside world? Does it NAT to 100.64.X.141 (the NAT address for this new subnet?) If not, debug the problem. Remember that the live admin building has still not been affected at this point.

4.5 Assign new VLAN to the actual building

Finally, locate the port where the Admin building is connected, and make the change.

interface FastEthernet0/NNN
  switchport access Vlan10

4.6 Kick the clients

Note that all the clients in this building will now be on the wrong subnet (they will have 192.168.0 IP addresses until their DHCP lease expires). So you may need to restart them, or temporarily shut down the network ports they are connected to, to force them to pick up a new IP address.

If you are being smart, you would have first reduced the lease time on the DHCP server to (say) 5 minutes, and waited for the old lease time to pass, before making the change. This would guarantee that all devices would pick up a new IP address within 5 minutes.

Note also that you may have some devices on static IP addresses which need manually locating and changing. The best way to locate them is to plug a laptop into another port on Vlan10 (i.e. your test port), configure it statically with an IP address from the old range (192.168.0.N), and then do an IP scan using a tool like "AngryIP" for Windows, or "nmap" for Linux (nmap -sP -n 192.168.0.0/24)

This will show you any devices on that subnet which have addresses from the old range; your ARP table will then show you the MAC addresses and you can trace them using the mac-address-table in your switches.

This is now starting to look very much like a real routed campus network.

5 Move old network behind the core

Notice that the link between the core and border routers is still the old, unreliable 192.168.0.0/24 network with most of the campus buildings on it, and therefore a network problem will still affect even the new subnets.

We could leave it there until all the other buildings have been renumbered, but that could take a long time. So instead, we're going to move the old 192.168.0.0/24 network behind the core router now, so it looks like this:

          +--------+
          | Border |
          +--------+
              |100.64.X.129
              |
              |100.64.X.130 (vlan3)
   (vlan10)+------+
   10.1.0.1|      |100.64.X.1 (vlan2)
-----+-----| Core |-----+------
NEW ADMIN  |      |  NEW SERVER
           +------+
              |192.168.0.1 (vlan1)
              |
      --+---+-+-+---+--
        |   |   |   |
    .. other buildings ..

Notice how the 192.168.0.1 address is moving onto the Core router. The core router will become the default gateway for the 192.168.0.0/24 network, instead of the border router.

Furthermore, the Core router is going to be responsible for relaying the DHCP broadcasts from the 192.168.0.0/24 users.

This change does require a network-wide outage, although it's still a relatively small number of steps which can be easily reversed.

Firstly, on the core router, create vlan 3 with the correct IP address and netmask.

! On the CORE router
vlan 3
interface Vlan3
  description point-to-point to border
  ip address 100.64.X.130 255.255.255.252
  no ip redirects
  no ip proxy-arp

At this point, we are going to have to break the network. Make sure you have CONSOLE connections to both the core router and the border router (get somebody to put their laptop on wireless to access the border router's console port)

Renumber the inside interface of the border router:

! On the BORDER router
interface Fa0/1
  description point-to-point to core
  ip address 100.64.X.129 255.255.255.252

Find which port the border router connects to on the core router, and change it to be in vlan 3.

! On the CORE router
interface FastEthernet0/NNN   << uplink to border
  switchport access vlan 3

At this point, the border and core should be able to ping each other on their new IP addresses - test this. (e.g. on core, ping 100.64.X.129)

However the connectivity to the rest of the campus is still broken. You need to move the 192.168.0.1 IP address onto the core router, and enable DHCP relay:

! On the CORE router
interface vlan1
  ip address 192.168.0.1 255.255.255.0
  ip helper-address 10.10.0.X

And you need to set up static routes in both directions. The border router needs routes to all networks which are behind the core router:

! On the BORDER router
ip route 192.168.0.0 255.255.255.0 100.64.X.130

no ip route 100.64.X.0 255.255.255.128 192.168.0.2
ip route 100.64.X.0 255.255.255.128 100.64.X.130

no ip route 10.1.0.0 255.255.255.0 192.168.0.2
ip route 10.1.0.0 255.255.255.0 100.64.X.130

And the core router needs a new default route to the outside world:

! On the CORE router
no ip route 0.0.0.0 0.0.0.0 192.168.0.1
ip route 0.0.0.0 0.0.0.0 100.64.X.129

After this, check from the core router you can ping upstream address 10.10.0.241.

In principle all the campus should be working again. However, the default gateway 192.168.0.1 is on a different device with a different MAC address. Client devices may have the wrong MAC address in their ARP tables. You can manually clear the ARP table on those devices (arp -d).

You might also shutdown and unshut the port they are connected to:

interface FastEthernet0/NNN    << port where laptop is connected
  shutdown
  ! wait 5 seconds
  no shutdown

This will force them to re-DHCP, and therefore checks that DHCP is working. If not, debug (see previous exercise for how to check DHCP logs)

CHECKPOINT: Ensure that everyone is working again, i.e. everyone on the 192.168.0 network and the new NOC network has full connectivity.

6 Tidy up static routing

We know that all the buildings will have either 100.64.X.0/24 public addresses or 10.0.0.0/8 private addresses. The border will need to have routes to reach them.

Rather than adding a separate static route for each subnet, we can add big routes which cover the entire address space we have available:

! On the BORDER
ip route 10.0.0.0 255.0.0.0 100.64.X.130
ip route 100.64.X.0 255.255.255.0 100.64.X.130

! now you can remove the more specific routes you have
no ip route 10.1.0.0 255.255.255.0 100.64.X.130
no ip route 100.64.X.0 255.255.255.128 100.64.X.130

However there is a downside: if someone pings an unused address, the packet may bounce back and forth between the border router and the core (which has a default route pointing back at the border).

We can solve this problem by adding a Null0 route on the core:

! On the CORE
ip route 10.0.0.0 255.0.0.0 Null0
ip route 100.64.X.0 255.255.255.0 Null0

Now packets for unused addresses will arrive at the core and be discarded.

CHECKPOINT: After this change, make sure everything is still working.

7 Migrate other buildings

Now you've successfully moved one building onto its new routed subnet, repeat for all the other buildings, one at a time.

Eventually there will be no users left on the 192.168.0.0 network, and no access ports in Vlan1. You can then remove the IP address and helper from Vlan1. Since Vlan1 is the default VLAN, you may or may not be able to remove it entirely from the switch. You can also remove the NAT for 192.168.0.0/24 on the border router, if you have not done so already.

8 Extension exercises

• How would you route multiple subnets to the same building, for example, separate subnets for wired staff, wired labs, and wireless? Discuss and/or draw up a plan
• On your DHCP server, build a DNS cache using unbound or bind, and change your DHCP server so that clients use your DNS cache instead of the upstream one
• Configure IPv6 on your network, using SLAAC
• Convert your IPv6 to using DHCPv6 instead of SLAAC

9 Appendix: configuration snippets

9.1 Cisco configuration

# Basic setup
hostname <NAME>
!
aaa new-model
aaa authentication login default local
aaa authentication enable default enable
username nsrc secret nsrc
enable secret nsrc
service password-encryption
line vty 0 15
 transport preferred none
line console 0
 transport preferred none
!
no logging console
logging buffered 8192 debugging
no ip domain-lookup
ipv6 unicast-routing

# Enable ssh
ip domain-name ws.nsrc.org
crypto key generate rsa modulus 2048
ip ssh version 2
line vty 0 15
 transport input ssh

# Disable VTP and PVST (Cisco proprietary protocols), use MST/RSTP instead
vtp mode transparent
spanning-tree mode mst

# Set root bridge priority to 4096
spanning-tree mst 0 4096

# List VLANs/create a VLAN
show vlan

# Create a VLAN (new software)
vlan 10

# Create a VLAN (old software - not in configuration mode)
vlan database
 vlan 10
 exit

# Configure a switch port as access port to a VLAN
interface FastEthernet0/1
 switchport mode access
 switchport access vlan 10

# Configure a switch port as a tagged trunk
interface FastEthernet0/1
 switchport mode trunk
 switchport trunk encapsulation dot1q
 switchport trunk allowed vlan 10,20,30

# Enable layer 3 functionality
ip routing

# Create an router IP interface on a VLAN
interface Vlan10
 ip address 192.0.2.1 255.255.255.0

# Enable DHCP relay
interface Vlan10
 ip helper-address 6.7.8.9

# Show forwarding table
show ip route

# Add static route
ip route 10.10.1.0 255.255.255.0 1.2.3.4

# Add default route
ip route 0.0.0.0 0.0.0.0 1.2.3.4

# Shutdown a port (to force client to re-DHCP)
interface FastEthernet0/1
 shutdown
 ! wait about 5 seconds
 no shutdown

# ARP cache manipulation
show ip arp
clear ip arp

9.2 Linux configuration

9.2.1 DHCP server

• The configuration file is /etc/dhcp/dhcpd.conf
• After config changes, restart using service isc-dhcp-server restart
• Logs can be viewed using grep dhcpd /var/log/syslog

• For each subnet you want to serve, add a subnet declaration like this.

  subnet 10.1.1.0 netmask 255.255.255.0 {
    option routers 10.1.1.1;
    option subnet-mask 255.255.255.0;
    option domain-name "ws.nsrc.org";
    option domain-name-servers 10.10.0.241;   # this is the class DNS server
    range 10.1.1.10 10.1.1.246;
    default-lease-time 300;
    max-lease-time 300;
  }

• You can monitor DHCP traffic using tcpdump -i eth0 -nnev -s0 udp port 67

• The server should only be listening on the eth0 interface. This is defined in /etc/default/isc-dhcp-server

9.2.2 Static routes

• To add a static route:

  route add -net x.x.x.x/x gw y.y.y.y

• To make this change persist across reboots, edit /etc/network/interfaces and add lines like this:

  auto eth0
  iface eth0 inet dhcp
  post-up route add -net x.x.x.x/x gw y.y.y.y
  pre-down route del -net x.x.x.x/x gw y.y.y.y