These exercices demonstrate some of the tools used for tasks that every system administrator should perform when installing or hardening a system.
# apt-get install lsofFirst you can see what is running on your machine by typing something like:
$ ps auxwwwYou will see lots and lots of stuff go by. So, let's look at this a bit more closely:
$ ps auxwww | less(press [spacebar] to go one page down, and [b] to go one page up)
Now, browsing through all this we can see there are a bunch of initial system processes that start to support our hardware (items in "[ ]") as well as lots of processes associated with the Gnome Display Manager (gdm and gnome). Let's filter all of this out and see what we are left with:
$ ps auxwww | grep -v "\[" | grep -v gdm | grep -v gnome(Hint: You might want to copy and paste this in to a command window)
What's left?
Have a look and see if you can identify everything in the remaining list. Your list of processes should look something like:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 2644 1552 ? Ss Jun29 0:01 /sbin/init
root 429 0.0 0.0 2152 756 ? S Jun29 0:00 upstart-udev-bridge --daemon
root 431 0.0 0.0 2624 1020 ? S<s Jun29 0:00 udevd --daemon
root 613 0.0 0.0 2620 984 ? S< Jun29 0:00 udevd --daemon
root 614 0.0 0.0 2620 984 ? S< Jun29 0:00 udevd --daemon
root 780 0.0 0.0 1852 548 ? Ss Jun29 0:00 dd bs=1 if=/proc/kmsg of=/var/run/rsyslog/kmsg
syslog 782 0.0 0.0 33832 1748 ? Sl Jun29 0:00 rsyslogd -c4
104 803 0.0 0.0 2860 1104 ? Ss Jun29 0:00 dbus-daemon --system --fork
105 806 0.0 0.1 5352 3280 ? Ss Jun29 0:00 hald --daemon=yes
root 824 0.0 0.1 19412 2932 ? Ssl Jun29 0:00 /usr/sbin/console-kit-daemon
root 887 0.0 0.0 3344 1196 ? S Jun29 0:00 hald-runner
root 975 0.0 0.0 1704 552 tty4 Ss+ Jun29 0:00 /sbin/getty -8 38400 tty4
root 991 0.0 0.0 1704 548 tty5 Ss+ Jun29 0:00 /sbin/getty -8 38400 tty5
root 995 0.0 0.0 3420 1140 ? S Jun29 0:00 hald-addon-input: Listening on /dev/input/event0 /dev/input/event1 /dev/input/event4 /dev/input/event5
root 996 0.0 0.0 3420 1152 ? S Jun29 0:00 hald-addon-storage: polling /dev/sr0 (every 2 sec)
root 1001 0.0 0.0 1704 552 tty2 Ss+ Jun29 0:00 /sbin/getty -8 38400 tty2
root 1002 0.0 0.0 1704 556 tty3 Ss+ Jun29 0:00 /sbin/getty -8 38400 tty3
root 1005 0.0 0.0 5364 1100 ? Ss Jun29 0:00 /usr/sbin/sshd
root 1006 0.0 0.0 1704 548 tty6 Ss+ Jun29 0:00 /sbin/getty -8 38400 tty6
105 1017 0.0 0.0 3264 1120 ? S Jun29 0:00 hald-addon-acpi: listening on acpi kernel interface /proc/acpi/event
root 1036 0.0 0.0 2092 880 ? Ss Jun29 0:00 cron
daemon 1037 0.0 0.0 1964 416 ? Ss Jun29 0:00 atd
root 1063 0.0 0.1 6692 2332 ? Ss Jun29 0:00 /usr/sbin/cupsd -C /etc/cups/cupsd.conf
root 1170 0.0 0.2 6704 4816 ? Ss Jun29 0:00 /usr/sbin/munin-node
root 1245 0.0 0.0 1704 552 tty1 Ss+ Jun29 0:00 /sbin/getty -8 38400 tty1
root 1278 0.0 0.1 5168 2580 ? S Jun29 0:00 /usr/lib/devicekit-power/devkit-power-daemon
root 10340 0.0 0.1 8588 2972 ? Ss 00:07 0:00 sshd: root@pts/0
root 10400 0.0 0.0 4352 1872 pts/0 Ss 00:07 0:00 -bash
root 10556 0.0 0.0 2644 1024 pts/0 R+ 00:13 0:00 ps auxwwwYou can type "man" or search in Google to figure out what all this is. For instance:
$ man udevd
$ man hald
$ man getty
$ man cupsd
$ man atd
$ man cron
$ man sshdEtc, etc.
Once you feel pretty comfortable with what's running on your system you might consider if you need each item. If there is something running that is unnecessary, then you should consider uninstalling the software:
# apt-get remove <pkg_name>or stopping the associated service:
# update-rc.d <pkg_service> removeThe next step is to see if any of these services are listening to the network for connections:
# lsof -iYou'll see something like:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
sshd 1005 root 3u IPv4 5150 0t0 TCP *:ssh (LISTEN)
sshd 1005 root 4u IPv6 5152 0t0 TCP *:ssh (LISTEN)
cupsd 1063 root 5u IPv6 5318 0t0 TCP localhost:ipp (LISTEN)
cupsd 1063 root 6u IPv4 5319 0t0 TCP localhost:ipp (LISTEN)
sshd 10340 root 3r IPv4 18747 0t0 TCP pc4.pacnog.bluesky.as:\
ssh->noc.pacnog.bluesky.as:34634 (ESTABLISHED)Again, Google and man to figure out what is going on:
$ man sshd
$ man cupsdWhat's cupsd? is this necessary on every server?
Notice that sshd is listening to all incoming connection requests (the "*"). This is a typical, potential security hole.
In our case, we will leave ssh up, but we are aware they are running and need to be patched for security updates as they come out.
For example, it is a good idea to lock down sshd a bit by not allowing the root user to log in with a passwords.
As you are not printing, let's turn off the cups printing service. Do you remember how to do this?
# ls /etc/init.d <-- to find the service script name
# service cups stop
# lsof -iNow we only see:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
sshd 1005 root 3u IPv4 5150 0t0 TCP *:ssh (LISTEN)
sshd 1005 root 4u IPv6 5152 0t0 TCP *:ssh (LISTEN)
sshd 10340 root 3r IPv4 18747 0t0 TCP pc4.pacnog.bluesky.as:\
ssh->noc.pacnog.bluesky.as:34634 (ESTABLISHED)To prevent this service to start when the machine is rebooted, type:
# update-rc.d cups removeIt's usually a good idea to see how your machine looks to other users.
Log in to a PC different than yours. For example:
$ ssh sysadm@extXMake sure that nmap is installed by doing:
# apt-get install -y nmapNow let's scan your machine using the nmap command:
# nmap -sV extX [Where "extX" is _your_ VM]You should see something like:
Starting Nmap 5.00 ( http://nmap.org ) at 2010-06-30 09:59 SST
Interesting ports on ext2.ws.nsrc.org (67.218.55.102):
Not shown: 998 closed ports
PORT STATE SERVICE VERSION
22/tcp open ssh OpenSSH 5.1p1 Debian 6ubuntu2 (protocol 2.0)
25/tcp open smtp Postfix smtpd
MAC Address: 00:0F:1F:E6:62:94 (WW Pcba Test)
Service Info: Host: pc2.pacnog.bluesky.as; OS: Linux
Service detection performed. Please report any incorrect results at \
http://nmap.org/submit/ .
Nmap done: 1 IP address (1 host up) scanned in 1.76 secondsThis looks reasonable. The machine is exposing smtp and ssh to the world as well as the type of OS that it is running.
Now let's scan a bit more aggressively:
# nmap -A -T4 extXTake a look at the information presented. This will take some time, but it will contain more detail.
Now, remember to log out of your classmate's PC!
$ exitYou can use nmap to scan entire networks and find all the machines and services that are running. This is what network attack scripts do - but, they usually scan for a specific port and service, then they launch an attack when they find a service that they think they can break.
Be careful with nmap! If you scan aggressively or against an entire network you will likely set off detection alarms and you could get in trouble. Let people know before you scan if you are not in charge of the remote machines.
Now read about nmap to understand what -sV, -A, -T4 and -F are doing:
$ man nmapLet's install fcheck. This is an intrusion detection tool that is very simple to set up and is preconfigured to do most of what you want:
# apt-get install fcheckOnce this is done you can look around to see how fcheck is configured. By default Ubuntu installs and configures fcheck in a reasonable manner and you probably don't need to do anything else.
$ man fcheckConfiguration of check is in /etc/fcheck/fcheck.cfg. Let's have a look:
# EDITOR /etc/fcheck/fcheck.cfgRead through the file to see what directories fcheck is checking, which directories are excluded, etc. The check process is run once every two hours on the 1/2 hour. You can view this by looking at:
$ less /etc/cron.d/fcheckThe text that reads:
30 */2 * * *is telling our system cron process to run the long check command listed in the file once every 2 hours on the 1/2 hour.
Now force fcheck to run for the first time and create a database:
# fcheck -acLook at the baseline file that fcheck has created:
# less /var/lib/fcheck/fcheck.dbfNow let's make a change to a file in one of the directories that fcheck is checking.
# editor /etc/hostsAdd a blank line at the end of the file. Save the file.
Now do another forced run of fcheck:
# fcheck -aYou'll see lots of stuff go by on the screen.
you see something like:
PROGRESS: validating integrity of /etc/
STATUS:
WARNING: [cv-macbook] /etc/hosts
[Sizes: 257 - 258, Times: Jul 22 21:36 2010 - Mar 14 16:10 2012]This tells you that the file /etc/hosts has changed. The cron job installed by Ubuntu will e-mail these kinds of reports to you.
Inotify in the kernel can provide real-time notification of filesystem changes. Install the incron package and configure incrond to monitor important filesystems.
# apt-get install incron
# tail /var/log/syslog
# cd /etc/incron.d
# EDITOR globals
add the following line (one line) to the globals file:
/etc IN_MODIFY,IN_CLOSE_WRITE,IN_CREATE,IN_DELETE /usr/bin/logger -p
news.warn "$% $@/$#"For a description of the syntax of incron table files, see:
$ man 5 incrontabThat's it. The changes you make to incron are updated automatically. Because incron can recognize changes, it even recognizes when you change the configuration for incron, and it updates.
Now add a file to the /etc directory:
# touch /etc/dogTake a look at /var/log/syslog. What does it say???
# tail /var/log/syslogFrom now on, any changes you make in the /etc directory will generate syslog messages.
There is a meta package called unattended-upgrades to do this. To install:
# apt-get install unattended-upgradesThat's it. Any time a security update is placed in the Ubuntu repositories it will be automatically installed on your system. You will probably want to look at how unattended-upgrades is configured.
# cd /etc/apt/apt.conf.dThis package is configured in the file 50unattended-upgrades. Let's have a look and we will make a change to the configuration:
# vi 50unattended-upgradesNote at the very top of the file. If you were to change this:
// Automatically upgrade packages from these (origin, archive) pairs
Unattended-Upgrade::Allowed-Origins {
"Ubuntu lucid-security";
// "Ubuntu lucid-updates";
};To look like:
// Automatically upgrade packages from these (origin, archive) pairs
Unattended-Upgrade::Allowed-Origins {
"Ubuntu lucid-security";
"Ubuntu lucid-updates";
};then all software package updates would be installed as well. You may, or may not, want to do this. This is generally safer for user desktops than for servers.
Let's change this line:
//Unattended-Upgrade::Mail "root@localhost";To be:
Unattended-Upgrade::Mail "root@localhost";That way your root account will get an email when an update is installed.
Note that you can even have your machine automatically reboot if required after an update.
Save the file and exit.
That's it. If a security update is applied you will be notified.
There is a nice tool called "chkrootkit" - This is used to see if a machine has been compromised with known software kits that install once security has been breached. You can read about this software here: http://www.chkrootkit.org/
To install, do this:
# apt-get install chkrootkitTo use it, do:
# chkrootkitYou should not see anything found or infected (hopefully!). However, it's possible for the tool to give you some false positives. You can go back to the http://www.chkrootkit.org/ web site for more information in the README and FAQ pages and you should use Google. If you don't see other people reporting false positivies like yours, then you probably need to format your hard drive, reinstall and restore data from backups.
Let's do something to make chkrootkit give you a warning:
Place your ethernet interfaces in to promiscuous mode (i.e. it listens for all packets on the network, not just packets coming to your machine).
# ifconfig lo promiscNow let's re-run chkrootkit:
# chkrootkitand you will see that it detects that the loopback network interface (lo) is now in promiscuous mode. To just see this vs. all the other messages do:
# chkrootkit | grep PROMISCIf your PC is running a DHCP client daemon, you may also see that eth0 is in promiscuous mode:
eth0: PROMISC PACKET SNIFFER(/sbin/dhclient3[564])Turn off promiscuous mode for lo:
# ifconfig lo -promiscSystem accounting gives us logs of all the commands that have run and terminated on the system. Let's see if we have the acct package:
$ which saDid "which" find the command? If not install the package:
# apt-get install acct
$ which saLet's run a command and see if acct records it.
$ whoami
# sa -uDid "sa" show a record for the command?
Let's try the "lastcomm" command as well:
$ lastcomm sysadm--End