Introduction

The NSRC has been helping ISPs around the world for many years. Over time we have seen many tools, tactics, and different methodologies for putting together ISPs. Based on our experience and expertise we present what we believe to be relevant information to help you get started and tools and methods that we have found to work. We present this information based on experience. And, based on this experience we do not, generally, recommend Microsoft solutions to build a reliable and secure ISP infrastructure. We do, however, present information about Microsoft products to allow you to place them in context. In addition, we cannot stress strongly enough the importance of planning for the future in every aspect of your ISP's development. If you are successful as an ISP, then your implementation must be able to scale with your success.

Connectivity Information

Satellites and Operators

Undersea Cables and Trans-Oceanic Backbones

Getting Address Space

To obtain network addressed we recommend that you read the RFC 2901, "The Guide to Administrative Procedures of the Internet Infrastructure". It is available in multiple different languages at:

Other Information about IP Addressing

Regional Internet Registries (RIRs)

As you progress in the process of obtaining address space, each of the four regional organizations that deal with IP address registration and administration have pages discussing how to request IP address space for new and continuing ISPs, links to these documents are located below.

Setting up Routing

As this is such a large topic we strongly suggest that you go over in detail the excellent resources presented at the AfNOG Workshop web site for the Scalable Network Infrastructure track at:

IP Numbers and AS Numbers

Understand how you request a range of IP addresses for your network and how data from your network will be routed to the rest of the Internet.

Operational Guidelines

Learn how to set up your network and your network hardware in a practical manner.

Core Services and Managing Them Locally

As an ISP you will be responsible for maintaining several core services that your clients use to gain Internet connectivity. Some of the most critical are DNS, DHCP, and maybe dialup. Each of these is discussed below with pointers to information about setting them up and using them.

DNS (Domain Name Server)

It's important that you set this up correctly from a functional standpoint. For instance, you run your primary DNS server locally, but ideally you should have your secondary, or backup, DNS server on a different network. This way if your other network functions are up, but your DNS box is down your clients can still make use of their Internet connections.

Below we present some first-level discussion of DNS, then we have more comprehensive list of Registries and DNS setup and maintenance links, including mulitple RFCs and their descriptions.

The Domain Internet Groper (DIG) comes as part of the Bind (http://www.isc.org/products/BIND/) distribution. DIG (the command "dig" is lowercase when you use it) allows you to find domain information as needed. This tool is used in place of nslookup.

Setting up and Maintaining a DNS Server

DHCP (Dynamic Host Control Protocol)

DHCP is not all that difficult to setup, but in a large environment it is important to decide upon the rules of engagement. If you have a fixed number of IP addresses then how you allocate them, for how long, and who gets static IP addresses becomes important. As you implement your DHCP services consider the following issues:

  • What devices get static IP addresses. Generally this includes things like printers, routers, servers, and anything else that must have the same IP address everytime it is turned on.
  • How long are your DHCP leases? That is, how long until an IP address is recycled back in to your IP address pool? For dialup connections this is usually as soon as the client disconnects.
  • Build a dhcpd.conf file that is well documented and laid out by located (networks) so that it is easy to update and maintain.

    One of the best DHCP resources is the man pages for dhcp. Simply type "man dhcp" at the operating system prompt under any UNIX system to get started. The current version of the DHCP server for UNIX can be found at:

And, information about DHCP is available from these sites:

Dialup (RADIUS)

Setting up a process by which you authenticate users dialing in to your network (modem pool) is likely to be a critical piece of your operation as an ISP. In addition this is one point at which you can monitor user access to your services and keep track of usage for billing or statistical purposes. Below is a short list of some excellent Radius resources:

Proxy and Cache Servers

World Wide Web proxy servers speed up the access to web based resources by locally storing copies of frequently accessed web sites. Instead of having to reconnect to a slow webserver, after the first view, the web page will be served from the local proxy cache. This results in much faster page views than reloading the page from the origional server. The more disk space on your proxy, the better chance you will have of improving the performance of your client's web browsing as well as reducing the total bandwitdth used for your incoming network connection(s). The most widely used, and freely available, proxy server for Linux and FreeBSD is squid. You can find this at http://www.squid-cache.org/. In addition, here are some more useful links:

MTA (Mail Transfer Agent)

There are threes interface involved in this discussion. These include:

In general you will run one command to transfer mail from your machine to another, or to receive incoming mail. Under UNIX the most used server to do this is sendmail. The next most used product is qmail. After this comes Exim. The NSRC recommends and uses Exim whenever possible running under FreeBSD. Exim is very secure and has been used in extremely large-scale implementations (up to a million users!).

  • Exim

    • Exim has been shown to scale to massive implementations, is generally easier to implement than sendmail, and is quite secure. You can read about the latest book, written by the auther of Exim (Philip Hazel), at http://www.oreilly.com/catalog/exim/. Otherwise, be sure to see the official web site at http://www.exim.org/.

  • Postfix

    • Another alternative to sendmail. Previously known as vmailer and as the IBM Secure Mailer when released in 1998. Postfix is preferred by administers at some sites. Read all about it, download it, and find updates at http://www.postfix.org/.

  • Qmail

    • qmail is famous for its ease of use, quick response to reader input for changes, and the $1,000 reward to the first person who could crack it - the prize went unclaimed and there is currently a $500 prize that you can read about at http://cr.yp.to/qmail/guarantee.html. You can read about qmail at http://www.qmail.org/.

  • Sendmail

    • Sendmail is the grandfather of Open Source MTA programs. O'Reilly has printed the definitive sendmail guide that you can read about at http://www.oreilly.com/catalog/sendmail2/, and you can see the Desktop reference at http://www.oreilly.com/catalog/sendmailqr/. The main sendmail site is located at http://www.sendmail.org/. One thing to be aware of is if you use sendmail there are numerous security exploits with this product. Whatever you do, do not run old versions of sendmail! There are well-known, and commonly exploited security issues. You should make it a point of keeping your sendmail versions up-to-date.

General Issues with MTAs

sendmail has been the de facto standard of mail delivery agents for years, but with the maturing of products such as Exim and qmail this is no longer as true.

If you need to scale your mail server there are two things that can help dramatically:

Email Delivery - IMAP/POP/Webmail/Console


This is a difficult topic. Which protocol or method do you wish to use to allow your users access to their email on your site? There is no perfect answer to this question. Generally the answer you come up with will be a balance of security, reliability, and convenience.

The fundamental issue will come down to whether you use IMAP, POP, or something like webmail as your primary mail access method. The advantages and disadvantages of each method are described below. If you choose to use IMAP, then there are several IMAP servers available for use. Generally IMAP servers will include POP servers as well. You can read about some of the IMAP servers available for FreeBSD/UNIX/Linux at http://www.imap.org/. A popular IMAP server for UNIX/FreeBSD/Linux come from the University of Washington and is availble at: http://www.washington.edu/imap/. And for a long list of IMAP products see:

http://www.imap.org/products/longlist.htm

Summary of Email Delivery Mechanisms and Links

  • IMAP: The email client that understands the IMAP protocol can see the message headers and folder names on your server, but actual email is not downloaded to the client machine until a message is opened. In addition, email stays on the server once a session is over.

    • Advantages
      • You can use any IMAP enabled client and see your email in a consistent manner from any workstation with an IMAP-enbabled client.

      The end-user does not need to backup their email as it remains on a central server, which is most likely to be backed up regularly.

    • Disadvantages

      • You (the system administrator) must allocate enough space to hold email for each user. If email is an important aspect of a user's life, then you may need to consider 100MB/User, or more, of space!

      The user must remain (generally) connected during an entire IMAP session. In many countries this is very expensive, or impractical.

      Even though, in theory, users can check their email from any IMAP-enabled client, this is non-trivial. The user is likely to leave traces of email, password and account information, etc. on any machine where they set this up. In addition, setup is non-trivial.

      If the user cannot access the IMAP server, then they do not have access to their email.

  • POP: With this protocol the email client downloads email from the server to the client machine. The POP protocol is smart enough to know what messages are new and in what state they are. That is, if you read them on-line, then when they are downloaded they will not be marked as new. In addition. the user can decide to leave the messages on the server, delete them, delete them after a certain period of time (POP3), or delete them only when they empty trash on their machine. The UW IMAP server comes with POP as well and can be found at:

    • http://www.washington.edu/imap/

    • Advantages

      • As long as the user can connect to the Internet, then they can probably download their messages using POP.

      The end user can have their email saved in multiple places (server, desktop at work, laptop for the road, desktop at home).

      POP allows you to "pop" (i.e., "get") email in a single, quick session, then work on the email received offline and send them at the next opportunity without needing to be constantly connected.

      Many, many clients speak and understand POP.

    • Disadvantages

      • Many users become very confused as to the state of their email. The often do not know where the latest download is or how to keep their mail syncronized. POP on multiple machines is not for novice users.

      Running POP from multiple locations can cause problems.

      Many uses forget to ever delete email from the server, and will fill their disk space, which results in rejected email by most systems.

      If the user evers gets (i.e., "pops") their email to a foreign system, then their email is on that system unless the user remembers to delete it.

  • Webmail: This method allows a user to connect to the Internet and get their email from within a web browser. Generally email stays on the system and it is generally much easier to deliver email in a secure fashion using webmail. RoundCube is an excellent web mail client, it is available at: http://roundcube.net/
    • Advantages

      • To secure webmail you just need to run a secure webserver. This is generally easier than generating certificats to run POP or IMAP servers in secure mode.

      Users can get their email from any machine with a web browser and an Internet connection. This bears repeating... Users can get their email from any machine in the world with a web browser and an Internet connection. This singlehandedly has made Microsoft's Hotmail one of the most successful email clients in the world.

      Their are many webmail servers to choose from and many are free.

    • Disadvantages

      • Email remains on the server. If you don't have an Internet connection, then you don't have access to your email.

      The webmail interface, filter abilities, etc. are often inferior to what standalone clients may offer.

      You may have to put up with advertising or large amounts of spam (see Microsoft Hotmail in this regard) to use a webmail service.

  • Console-Based: This is generally any email client that runs directly on the server where your email resides, but that presents your email in a text-only format usually in a telnet or ssh terminal session window. We strongly recommend you use console-based clients with ssh connections only. A few common console-based email programs include Alpine (http://www.washington.edu/alpine/), which is the most popular. The "mail" program is almost always available in any UNIX/Linux/FreeBSD flavor. Mail is much more rudimentary, but is functional for sending and receiving email. Note that console-based clients do not really apply to Windows servers as the concept of connecting to the server wth a shell is not well supported.
    • Advantages Your email is always available and backed up on a central server.

      Console-based email packages such as Pine are 100% compliant when it comes to email standards (i.e., they tend to send ASCII only!).

      You can manipulate your email directly and with considerable power once you learn the console-based client well.

      You can access your email from any Internet connected machine that runs telnet (most all machines, but this is not secure!) or ssh.

    • Disadvantages
      • You must connect to the server to get your email. This is not strictly true. If you understand how to do this you can download your current "Inbox" from the server to a local machine and run a console-based email client, like Pine, on your local machine. This is similiar to what you can do with POP.

      If you connect to the server, then you must stay connected to send and receive email. This is expensive and impractical in some situations or countries.

      If you are connected over a slow connection, then typing in telnet or ssh can be painfully slow.

  • Conclusion

    When you pick the system that you are going to support you will have several tradeoffs between ease of use, security, reliability of the connection method, and cost. Webmail has become very popular and allows you to secure the email sessions easily. Remember, SSL connections require considerably more CPU time so you must scale your hardware appropriately. Webmail, however, requires a constant connection. POP clients are good for users who want to get their email, download it, disconnect, respond, reconnect and send. But, POP is largely insecure. Most POP clients will talk to a POP server via SSL. We strongly recommend you implement this since POP without SSL passes a user's password and ID in clear text each time they check for mail.

    • One final issue to remember if you, or your users, run multiple protocols for accessing email is that these methods can cause problems with the state of email. For instance, some versions of POP and IMAP do not mix well. If you "touch" your mailbox using IMAP, and then attempt to access it using POP you may find that POP suddenly thinks that all your messages are new again. In addition, dependent on the servers in use, a user trying to access their email from multiple locations using multiple protocols at once can cause problems. Generally the first process accessing the email will lock out others from having write access, but this is not always guarranteed under all circumstances.

Mailing Lists (Majordomo, Mhonarc)

There are many tools available for managing mailing lists, but the one tool we recommend over the rest is Majordomo. You can read about Majordomo in detail at the links below:

Additional tools and mailing list software that you may want to look at include:

Security

Securing your system from attack is a basic responsibility of any good system administrator. Doing this is not an easy task and staying on top of how to secure you system is of critical importance. At this stage of the Internet ) if you do not secure your site, then you are guaranteed to be compromised. Your site will be broken in to and you are likely to lose data, have data corrupted, and lose service. Securing your system and knowing how to respond to security threats and breaches is of the utmost importance. If you have any doubts about this you should see statistics from the HoneyNet Project at http://project.honeynet.org/, which includes reports of one unsecured Linux box that was compromised within 45 minutes of being placed on the net.

Below we give you a few basic guidelines about securing network servers, but before you install, configure and place a server on-line we strongly suggest that you read our entire security document located at (opens in a separate window): http://nsrc.org/security/ This document contains detailed advice, pointers, and specific information about the steps you need to secure your network servers. You should do this before you allow any of your servers to be connected to the Internet.

Security is an ongoing process, but there are some basic guidelines you can follow to secure most servers. Securing an entire network involves additional planning and work. Larger organizations will usually have an entire staff person dedicated to security. Generally, security is much easier to implement if you do it from the start and you provide just the services you need vs. providing everything out of the box and then trying figure out what you can turn off.

General Guidelines

Very quickly, here are some general guidelines for securing a server on your network. If you are new to security be sure that you go to some of the links listed below and read them in depth. A few pointers include:

  • Turn off unnecessary services: If you don't need to run that printer daemon on your FreeBSD box, then don't. It has security holes!
  • Use Filters: That is filter the packets that come in to your system and do not let all of them through to the services that are running. This includes using tools like inetd or xinetd on your Linux server.
  • Segregate services on different boxes: If you plan on running telnet or sendmail or other insecure servers, and you need to run core services to keep your business up, then consider splitting these to separate boxes. That way if one is compromised your whole network set of services is not compromised.
  • Put some services behind firewalls: Many organizations see firewalls as a guaranteed way to protect their network. But, the problem with a firewall is that you often must open ports on the firewall to allow your users access to the services they want to use. And, you probably guessed this, the services your clients want to use are the most likely ones to come under attack. All is not lost, however, as a firewall can certainly reduce security problems from port scans, less used services, etc.
  • Keep up with patches! Pay attention to security patches for the operating system and services you are using. Apply them, especially when they address security holes. If a security hole is found in a service you run it is almost a guarrantee that your server will be attacked and compromised quickly (hours or days) once this hole becomes well know.
  • Consider System Integrity Checkers: Including things such as the samhain (available at: http://www.la-samhna.de/samhain/). These products have secured known databases of your filesystem. If anything changes that you do not expect to change, then you are likely to be notified and you can deal with the problem. Tripwire, and such systems are hard to install and are really only effective if you install them right when you first build your server. Once you rserver has been built and running for a while, then you do not know if it has been compromised or not.

Backup

This cannot be stressed enough. At some point you will lose data. Your system will experience a hardware failure. You may have a security breach that results in data loss or corruption. The power will go out or or spike. You will experience a natural or man-made disaster. One of these, or more, is guarranteed to happen. Your only method for recovering from such situations is to have a safe, usable, tested (many people forget this step), and reliable backup of your data.

How you decide to implement this will depend on your hardware setup, amount of data that needs backing up, and your budget. The general rule of thumb is the easier and more data backed up, then the more expensive the solution. Often administrator's attempt to backup entire systems, when the real goal is to be able to get your user's data back and your current configuration up and running.

When designing your network and server setup try to design so that unique configuration files and user data is segmented in to distinct areas. If, for instance, you know that you need to backup /home, /data/ and a few system directories like /etc, /usr/local, etc., then your backup setup and configuration will be much easier.

Your next step is to decide how quickly you need to be able to recover your system and user's data in case of failure. Remember, even if you decide this must be fast, you will need the additional hardware and ability to restore to that hardware to make this happen.

Here are some good sites that discuss backup solutions (commercial and freeware) for servers:

Physical Infrastructure

When planning to set up an ISP be sure that you pay close and careful attention to your physical space and infrastructure. It is easy to think about hardware and software, but forget about the supporting pieces. Below are items that you should consider and links to some good pages that discuss these are in more detail:

  • Space and Power

    Do you have a large enough space set aside to hold your staff and your equipment? In addition, can you get the necessary power and phone lines to your chosen location. Are there any issues with the location and bringing the network to that location? What about power? If you include air conditioning, servers, desktop machines, modems, routers, switches, etc are there enough? You should probably do some planning to figure out how much power your equipment will need, then how far you can scale before you run in to problems.

  • Racks and Wiring

    As your organization grows you will quickly realize that having a well thought out and clean system for your patch cables in place can save you countless hours of crawling around, tracking ends, and accidentally disconnecting machines. In addition, this can be faciliated by placing your switches, routers, hubs, and CPUs in racks.

  • UPS and/or Generators

    If your area experiences power outtages, brown-outs, or power spikes, then you will need more robust UPS equipment. But, in all cases you need to protect your equipment with a Uninterrupted Power Supply (UPS) system and/or a generator. Power that is not clean can cause serious damage to equipment. If you lose your modems due to a power outtage or spike, then how long will it take before you are back up and running? If the answer is days or weeks, then your business is not likely to survive such outtages. A reliable and robust system of UPSs and/or generators becomes critical. A great place to start planning for what you will need is the American Power Conversion web site at http://www.apcc.com/. They have excellent UPS products, but they also have very useful on-line planning tools as well.

Client Support/Help Desk

Implementing a proper Help Desk from the beginning will be critical to your long term success. If you can get clients up and running quickly with your service and you can support them well, then it will be all that much easier to grow. For a detailed look at how to set this up and to see what some other organizations have done take a look at our Help Desk section at:

Trying to implement a help desk after you have attracted a large user base can be quite painful. If you read through the pages above, particularly the ones at the NSRC, you will get a good feeling for what you are likely to be implementing as your ISP becomes successful and grows. A Help Desk and Support organization does not stop at just creating a number, or email address where your customers can reach you, but becomes a philosophy behind how you provide service. Remember, this document stresses scalability in all phases of your operation. This includes how what you do will affect your ability to be able to support your clients. The NSRC Help Desk pages, particularly the Setting up a Functional Support Organization section (http://www.nsrc.org/helpdesk/creating.html) goes over these ideas in detail with practical examples and solutions.

Conclusion

[Return to Top]

This document is meant to give you a first level look at the pieces involved with setting up an ISP and then point you to much more detailed documents and tools that you can use to get up and running. We welcome your feedback. As you can see, there are a very large number of links in this document. If you wish to report broken links, suggest new, different or better links to resources, changes to content, etc. please fee free to email us at nsrc@nsrc.org.

If you have other questions or coments about content in this document please feel free to send email to nsrc@nsrc.org as well.

Index