Monday, December 31, 2012

Using Rancid to backup your configs (Part 2)

In part 1 of this guide, we configured Rancid to backup your router configs.  Now we need to configure an SSL web interface for CVS.

The web interface I am using is ViewVC.  ViewVC is made up of a cgi script and a MySQL database backend.We'll also need to install RCS, and some python packages.  All of this is done as the root user.

    Download ViewVC and RCS:
    • wget ftp://ftp.cs.purdue.edu/pub/RCS/rcs-5.8.tar.gz
    • wget http://viewvc.tigris.org/files/documents/3330/49243/viewvc-1.1.17.tar.gz
          Download ez_setup to get the python packages we need.  Create its own directory to store the files.
          • mkdir ~/python
          • cd ~/python
          • wget http://peak.telecommunity.com/dist/ez_setup.py
          • Type: python ./ez_setup.py
          • Then run easy_install babel 
          • easy_install Genshi
          • easy_install Pygments
          • easy_install docutils
          • easy_install textile
          • easy_install python-mysqldb
          Unpack and install RCS:
          • cd ~
          • tar -zxvf rcs-5.8.tar.gz
          • cd rcs-5.8
          • ./configure
          • make
          • make install  
          Unpack and install ViewVC:
          • cd ~
          • tar -zxvf viewvc-1.1.17.tar.gz
          • cd viewvc-1.1.17
          • ./viewvc-install 
          Edit the ViewVC config file:
          • nano /usr/local/viewvc-1.1.17/viewvc.conf
          • Change the following to look like this, adding any missing options:
          • #cvs_roots = cvs: (Yes, it needs the leading #)
          • root_parents = /usr/local/rancid/var/CVS : cvs
          • rcs_path = /usr/local/bin/
          • address = <a href=mailto:youradmin@yourdomain.com>IT Support</a>
          • use_enscript = 1
          • enscript_path = /usr/bin/
          • use_highlight = 1
          • highlight_path = /usr/bin
          Move the viewvc.cgi file to the right place and set the proper permissions and attributes:
          • cp /usr/local/viewvc-1.1.17/bin/cgi/*.cgi /var/www/cgi-bin
          • chmod +x /var/www/cgi-bin/*.cgi
          • chown apache:apache /var/www/cgi-bin/*.cgi
          Create server key pair to enable ssl on the web server:
          • mkdir /etc/httpd/certificate
          • cd /etc/httpd/certificate
          • Generate the private key, when prompted supply a passphrase of your choosing: openssl genrsa -aes256 -out server.key 2048
          • Generate a certificate request based on the private key you just generated.  Fill in the prompts as appropriate: openssl req -new -key server.key -out server.csr
          • Submit the request to the CA of your choosing.  In my case, I am using a certificate issued by our own Microsoft Enterprise CA.
          • Put the issued certificate in the folder with the private key.  In my case, I called the file server.cer
          • If you leave the passphrase on the private key, you will need to enter it everytime the server is rebooted, or apache is restarted.  You can remove the passphrase with this command, entering your passphrase when prompted:
          • cp server.key server.key.org
            openssl rsa -in server.key.org -out server.key
          Configure apache and iptables for ViewVC:
          • nano /etc/httpd/conf/httpd.conf
          • Add the following to the config, I added mine at the bottom of the file:
          • NameVirtualHost *:443

            <VirtualHost *:443>
                DocumentRoot /var/www
                ServerName rancid.mydomain.com
                ScriptAlias /cgi-bin/ "/var/www/cgi-bin/"
                ScriptAlias /viewvc /var/www/cgi-bin/viewvc.cgi
                ScriptAlias /query /var/www/cgi-bin/query.cgi
                ServerSignature On
                SSLEngine on
                SSLProtocol all -SSLv2
                SSLCertificateFile /etc/httpd/certificate/server.cer
                SSLCertificateKeyFile /etc/httpd/certificate/server.key

            <Directory "/var/www/cgi-bin">
                AllowOverride None
                Options None
                Order allow,deny
            </Directory>

            </VirtualHost>
          • Edit iptables to allow apache to listen to incoming connections on port 443
          • nano /etc/sysconfig/iptables
          • Add the following line above the rule permitting inbound SSH:
          • -A INPUT -m state --state NEW -m tcp -p tcp --dport 443 -j ACCEPT
          • Restart apache service httpd restart
          Configure MySQL and create the ViewVC users as well as the ViewVC database.
          • Start MySQL: service mysqld start
          • Set the MySQL root user password, where yourpassword is a password of your choosing: mysqladmin -u root -password yourpassword
          • Create a ViewVC user and grant them permissions to create and use a database.
          • mysql -u root -p - Enter the password you just set when prompted.
          • CREATE USER 'youruser'@'localhost' IDENTIFIED BY 'yourpassword'; Where youruser and yourpassword are of your choosing.  You will need this username and password in a minute.
          • GRANT ALL PRIVILEGES ON *.* TO 'youruser'@'localhost' WITH GRANT OPTION;
          • FLUSH PRIVILEGES;
          • Exit MySQL quit
          • Now you need to create the database supplying the credentials you just created when prompted. Keep the default database name, ViewVC: /usr/local/viewvc-1.1.17/bin/make-database
          • We need a read-only database user:
          • mysql -u root -p
          • CREATE USER 'youruserRO'@'localhost' IDENTIFIED BY 'yourpassword';
          • GRANT SELECT ON ViewVC.* TO 'youruserRO'@'localhost' WITH GRANT OPTION;
          • FLUSH PRIVILEGES;
          • quit
          • Edit the ViewVC config file to tell it how to access the database:
          • nano /usr/local/viewvc-1.1.17/viewvc.conf
          • Add this to the [cvsdb] section of the config file:
          • enabled = 1
            host = localhost
            port = 3306
            database_name = ViewVC
            user = youruser
            passwd = yourpasssword
            readonly_user = youruserRO
            readonly_passwd = yourpassword
            row_limit = 1000 
          • Populate the ViewVC database with the info from CVS, type this all on one line: /usr/local/viewvc-1.1.17/bin/cvsdbadmin rebuild /usr/local/rancid/var/CVS/CVSROOT
          The last step is to ensure that MySQL and apache start when the server boots up.  The following will do just that.
          • chkconfig --levels 2345 mysqld on
            chkconfig --levels 2345 httpd on
            You should now have a working web interface for CVS.  Navigate to https://ipaddress or hostname/viewvc. You should see the repository groups specified in part 1 of this guide.  If you click on a group name, you should see configs, and if you click that, you should see the devices we added in part 1.

            Part 3 of this guide will cover configuring email notifications as well as having Rancid poll your devices when a change is made to their configuration.

              Sunday, December 30, 2012

              Using Rancid to backup your configs (Part 1)

              Where I work, budgets are tight.  It seems like we never have enough money to buy all the things we need.  That means looking at leveraging what we already own, and looking at free products.  We didn't have a means to back up our network gear configs.  I stumbled across a free, open source product called Rancid (Really Awesome New Cisco ConfIg Differ) while I was looking for a free Tacacs server.  Despite its name, Rancid is not limited to Cisco gear only, see this for details.  Rancid uses either CVS or SVN to store configs.  As the name suggests, it also diffs them and can show you the changes made.  Rancid can be configured to poll your gear on a schedule, poll your gear when changes are made, or both.  The folks at Shrubbery.net have links to some helpful config documents, some suggestions in their FAQ's and also maintain a Rancid mailing list.  The list archives are a good source of info, but from what I have seen so far, the list is currently very low activity. 

              I used the walkthrough done by Rhys Evans here as the basis for my config.  Anything beyond that I figured out from the FAQ's or from Google searches, which often pointed me to the list archives.  I had to make a few changes Rhys's guide to make my install work.  This guide assumes a basic knowledge of Linux.  You'll need to know how to edit config files at the command line.  I used CVS.  There are examples out there of how to configure Rancid to use SVN, if you'd rather do that.  Here's what I did.

              I started with a minimal CentOS 6.3 64-bit install in a VMware virtual machine.  I assigned the machine a static ip address during the setup process, and gave it a hostname of rancid.mydomain.com (substitute your domain name for mydomain.com).  All of this is done from an SSH connection to the server.

              We begin by installing the pre-requisites.

              • yum upgrade - because this is a new install, there will be updates to install.
              • yum install nano wget - minimal install does not include wget.  My preferred text editor is nano, so I installed that also.
              You'll need packages from the EPEL (Extra Packages for Enterprise Linux) repository, so we'll install it next.
              • Download it - wget http://mirrors.kernel.org/fedora-epel/6/i386/epel-release-6-8.noarch.rpm
              • Install it - rpm -ivh epel-release-6-8.noarch.rpm
              Now we'll install the rest of the pre-requisites. 
              • yum install expect cvs python httpd mysql mysql-server gcc make autoconf kernel-devel mod_python python-devel
              • yum groupinstall “Development Tools” diffutils
              • yum install php-common php-gd php-mcrypt php-pear php-pecl-memcache php-mysql php-xml MySQL-python mod_ssl
              Reboot your server.
              • shutdown -r now
              Once you log back in, you need to create a group and a user for Rancid to run under.  I used netadm and rancid for the group and user respectively as shown in Rhys's guide.  You can use whatever names you like, you'll need to substitute those names in the latter parts of the config.  I am using /usr/local/rancid as the install location for Rancid, you can use any directory you like, just substitute it where appropriate.
              • groupadd netadm
              • useradd -g netadm -c "Networking Backups" -d /usr/local/rancid rancid
              Now we need to create the directory to store the source.
              • mkdir /usr/local/rancid/tar
              • cd /usr/local/rancid/tar
              Download the latest version, 2.3.8 as of this writing.  Once downloaded, extract and install.
              • wget ftp://ftp.shrubbery.net/pub/rancid/rancid-2.3.8.tar.gz
              • tar -zxvf rancid-2.3.8.tar.gz
              • cd rancid-2.3.8
              • ./configure --prefix=/usr/local/rancid/
              • make install
              Copy sample .cloginrc file to the Rancid install and set security on the file.
              • cp cloginrc.sample /usr/local/rancid/.cloginrc
              • chmod 0640 /usr/local/rancid/.cloginrc
              Configure ownership and permissions on the Rancid installation.
              • chown -R rancid:netadm /usr/local/rancid/
              • chmod 770 /usr/local/rancid/
              Now you need to edit the Rancid config file.  The group(s) specified in this step are the categories the configurations will appear under.  I have mine separated out by type.
              • nano /usr/local/rancid/etc/rancid.conf
              • Look for LIST_OF_GROUPS and add group names.  Mine says: LIST_OF_GROUPS="Routers Firewalls Wireless_Controllers Load_Balancers"
              Next you'll need to edit the aliases file to add aliases for your groups you just configured.  In my case, I am the only one that needs mail notifications.  You can configure different groups of devices to go to different people or groups.
              • nano /etc/aliases
              • Add this to the file. You'll want to create the same groups for each group in your rancid.conf: 
              • rancid-admin-Routers: rancid-Routers
              • rancid-Routers: noc
              • Edit the noc alias as appropriate. noc: brian.gill@mydomain.com
              • Once you've saved the file, you need to let your server know about the changes. Type in newaliases.
              Time to add the Rancid info to CVS.
              • switch to the rancid user created earlier. su -rancid
              • /usr/local/rancid/bin/rancid-cvs - This sets up the Rancid info in CVS.  You should see output similar to:
                    No conflicts created by this import
                    cvs checkout: Updating Routers
                    cvs checkout: Updating Routers/configs
                    cvs add: scheduling file `router.db' for addition
                    cvs add: use 'cvs commit' to add this file permanently
                    RCS file: /usr/local/rancid//var/CVS/Routers/router.db,v
                    done
                    Checking in router.db;
                    /usr/local/rancid//var/CVS/Routers/router.db,v <-- router.db
                    initial revision: 1.1
                    done

              Now we can add crontab entries to schedule automatic polling of the devices and to cleanup the Rancid log files.  The following entries will have Rancid poll your devices once an hour 1 minute after the top of the hour and delete logs with a modified date older than 2 days at 11:50 pm every night.  You can adjust these to meet your needs.  The crontab command uses vi to edit the file.  In case you aren't familiar, you need to press i to enter insert mode.  Once you've added your entries hit escape to exit insert mode, and type :wq to save your changes and exit vi.
              • Type in crontab -e and add the following 2 lines
              • 1 * * * * /usr/local/rancid/bin/rancid-run #hourly router dump
                50 23 * * * /usr/bin/find /usr/local/rancid/var/logs -type f –mtime +2 -exec rm -rf {} \;

              Now we need to supply Rancid with a list of devices and the credentials to log in to those devices.  You'll need to edit the router.db files for each group you created earlier.
              • nano /usr/local/rancid/var/Routers/router.db
              • Add a line in the format "hostname or ip:type:up", e.g. 192.168.1.1:cisco:up
              • nano /usr/local/rancid/.cloginrc
              • You'll need to comment out the sample groups in the file by adding a # in front of the line.
              • Add device credentials
              • add user 192.168.1.1 user
              • add password 192.168.1.1 password enablepassword (NOTE: passwords that contain special characters will need to have those characters escaped.  For example, if your password is password!, the ! will need to be escaped, like this: password\!)
              • add method 192.168.1.1 ssh (Rancid defaults to telnet)
              Once you've saved your changes, you can test your Rancid install.
              • /usr/local/rancid/bin/rancid-run
              Once that completes, check the Rancid logs to verify that everything ranc correctly.
              • cd /usr/local/rancid/var/logs
              • ls (list the log files which are in the format Group.yyyymmdd.hhmmss)
              • cat logfilename
              You should see something like this:
                    [root@rancid logs]# cat Routers.20121228.200913
                    starting: Fri Dec 28 20:09:13 EST 2012
                    cvs add: scheduling file `192.168.1.1' for addition
                    cvs add: use 'cvs commit' to add this file permanently
                    RCS file: /usr/local/rancid/var/CVS/Routers/configs/192.168.1.1,v
                    done
                    Checking in 192.168.1.1;
                    /usr/local/rancid/var/CVS/Routers/configs/192.168.1.1,v  <--  192.168.1.1
                    initial revision: 1.1
                    done
                    Added 192.168.1.1

                    Trying to get all of the configs.
                    All routers sucessfully completed.

                    cvs diff: Diffing .
                    cvs diff: Diffing configs
                    cvs commit: Examining .
                    cvs commit: Examining configs
                    Checking in router.db;
                    /usr/local/rancid/var/CVS/Routers/router.db,v  <--  router.db
                    new revision: 1.2; previous revision: 1.1
                    done
                    Checking in configs/192.168.1.1;
                    /usr/local/rancid/var/CVS/Routers/configs/192.168.1.1,v  <--  192.168.1.1
                    new revision: 1.2; previous revision: 1.1
                    done
                 
                    ending: Fri Dec 28 20:09:30 EST 2012

              You should also see a file called 192.168.1.1 in the folder /usr/local/rancid/var/Routers/configs.  This will be the most current config download from your device.  You could use this to restore a device if needed.  Note: the passwords are all redacted from these files.

              Now you have a working Rancid installation.  In later parts of this guide, I'll walk through configuring a web interface for CVS, having Rancid pull your configs when a change is made, setting up e-mail alerts when changes are made.