Bash Script Toolbox
August 20, 2008
Some time ago I had very little linux/unix experience. An opportunity came up in a contract I was working on to work on a project that was to use almost exclusively unix bash scripting to verify software and hardware configurations on the numerous environments that we had. I volunteered and here is a compilation of some of what I learned. I'm sure you could find a more exhaustive list of commands and what they do in a bunch of books out there. However, if you use some of these simple commands creatively, you can build a pretty effective verification tool, verifying server configurations, networking firewalls and routing, deployed software, disk space, memory usage and more! Accompany that with the diff command and somekind of baseline for you system, maybe a little reporting web page to make the diff easier to read and you've got a groovy tool that will impress people.
1. 'man' command 'man' is short for 'manual'. By simply typing man 'someCommand' you will get the 'help' or the manual for that command. For example, 'man grep' will display to the console the grep manual.
2. 'env' command 'env' allows you to run a program in a modified environment. You modify the environment by supplying name=value properties for your 'modified environment'. Just typing 'env' in a console will display all the properties already set like PATH, JAVA_HOME, HOME, SYSTEMROOT, etc.
3. 'grep' command 'grep' is a type of search command that you can run in a linux/unix console or bash shell. It searches named input files (or standard input if no files are named) for lines containing a match to a given pattern. It works like 'grep pattern fileName'. For example, if I ran a grep on a given index.html file, looking for the string 'table', the command could look like 'grep table index.html'. You can also add a '-i' parameter to the command to make it ignore case in the search, or add a '-r' to do a recursive search of all files in a directory.
4. 'cat /proc/version' command Running 'cat /proc/version' outputs the content of the version file found in the /proc directory of the shell you are running. I found this useful because we had cygwin installed on our windows boxes, so I could run this command and determine whether I was on a windows box or a unix/linux box, and adjust my scripting logic accordingly. In the /proc directory there are also other interesting files to run 'cat' against. /proc/cpuinfo will give you detailed information on the processor(s) on your OS, including speed, vendor, model, count, etc. /proc/meminfo will give you detailed data on total memory available on your system and how it's configured and being used, including swap.
5. 'which' command 'which' will tell you which executable you're running for a particular command and where it's located. For example, on a windows box that has cygwin installed, you could likely find more than one zip command. Running 'which zip' will output the path that the cygwin console is using to run zip.exe.
6. 'ls' command Executing 'ls' lists a directories contents. I use 'ls -al' a lot, which lists all the contents of a directory, including the hidden files, along with owners, groups and perms for each file and folder in the directory. Sometimes some of the other options for this command are useful. For example, if you are writing a script to automatically delete older files in a directory, you could run 'ls -lrt' which would list the files in reverse order by modification time.
'ls' is great for verifying server configurations. Using '<' you can output the result of an 'ls someDir < myFile.txt' into a file called myFile.txt. Using that file as a baseline, you can come back at some later point in time, do an 'ls someDir < newFile.txt' and then use the 'diff' command (see further down) to compare the two files.
7. 'df' command 'df' command displays the file mounts on a system, where they are mounted, and space statistics for each one. This is handy for monitoring disk usage so you can make sure you don't run out of space. Related to this, I use 'du -sh *' often to help me determine which folder is taking up the most disk space in a directory.
8. 'diff' command Running the 'diff' command is essential if you want to build a verification system. 'diff' allows you to compare files line by line. This can be done in a number of different ways, the command is quite flexible. Check the manual for details. In writing a small verification system, I would output the data from various other commands I'd run remotely (using ssh) to files on my local box and them use 'diff' to compare them.
9. 'ssh' command 'ssh' is a program for logging into a remote machine and for executing commands on a remote machine. Writing a verification system is pretty hard without using this tool. The trick with it is to make sure your remote machine has an ssh service installed and listening on port 22, and getting keys and certs set up so you don't have to supply a password everytime you want to log in. Look up ssh-keygen for more information on setting up keys for the client and the server.
10. 'scp' command 'scp' is another invaluable tool in creating a verification application. 'scp' allows you to securely copy files and folders to/from one machine to another. This works over an ssl connection, so if you don't have ssl up and running, you can forget about this one.
11. 'netstat' command 'netstat' is a great command to allow you to see which ports a box has open and the state of those ports. It also shows the protocol that the port is using, as well as who is connected to it on both sides of the connection. Related to this (in my mind anyway) is another program called nmap, which can do a port scan of a remote box many different ways (some of them a bit sneaky). This is good to help you determine if you firewall has any holes.
I hope you enjoyed this little overview. If you have questions or comments, please send them to me @ perry.mckenzie@netfocusconsulting.com.