Replace Backquotes with Brackets for Command Substitution

If there is a script which has got backquotes around some commands to assign them to a variable, it can be useful to replace the backquotes with the easier to read but less portable $(...).
ie if you have the following piece of code:
currdir=`pwd`
and want to replace it with
currdir=$(pwd)
then, in Notepad++ use the following in the find and replace menu:
Find: `(.*)`
Replace: \$\($1\)

Stress Testing A Linux Box

Using Fedora's Stress Command

To stress test the machine, we should install the stress command.

Login as root

If the compile tools are not installed then install them:
yum install gcc gcc-c++ autoconf automake Then install the command itself: wget http://pkgs.fedoraproject.org/repo/pkgs/stress/stress1.0.4.tar.gz/a607afa695a511765b40993a64c6e2f4/stress-1.0.4.tar.gz
tar zxvf stress-1.0.4.tar.gz
cd stress-1.0.4
./configure
make
make install
To do the stress tests themselves:

Stress CPU for 10 minutes: stress –c 5 –t 1200s Stress Memory for 10 minutes stress –m 10 –t 1200s Stress disk io for 10 minutes stress –d 10 –t 1200s The first command will push CPU to 500%. It only works on one core, so it may need to run several time simultaneously, depending on how many cores there are. The second command produces 10 spinning process, and the third 10 spinning disk write processes.

The commands can be combined (-c 5 –m 10 –d 10 –t 1200s).

Using A Script

But what if the system is locked down and the stress command cannot be installed? It is possible to perform the stress test using scripting instead. The following scripts do the job.

This script should hammer the CPU load. It runs a huge number of awk calculations as several background processes
#!/bin/bash
if [ $1 ]; then
        NUM_PROC=$1
else
        NUM_PROC=10
fi

uptime

for i in $(seq 1 $NUM_PROC ); do
   awk 'BEGIN {for(i=0;i<10000;i++)for(j=0;j<10000;j++);}' &
   pidarr[$i]=$!
   echo ${pidarr[$i]}
done

ps -fp "${pidarr[*]}"

uptime
We can use dd and a variation of the above script to thrash the disk IO:
#!/bin/bash
if [ $1 ]; then
        NUM_PROC=$1
else
        NUM_PROC=10
fi

for i in $(seq 1 $NUM_PROC ); do
   dd if=/dev/sda of=/dev/null &
   pidarr[$i]=$!
   echo ${pidarr[$i]}
done

ps -fp "${pidarr[*]}"
And for memory, we can use another variation:
#!/bin/bash
if [ $1 ]; then
        NUM_PROC=$1
else
        NUM_PROC=2
fi

BS=$(free | grep "^Mem" | awk '{print $2}' | head -1)

for i in $(seq 1 $NUM_PROC ); do
   dd if=/dev/urandom bs=$BS of=/dev/null count=1050 &
   pidarr[$i]=$!
   echo ${pidarr[$i]}
done

ps -fp "${pidarr[*]}"
This caches huge blocks of random numbers (using a block size of the total amount of used space in the system). Doing it a couple of times simultaneously will do the trick.

Windows New Lines Cause \r Scripting Errors

Using notepad++ to write Linux or Unix shellscripts creates scripts where each line ends in Windows format <line feed><carriage return> rather than Unix format <carriage return> new lines.

This will typically throw up an error when trying to run a script, for example:

$ ./dbusers.sh -s UC1 -rUC2 user1 user2 user3
./dbusers.sh: line 33: $'\r': command not found
./dbusers.sh: line 36: syntax error near unexpected token `$'in\r''
'/dbusers.sh: line 36: `   case $value in

This can be fixed in notepad++ in the Settings menu:

Settings / Preferences / New Document and change the Format (Line ending) to Unix/OSX.

This only applies to new documents, so close any shell scripts which are open for editing in Notepad, then convert them in Linux to get rid of the extra line feeds. This can be done either using a sed command, or the dos2unix command. Not every distribution has the dos2unix command but sed is universal.

sed -i 's/\r\n/\n/g' <script>

This simple sed command searches for <linefeed><carriage return> (\r\n) and replaces it with a simple <carriage return> (\n).

The script can now be loaded back into notepad++ where it will work correctly.

Create a Solaris Zone Using a Here Document

Creating a new zone in Solaris takes several steps and is a laborious task, especially if several zones are to be created at once. The process can be sped up by setting up the details of the new zone in variables, then sending all of the commands into zonecfg using a here document.

The code below assumes two network interfaces, and that the zones are stored in /zones. It should be customised as required to add filesystems, memory capping etc (see Further Options, below).

First set up the zone name ($NAME), the number of cores ($CORES), the two network interfaces ($IF1 and $IF2)[1] , and IP addresses ($ADD1 and $ADD2):
NAME=amypond
CORES=2
IF1=hre228001
IF2=hre229001
ADD1=172.21.138.107
ADD2=172.21.139.107

Copy and paste the following to the command line (all in one go):
zonecfg -z $NAME <<EOF
create
set zonepath=/zones/$NAME
set autoboot=true
set bootargs="-m verbose"
add dedicated-cpu
set ncpus=$CORES
end
add net
set physical=$IF1
set address=$ADD1
end
add net
set physical=$IF2
set address=$ADD2
end
info
verify
commit
EOF
zoneadm -z $NAME install
zoneadm -z $NAME ready
zoneadm -z $NAME boot


Further Options

To add a CD-ROM, add the following lines immediately after an end command above:
add fs
set dir=/mnt
set special=/cdrom
set type=lofs
add options [ro,nodevices]
end

To mount a filesystem from the global zone, amend and add the following lines immediately after an end command above:
add inherit-pkg-dir
set dir=/opt/sfw
end

To add memory capping, amend and add the following lines immediately after an end command above:
add capped-memory
set physical=50m
set swap=100m
set locked=30m
end

Further configuration can be done via the console:
zlogin -C $NAME


[1] These can be obtained from an ifconfig -a on any of the other zones on the host. ⤴

Always Use Sudo

This is part of a series of articles on Red Hat Server Hardening.

Nobody - not even administrators - should ever log in as root unless absolutely necessary. If an administrator needs to run a command with root privileges, they should use sudo.

The sudo tool allows ordinary users to have limited root level administrative access for certain tasks. This allows users to perform specific superuser operations without allowing them full superuser status.

To use sudo to run a command, precede it with the sudo command:
sudo date

The first time a user issues a sudo command during a login session, they will be prompted to enter the administrative password.

The accounts capable of using sudo are specified in the /etc/sudoers file, which is edited with the visudo utility. This file lists users and the commands they can run, along with the password for access (unless the NOPASSWD option is set, then users will not need a password).

A /etc/sudoers entry has the format
     user     host=command

user	The name of the user being granted access
host	A host on the network. For all hosts, use ALL.
command	A list of one or more commands, qualified by options such as whether the password is required. For all commands, use ALL.

So, for example, to give user paul full root-level access to all commands on all hosts:
paul   ALL = ALL

To run as another user, instead of as root, place the alternative user in parentheses before the command. For example, to allow user paul to run as user ringo on the beatle host:
paul   beatle = (ringo) ALL

The command may have an option associated with it. Possible options are:

NOPASSWD / PASSWD	Determines whether or not the user will require a password to run the command.
NOEXEC / EXEC	If sudo has been compiled with noexec support, this determines whether or not an executable will be allowed to run further commands itself.
SETENV / NOSETENV	Determines whether or not users are allowed to override environment variables with the sudo -e command.
LOG_INPUT / NOLOG_INPUT	Determines whether or not the input to the command is written to the log file.
LOG_OUTPUT / NOLOG_OUTPUT	Determines whether or not the output from the command is written to the log file.

By default, relevant logs are written to /var/log/secure.

Therefore, to allow user paul to run the kill command on beatle with a password, but to run the lprm command without a password:
paul   beatle = PASSWD: /usr/bin/kill, NOPASSWD: /usr/bin/lprm

A user can see what commands he or she can run by running: sudo -l

Installing Apache on Red Hat 6

Like most other things in Red Hat, Apache can be quickly and easily set up using the package manager. The process is simple. The only thing to remember is that the package and the service are not called Apache, but httpd.

Install Apache

Open a root shell
su -
Install the Apache web server:
yum -y install httpd
Configure the system to start Apache at boot
chkconfig httpd on
Start the Apache web server:
service httpd start

Test the Apache Installation

To test, copy and paste the following into /var/www/html/index.html:
<html>
<head>
<title>Dougie&#39;s Linux Hints Test Web Page</title>
</head>
<body>
This is a test Web Page.
</body>
</html>
Save the file. Then, still from the root shell, run the command:
elinks http://localhost
This will access the webpage created above, proving the web server is up and running.

Remove KDE or GNOME From Linux

X Windows desktops like KDE or GNOME are not required on a server, and waste valuable resources. These should therefore be removed:
yum groupremove “X Window System”

This will remove around 100-150 packages from the server.

Doing this will prevent an intruder from starting an X-Windows session on the server by typing startx at the shell prompt.

Installation of X-Windows can also be completely prevented during initial system installation.

Partitioning and Mounting Disks For Security

This is part of a series of articles on Red Hat Server Hardening.

By creating different partitions, data can be separated and grouped. When an unexpected accident occurs, only data on that partition will be damaged, while the data on other partitions will survive.

During the initial installation, mount filesystems with user-writeable directories, such as the following, on separate partitions:

• /usr
• /home
• /var
• /var/tmp
• /tmp

Apache and FTP server root directories should also be mounted on separate partitions.

To limit user access to filesystems, add the mount options from the following table to the filesystems configuration in /etc/fstab . The defaults option is equal to rw,suid,dev,exec,auto,nouser,async.

Option	Description
noexec	Prevents the execution of binaries (although scripts will not be prevented from running).
nosuid	Prevents the setuid bit from having an effect.
nodev	Prevents the use of device files.

Modify the /boot directory to be read only (ro). This reduces the risk of unauthorized modification of critical boot files.

For example, to modify the /etc/fstab entry to limit user access on /dev/sda5 (ftp server root directory):

Find the line that reads:
/dev/sda5  /ftpdata          ext3    defaults 1 2
And change it to:
/dev/sda5  /ftpdata          ext3    defaults,nosuid,nodev,noexec 1 2

Physical Security of Linux Servers

This is part of a series of articles on Red Hat Server Hardening.

The physical security of a Linux Server is the first line of defence against attackers. Who has direct physical access to the server, and should they? Should the server be secured out of hours or while you are away?

BIOS Security

The server's BIOS should be configured to disable booting from CDs/DVDs, floppies, and external devices. If possible, a password should be set to protect these settings.

GRUB Boot Loader Security

The GRUB boot loader should be password protected. The primary reasons for doing this are:

1. Preventing Access to Single User Mode - If attackers can boot the system into single user mode, they are logged in automatically as root without being prompted for the root password.
2. Preventing Access to the GRUB Console - If the machine uses GRUB as its boot loader, an attacker can use the use the GRUB editor interface to change its configuration or to gather information using the cat command.

By generating an MD5 hashed password string and including it in the GRUB configuration file, GRUB can be configured to address these issues.

To do this, first decide on a password, then open a shell prompt, log in as root, and type:
/sbin/grub-md5-crypt
This will ask for a password to be entered. Type the new GRUB password twice. This returns an MD5 hash of the password.

Next, edit the GRUB configuration file /boot/grub/grub.conf.

Open the file and below the timeout line in the general section of the document, add the following line:
password --md5 <password-hash>
Replace <password-hash> with the value returned by /sbin/grub-md5-crypt.

Remove the line that contains the hiddenmenu parameter, and save the changes.

The next time the system boots, the GRUB menu does not allow access to the editor or command interface without first pressing p followed by the GRUB password.

Pluggable Authentication Modules (PAM) Overview

Pluggable Authentication Modules (PAM) is an authentication service that lets a system determine the method of authentication to be performed for users. Traditionally in Linux, authentication has been performed by looking up passwords - the login process looks up the user's password in the password file and verifies it against what the user has entered.

With PAM, user's authentication requests are directed to PAM, which in turn uses a specified method to authenticate the user. This could be a simple password lookup, or it could be a request to an LDAP server, or some other method of authentication. Authentication is centralized and controlled by a specific service, PAM. The actual authentication procedures can be dynamically configured by the system administrator.

There are two types of PAM files: Configuration Files and Modules. Modules carry out the authentication process. These vary according to the kind of authentication needed. An administrator can add or replace modules by simply changing the PAM configuration files

PAM Configuration Files

PAM Configuration files are kept in the /etc/pam.d directory. PAM uses different configuration files for different services that request authentication.

Note:- If the /etc/pam.d/ directory does not exist, PAM will look for the /etc/pam.conf file instead. This is for historical reasons only.

The /etc/pam.d/ directory contains a configuration file for each PAM-aware application or service. The configuration file has the same name as the service to which it controls access. PAM-aware applications and services are responsible for defining their own PAM configuration files. For example, the /etc/pam.d/login PAM configuration file is installed by the login application.

Each PAM configuration file contains a group of directives formatted as follows:
<module interface>  <control flag>   <module name>   <module arguments>


PAM Module Interface

There are four types of PAM module interface, each of which corresponds to a different phase of the authorization process:

Module Interface	Definition
auth	This module interface authenticates use. For example, it requests and verifies the validity of a password. Modules with this interface can also set credentials, such as group memberships or Kerberos tickets.
account	This module interface verifies that access is allowed. For example, it may check if a user account has expired or if a user is allowed to log in at a particular time of day.
password	This module interface is used for changing user passwords.
session	This module interface configures and manages user sessions. Modules with this interface can also perform additional tasks that are needed to allow access, like mounting a user's home directory and making the user's mailbox available.

PAM Control Flag

All PAM modules return a status of success or fail. The Control Flag determines what PAM does with that status, and how important the success or failure of the module is to the authentication process. 

The control flag will usually have one of the following five values:

Control Flag	Definition
required	The module result must be successful for authentication to continue. If the test fails at this point, the user is not notified until the results of all module tests that reference that interface are complete.
requisite	The module result must be successful for authentication to continue. However, if a test fails at this point, the user is notified immediately with a message reflecting the first failed required or requisite module test.
sufficient	The module result is ignored if it fails. However, if the result of a module flagged sufficient is successful and no previous modules flagged required have failed, then no other results are required and the user is authenticated to the service.
optional	The module result is ignored. A module flagged as optional only becomes necessary for successful authentication when no other modules reference the interface.
include	Unlike the other controls, this does not relate to how the module result is handled. This flag pulls in all lines in the configuration file which match the given parameter and appends them as an argument to the module.

PAM Module Name and PAM Module Parameters

This is simply the name of the module, followed any parameters that are required to run it.

As previously noted, the authentication process is split into four phases - auth, account, password and session - specified in the configuration file by the Module Interface parameter. Each phase may consist of zero, one or several modules. The overall success or failure of each phase is determined by the combination of the control flags.

PAM Modules

PAM Modules are located in the /lib/security directory. Each returns either a success or failure.

Most of the modules and configuration files included by default with PAM have their own manpages.

It is also possible to write modules from scratch. Documentation on writing modules is included in the /usr/share/doc/pam-<version#> directory.

Password Control

This is part of a series of articles on Red Hat Server Hardening.

Once a user account has been created, the user's access to it can be controlled.

Locking User Accounts

The passwd command can be used to lock and unlock a user's account.
passwd -l username

will lock an account, and
passwd -u username

will unlock it.

Lock Any Accounts With No Password Set

Login as root, and enter the following command
awk -F: '($2 == "") {print $1}' /etc/shadow
This will produce a list of all accounts that have no password set.

Note:- Some systems still store the password in the /etc/passwd file (they shouldn't, but they do!). If this is the case, use /etc/passwd instead of /etc/shadow above.

The lock all empty password accounts:
passwd -l <Account Name>

Change Password Expiration Limits

The chage command displays password expiration details along with last password change date.

To view any existing user’s password ageing information such as expiry date and time, use the following command:
chage -l username

To change password ageing of any user, use the following command.
chage -M 60 -m 7 -W 7 <username>

This sets a maximum age of 60 days, a minimum age of 7 days, and allows the user 7 days warning before the password expires.

Options for chage are as follows:

Option	Description
-m	Minimum number of days a user must go before being able to change their password
-M	Maximum number of days a user can go without changing their password
-d	The last day the password was changed
-E	Specific expiration date for a password, date in format YYYY-MM-DD or MM/DD/YYYY
-I	Allowable account inactivity period (in days) after which password will expire
-W	Warning period. The number of days before expiration when the user will be sent a warning message
-l	Display current expiration controls

Prevent Reuse of Old Passwords

Users should be prevented from reusing old passwords. Old passwords are stored in /etc/security/opasswd. This file must be created before switching on password history, otherwise all user password updates will fail because the pam_unix[1] module will constantly be returning errors from the password history code due to the file being missing.

After creating the file, change the permissions to keep it secure:
touch /etc/security/opasswd
chown root:root /etc/security/opasswd
chmod 600 /etc/security/opasswd

Open the /etc/pam.d/system-auth file and add the following line to the auth section:
auth        sufficient    pam_unix.so likeauth nullok

Add the following line to the password section:
password   sufficient    pam_unix.so nullok use_authtok md5 shadow remember=5

This will prevent a user from re-using any of their last 5 passwords.

Force Users to Set Strong Passwords

A number of users use soft or weak passwords and their password might be hacked with a dictionary based or brute-force attacks. The pam_cracklib module, available in the PAM (Pluggable Authentication Modules) module stack, forces users to set strong passwords.

Open the /etc/pam.d/system-auth file and amend or add the entry for pam_cracklib.so:
/lib/security/$ISA/pam_cracklib.so retry=3 minlen=8 lcredit=-1 ucredit=-2 dcredit=-2 ocredit=-1

This sets the number of times the password may be entered before it fails to 3. The password must be at least 8 characters long, and contain at least 1 lower case character, 2 upper case characters, 1 digit and one other.

[1] qv PAM Overview ↩

Configure or Disable SSH

This is part of a series of articles on Red Hat Server Hardening.

SSH is the usual means of accessing and interacting with a server. Unless the keyboard and monitor are physically connected directly to the server, then access will most likely be via SSH. However, if SSH is not required, then disable it:

/sbin/chkconfig sshd off

The default SSH configuration means that automated cracking scripts and bots trying to break into a server know exactly where to go and what to do. They know the name of the root account, and they know they can SSH onto the server on port 22. The first line of defence is therefore to disable direct root access via SSH, and change the access port.

Changes to SSH are made via the SSH configuration file: /etc/ssh/sshd_config.

Prevent direct root login from SSH

In the config file, find the line that reads:
PermitRootLogin yes

Change yes to no. This prevents users from logging into the server as root via SSH. This adds an extra layer of security. Any hacker trying to get into root will have to get in as a normal user first, then try to access root from there. Warning: Make sure you have a regular user account first before doing this, otherwise you will not be able to access root.

Limit SSH access to a subset of users

If possible, limit SSH access to a subset of users. If there are many user accounts on the server, but only a few need to log into it via SSH, then doing this is a worthwhile exercise.This makes a hacker's job even more difficult because they will have to guess the both the name of an authorised user, and their password.

The AllowUsers parameter is not included in /etc/ssh/sshd_config by default, so it will need to be added:
AllowUsers john paul george ringo

Alternatively, create a group called sshusers and only add the users that need remote access:
groupadd sshusers
usermod -aG sshusers john
usermod -aG sshusers paul
usermod -aG sshusers george
usermod -aG sshusers ringo

Then, add the following line to /etc/ssh/sshd_config:
AllowGroups sshusers
Note:- The AllowUsers and AllowGroups parameters are mutually incompatible, with AllowUsers taking precedent.

Change the default SSH port

Change the default SSH port number of 22 to some other higher level port number.

Note:- The Internet Assigned Numbers Authority (IANA) is responsible for the global coordination of the DNS Root, IP addressing, and other Internet protocol resources. It is good practice to follow their port assignment guidelines. Having said that, port numbers are divided into three ranges: Well Known Ports, Registered Ports, and Dynamic and/or Private Ports. The Well Known Ports are those from 0 through 1023 and SHOULD NOT be used. Registered Ports are those from 1024 through 49151 should also be avoided too. Dynamic and/or Private Ports are those from 49152 through 65535 and can be used.

Choose an appropriate port, also making sure it not currently used on the system, and update the following line in /etc/ssh/sshd_config:
Port <New Port Number>
Make sure, obviously, that anyone who needs to ssh onto the server knows the correct port number. To log in, they will need to add -p <new port number> to the end of the ssh command.

Once all of the necessary changes have been made to /etc/ssh/sshd_config, restart the service so that these changes take effect.
service sshd restart

Determine What Country A Website User Is In

One of my other concerns is a website which sells MP3s of original children's music.

These sell all over the world, so it is important that prices are displayed in local currency if possible. The easiest way to do that is by IP address. When the site was originally created, it was built using HTML and javascript, with only a very small amount of PHP to access some tables of IP ranges and the appropriate country. When the website was rebuilt around a MySQL database, this method of identifying the user's country remained, because it worked.

But these things are ever evolving, and the IP ranges I was using quickly went out of date. I have been aware for some time that I need to find a way on automatically making sure that the IP ranges are up to date.

After a bit of looking around, I have found that an up to date CSV file of IP ranges can be downloaded from http://software77.net/geo-ip/. The plan is to download the CSV file automatically on a regular basis, then rebuild the existing PHP arrays.

This can all be accomplished with a simple bash script, run from cron once a week. The IP file is donationware, so I have arranged a regular $5 monthly payment for the privilege which seems fair, I think.

The first step is to download the file. This can be accomplished with a simple wget:
wget http://software77.net/geo-ip/?DL=1 -O ./IpToCountry.csv.gz


The file is compressed, so it needs to be uncompressed. I also extract only the fields that are required. The fields on the incoming file are IP From, IP To, Registry, Assigned, 2-Letter Country Code, 3-Letter Country Code and Country. I only need IP From, IP To and the 2-Letter Country Code.

Both of these tasks can be accomplished in a single command line:
gunzip -c IpToCountry.csv.gz  | awk -F, '!/^#/ {gsub(/"/, "", $0);print $1, $2, $5}'  > ./IpToCountry.csv


This uncompresses the file, strips off all of the comment lines, removes any inverted commas, extracts the necessary fields and creates a stripped down, custom built file that can be used as required.

Next comes the rebuilding of the PHP IP arrays. These consist of 256 numbered files, starting at 0.php through to 255.php. Each of these contains a PHP script defining an array of ranges of IP addresses and the country that they belong to. The appropriate PHP script is included at run time, dependent on the first segment of the IP address.

The first step is to write the header for each script. This opens the php tag and starts to declare the array:
<?php
//-
$ranges=Array(


This  header is identical for all 256 files.

Next, the arrays themselves must be created. These can be converted directly from each line in the CSV file in the format:
"IP From" => array("IP To","2-Digit Country Code")


IP From and IP To are both formatted as a 32-bit integer, rather than the 4 segments traditionally recognised as an IP address. To convert the 4-segment IP address to the 32 bit integer it represents, multiply each segment by increasing factors of 256.

eg for IP address "1.2.3.4"
     (1*256*256*256)+(2*256*256)+(3*256)+4

Similarly, the IP segments can be determined by reversing the process. In this instance, we only need the first of the 4 segments, to determine which file we are writing to. This can be achieved by dividing IP From by (256*256*256), and using only the integer returned.

Finally, a footer is added to the end of all of the files. Like the header, this is identical for all 256 files, simply closing off the array and closing the php tag:
);
?>


The finished script looks like:
#!/bin/bash
#######################################################
#
# csv2IP.sh
#
# Douglas Milne 2 June 2014
#
# Download a csv file of IP address ranges for countries
# and convert to php arrays
#
#######################################################

# create a temporary directory to build the files
# Files are built here, then moved to the correct location on completion
# This minimizes the amount of time the files are unavailable as recreating them can take several minutes
mkdir ~/iptemp 2>/dev/null
cd  ~/iptemp

# Download the csv file
wget http://software77.net/geo-ip/?DL=1 -O ./IpToCountry.csv.gz >/dev/null 2>&1
status=$?
if (( status != 0 ))
then
   echo "Error downloading csv file"
   exit 2
fi
# Uncompress the CSV file and select only the To, From and Country columns
gunzip -c IpToCountry.csv.gz  | awk -F, '!/^#/ {gsub(/"/, "", $0);print $1, $2, $5}'  > ./IpToCountry.csv

# Create a new .php file for the first digit of possible ip addresses, ie 0-255, and write a header to it
for ((i=0; i<=255; i++))
do
   echo -e "<?php\n//-\n\$ranges=Array(" > $i.php
done

# Add the IP ranges as specified in the CSV file to the php files.
# The first digit of each ip address, and therefore the file to write to,
# is determined by the integer result of dividing the address by 16777216
cat IpToCountry.csv | awk '{print $1,$2,$3}' | while
   read ipFrom ipTo Country
do
   (( ipmsb = ipFrom / 16777216 ))
   echo -e "\"$ipFrom\" => array(\"$ipTo\",\"$Country\")," >> $ipmsb.php
done

# Add a footer to each of the php files
# and move the files to the correct location.
for ((i=0; i<=255; i++))
do
   echo -e ");\n?>" >> $i.php
   mv $i.php ~/ip_files
done


By way of example of the output, the file for all IP addresses from 45.0.0.0 to 45.255.255.255 is
$ cat 45.php
<?php
//-
$ranges=Array(
"754974720" => array("755105791","US"),
"757071872" => array("759169023","ZZ"),
"765460480" => array("767557631","UY"),
);
?>
This tells us that some of these are in the US, some are reserved and some are in Uruguay. Most of the output files are considerably larger than this, and some are smaller.

The script is run from cron a couple of times a week. Software77 request that a time other than right on the hour is chosen for download, so that everybody isn't tryng to download at once. It's worth reading the comments in the CSV file and on their website, because breaking the rules can result in a barring.

So how does a webpage make use of this information? The following PHP function takes the IP address of the client
function iptocountry($ip) {
    $numbers = preg_split( "/\./", $ip);  
    include("ip_files/".$numbers[0].".php");
    $code=($numbers[0] * 16777216) + ($numbers[1] * 65536) + ($numbers[2] * 256) + ($numbers[3]);  
    foreach($ranges as $key => $value){
        if($key<=$code){
            if($ranges[$key][0]>=$code){$two_letter_country_code=$ranges[$key][1];break;}
            }
    }
    return $two_letter_country_code;
}


This function converts the IP address into a 32-bit integer, includes the appropriate array file, then searches through the array until it finds the range that contains the 32-bit integer
This can be called using the "REMOTE_ADDR" entry in the $_SERVER array.

$two_letter_country_code=iptocountry($_SERVER['REMOTE_ADDR']);

Display the Amount of Free Space on Each Filesystem

Another old script that I came across. This one displays the amount of free space on each filesystem on a server, and displays a warning if it falls below a certain level. The output can be displayed in html format for sending as an email (via the htmmail script).

The script was written for AIX - one day I might adapt it for Linux, which has the very useful free command, but without the bells and whistle of the script below.

Save the file as fsfree.ksh, and run it from cron on a daily basis for reports on filesystem usage.

#!/bin/ksh
##################################################################
#
# fsfree.ksh
#
# Purpose: To display free space on each filesystem on a server
#          and to warn if it falls below a certain percentage.
#
# Return status:
#            0  All filesystem space is above the limit
#            1  One or more filesystems are below the limit
#            2  The script has not been correctly executed
#
# Syntax:  fsfree.ksh [ -a ] [ -g | -m | -p ] [ -h ] [-q ]
#                   [ -l limit | -f file ]
#
#          Flags
#           -a  Displays all filesystems
#
#               By default, the script will display only those
#               filesystems which are below the limit.
#
#           -f  Use the specified file to define the limits
#               for the individual filesystems
#
#               The format of the file should be:
#                    Filesystem   Limit
#
#               An initial file may be generated using the
#               free command as follows:
#
#                    fsfree.ksh -pal 0 > limitfile.dat
#
#               The file should then be edited as appropriate
#     
#           -g  Displays total and free space in Gigabytes
#
#               Space is displayed in Kilobytes by default
#
#           -h  Output is in html format
#
#           -l  When the percentage free on the filesystem falls
#               below the value of 'limit', the filesystem will be
#               highlighted on the output.
#
#               The default limit is 10%
#
#           -m  Displays total and free space in Megabytes
#
#               Space is displayed in Kilobytes by default
#
#           -p  Total and free space is not displayed
#
#           -q  Quiet mode. No output is displayed.
#
# Author:  Douglas Milne
# Date:    13th May 2008
#
##################################################################
#
# Version 1.0    Initial Release
#
##################################################################

##################################################################
#
# VARIABLES Section
#
#
# Set default variables
#
dispall=false
htmflag=false
pcntonly=false
quietmode=false
limitsource=default
divisor=1
stype="Kb"
scale=0
dlimit=10
returnstat=0
errstring=""

# Parse the flags and change default variables accordingly
while getopts :ahpgml:f:q value
do
   case $value in
      a) dispall=true
         ;;
      h) htmflag=true
         ;;
      p) pcntonly=true
         ;;
      g) divisor=1048576
         stype="Gb"
         scale=3
         ;;
      m) divisor=1024
         stype="Mb"
         scale=3
         ;;
      l) if $(echo $OPTARG | grep -q [0-9])
         then
            dlimit=$OPTARG
            limitsource=limit
         else
            errstring="Limit must be a number between 1 and 100"
            dlimit=""
         fi
         ;;
      f) limitfile=$OPTARG
         limitsource=file
         if [ ! -f $limitfile ]
         then
            errstring="The file $limitfile does not exist"
         fi
         ;;
      q) quietmode=true
         ;;
     \?) errstring="$0: unknown option $OPTARG"
         ;;
   esac
   if [ "$value" == ":" ]
   then
      case $OPTARG in
         l) errstring="Limit argument must be included and must be a number between 1 and 100"
            ;;
         f) errstring="The name of the file may not be blank"
            ;;
     esac
   fi
done
    
shift $(expr $OPTIND - 1)

##################################################################
#
# FUNCTIONS Section
#

syntaxerr ()
##################################################################
#
#  Syntax:      syntaxerr error
#
#  error        Text describing the error
#
#  Writes a copy of the error and the syntax of the script
#  to stderr.
#
##################################################################
{
   err=$1
   echo "$err\n\nSyntax:  fsfree.ksh [ -a ] [ -g | -m | -p ] [ -h ] [-q ]\n                    [ -l limit | -f file ]\n" >&2
   exit 2
}

limitdef ()
##################################################################
#
#  Syntax:      limitdef filesystem
#
#  filesystem   A filesystem name
#
#  The function will return the limit required for the given
#  filesystem. This may be the default limit, the limit defined
#  from the command line, or the limit defined in the limitfile
#
##################################################################
{
   fs=$1
   case $limitsource in
      default) echo $dlimit
               ;;
        limit) echo $dlimit
               ;;
         file) lim=$(grep ^/ $limitfile | grep "^$fs " | sort | head -1 | awk '{print $2}' | sed -e's/\%//g')
               if [ "$lim" == "" ]
               then
                  echo $dlimit
               else
                  echo $lim
               fi
               ;;
   esac
}

display ()
##################################################################
#
#  Syntax:      display text
#
#  text         The text to display
#
#  The function will print the text to the standard output, but
#  only if the script is not running in quiet mode
#
##################################################################
{
   text=$1
   if [ $quietmode == false ]
   then
      echo "$text"
   fi
}

##################################################################
#
# MAIN Section
#

# Error checking

if [ "$errstring" != "" ]
then
   syntaxerr "$errstring"
fi

# Display the header
if [ $htmflag == true ]
then

# If html format is required, then create the head and body of the
# html page.
#
# A table is used to format the output. Define the table, and output
# the header.
#
   display "<html><head>"

# Instead of below, it might be useful to cat in a standard css file instead
   display "<style> th{font:10pt Arial;font-weight:bold;color:blue;background-color:LightBlue;padding-left:10px;padding-right:10px;} p{font:10pt Arial;} li{font:10ptArial;} td{font:10pt Arial;padding-left:10px;padding-right:10px;} .normal{color:black;} .alert{color:red;font-weight:bold;} .code{font:10pt Courier;color:black;}</style>"


   display "</head><body>"
   display "<p class=\"normal\">$(hostname) filesystem status at $(date)"
   if [ $limitsource != "file" ]
   then
      display "<br><br>Limit is $dlimit%"
   fi
   display "</p>"
else
   display "\n$(hostname) filesystem status at $(date)\n"
   if [ $limitsource != "file" ]
   then
      display "Limit is $dlimit%\n"
   fi
fi

if [ $htmflag == true ]
then

#
# A table is used to format the output. Define the table, and output
# the header.
#
   display "<table>"
   display "<tr>"
   display "<th style=\"text-align: left\">Mount Point</th>"
   if [ $pcntonly = false ]
   then
#
# Only output the total Free and Total Size headers if the -p flag
# has not been used
#
      display "<th style=\"text-align: right\">Total<br> Free ($stype)</th>"
      display "<th style=\"text-align: right\">Total<br> Size ($stype)</th>"
   fi
   display "<th style=\"text-align: right\">%age<br> Free</th>"
   if [ $limitsource == "file" ]
   then
      display "<th style=\"text-align: right\">%age<br>Limit</th>"
   fi
   display "</tr>"
else


# If html format is not required, then output the header as standard
   boldon=$(tput smso)
   boldoff=$(tput rmso)
   typeset -L20 mntpnt
   typeset -R11 free total
   typeset -R3 pcnt limit
   display "Mounted on          \c"
   if [ $pcntonly = false ]
   then
#
# Only output the total Free and Total Size headers if the -p flag
# has not been used
#
      display "    Free ($stype)   Total ($stype)\c"
   fi
   display " %Free\c"
   if [ $limitsource == "file" ]
   then
      display " Limit"
   else
      display
   fi
fi

df -k > /tmp/freedf

#
# For each filesystem in turn, read the mountpoint, total free
# space and total size
#
cat /tmp/freedf | tail +2 | awk '{print $7,$3,$2}' | while
read mntpnt free total
do
# If numeric values have been read, then calculate the output
   if $( echo $free | grep -q [0-9])
   then

# 1) Calculate the percentage of free space
      let pcnt=free\*100/total
# 2) Calculate the free and total space as Kb, Mb or Gb as required
      free=$(echo "scale=$scale; $free / $divisor" | bc)
      total=$(echo "scale=$scale; $total / $divisor" | bc)

      limit=$(limitdef $mntpnt)
# If the percentage free is less than the limit allowed, then switch on
# highlighting. For html output, use the alert class. For standard output,
# use bold type. Set the return status to 1.
      if (( pcnt <= limit ))
      then
         if [ $htmflag == true ]
         then
            class=alert
         else
            display "$boldon\c"
         fi
         returnstat=1
      else

# If the percentage free is greater than the limit allowed, then use the
# normal html class. This irrelevalnt for non-html output
         class=normal
      fi
      if [ $dispall == true -o $pcnt -le $limit ]
      then
         if [ $htmflag == true ]
         then

# If using html output, then create a table row
            display "<tr>"
            display "<!-- Mount Point -->"
            display "<td class=\"$class\">$mntpnt</td>"
            if [ $pcntonly = false ]
            then

# Only display free and total size if they are required
 
               display "<!-- Free Space -->"
               display "<td class=\"$class\" style=\"text-align: right;\">$free</td>"
               display "<!-- Total Space -->"
               display "<td class=\"$class\" style=\"text-align: right;\">$total</td>"
            fi
            display "<!-- Percentage Free -->"
            display "<td class=\"$class\" style=\"text-align: right;\">$pcnt%</td>"
            if [ $limitsource == "file" ]
            then
               display "<!-- Limit -->"
               display "<td class=\"$class\" style=\"text-align: right\">$limit%</td>"
            fi
            display "</tr>"
         else
# If using standard output, write a row.
            display "$mntpnt  \c"
            if [ $pcntonly = false ]
# Only display free and total size if they are required
            then
               display "$free  $total  \c"
            fi
            display "$pcnt%  \c"
            if [ $limitsource == "file" ]
            then
               display "$limit%\c"
            fi

            if (( pcnt <= limit ))
            then
               display "$boldoff"
            else
               display
            fi
         fi
      fi
   else
# if the values read were not numeric, then do not perform any calcualtions
# simply output them as read.
      if [ $dispall == true ]
      then
         if [ $htmflag == true ]
         then
            class="normal"
            display "<tr>"
            display "<!-- Non numeric values read -->"
            display "<td class=\"$class\">$mntpnt</td>"
            if [ $pcntonly = false ]
            then
               display "<td class=\"$class\" style=\"text-align: right;\">$free</td>"
               display "<td class=\"$class\" style=\"text-align: right;\">$total</td>"
            fi
            display "</tr>"
         else
            if [ $pcntonly = false ]
            then
               display "$mntpnt  $free  $total"
            else
               display "$mntpnt"
            fi
         fi
      fi
   fi
done

if [ $htmflag == true ]
then
# If using html output, close the table and the html page
   display "</table></body></html>"
fi

return $returnstat

#
#############################################################


Send HTML Format Email From Linux

The following script is very useful for sending HTML formatted emails from Linux.

Simply pipe the HTML into it, and the script adds the necessary header information before sending it on it's way. The script assumes that sendmail is installed and configured.

I wrote the original version of this script back in 2006, and it has proved so useful - for presenting daily system reports with warnings in red, for example - that I have used versions of it in most companies that I have done work for ever since.

#!/usr/bin/bash
##################################################################
#
# htmmail
#
# Purpose: To send html formatted emails
#
# Syntax: htmmail [-s Subject] address [address] [address...]
#
# Subject The subject of the email
# address The email address of the intended recipient
#
# The content of the email may be included by piping it
# into the htmmail command.
#
# Author: Douglas Milne
# Date: 27th May 2014
#
##################################################################
#
# Version 1.0 Initial Release
#
##################################################################

# Get options
# s flag argument is the subject of the email
#
while getopts s: value
do
case $value in
s) SUBJECT=$2
;;
\?) echo "$0: unknown option $OPTARG"
;;
esac
done

shift $(expr $OPTIND - 1)

# Remaining arguments are email addresses to send the email
TO=$*

# Set up email header, including content type field
# signifying html format.
# To and Subject as specified above

/usr/lib/sendmail -t << EOF
mime-version: 1.0
content-type: text/html; charset="iso-8859-1"
To: $TO
Subject: $SUBJECT

$(cat)

EOF


What is RAID?

RAID stands for Redundant Array of Inexpensive (or Independent) Disks. In simple terms, RAID is a method of storing data across several disks. There are several reasons why this is a good idea:

• Provides better performance
• Provides redundancy in the storage system, thus providing greater reliability
• Allows several smaller physical hard drive to be treated as one large logical hard drive
• Allows greater flexibility in adding or removing hard drives

RAID Data Distribution Techniques

RAID is achieved via a combination of a variety of methods of distributing data across a number of disks. Individually, these allow for faster access, or for data recovery. Different RAID levels use different combinations depending on the requirements of the system requirements.

Striping

Striping distributes the data across 2 or more disks. This allows for faster access to the data as the two disks can be reading or writing data simultaneously

Mirroring

Mirroring puts a copy of the data on a second disk. New disks must be added to the system in pairs - one for data and one for the mirror. If a disk fails, the data can be recovered from the mirror.

Parity

Parity performs an Exclusive-OR (XOR) between bytes on two disks and writes the results to a third. If any of the three disks fails, the missing data can be reconstructed by performing an XOR between the two remaining disks.

RAID Levels

RAID may be implemented at different levels, depending on the levels of redundancy and performance requirements of the overriding system. These two factors trade against each other: A higher level of redundancy reduces overall performance, but increases the safety of the data and the reliability of the system.

Linear

• Treats RAID hard drives as one virtual drive
• No striping, mirroring or parity reconstruction
• Storage is sequential
• No recovery capability

0 - Striping

• Implements disk striping across drives with no redundancy (no mirroring and no parity)
• Very efficient
• Standardized stripes across drives
• Faster access
• Requires a minimum of 2 disks
• Should not be used for any critical system

1 - Mirroring

• Implements redundancy through mirroring  (but no striping. and no parity)
• The same data is written to each RAID drive
• Each disk has a complete copy of all of the data
• If one or more of the disks fail, then the others still have the data
• Very safe, but inefficient
• Requires a minimum of 2 disks
• Good performance

5 - Distributed Parity

• Implements data reconstruction capability using parity information. Blocks are striped and parity information is therefore distributed across all drives
• An alternative to mirroring
• Parity information is saved instead of full data duplication
• Requires a minimum of 3 disks
• Good redundancy
• Provides a good balance between performance and redundancy, and can be very cost-effective.
• Write operations will be slow.
• Use for systems that are heavily read oriented.

10 (or 1+0) - Striped Mirroring

• Implements redundancy through mirroring  (but no parity)
• A striped copy of the data is written to half of the RAID drives, then mirrored to the other half
• Each half has a complete striped copy of all of the data
• If one or more of the disks fail, then the others still have the data
• Requires a minimum of 4 disks
• Good performance and good redundancy
• Easily the best option for mission critical applications

Disable CTRL-ALT-DEL On A Linux Box

Pressing the CTRL-ALT-DEL combination of keys on a Linux box forces it to reboot

This is set up within /etc/inittab.

To change how this key combination behaves, edit this file.

Search for the line:
ca::ctrlaltdel:/sbin/shutdown -t3 -r now
and change it to:
ca::ctrlaltdel:/bin/echo "CTRL-ALT-DEL is disabled"
Save the file, then run:
init q
to reload the inittab and activate the change.

How to List Sizes of Sub-Directories

Performing a simple du -h on a Linux directory produces way too much information if there are a lot of sub-directories. To find the sizes of only the directories in the current directory, which is often more useful, look at the last line of the du -h only for sub-directory. To list size of all subdirectories
for x in $(ls); do du -h $x | tail -1; done


SSL Overview

It is important that transmissions between a web server and a browser are secure. There are three tasks that must be performed in order to do this:

1. Verify the identities of the hosts participating in the transmission by performing authentication procedures.
2. Check the integrity of the data by adding digital signatures containing a digest value - a unique value representing the data.
3. Secure the privacy of the transmission by encrypting it. Transactions between a browser and the sever can then be encrypted, with only the browser and the server able to decrypt the transmissions.

SSL

The protocol most often used to implement secure transmissions is the Secure Sockets Layer (SSL) protocol. SSL was originally developed by Netscape for secure transactions on the web.

SSL uses a form of public- and private-key encryption for authentication. Data is encrypted with the public key, but can only be decrypted using the private key. Once the data is authenticated, an agreed-upon cipher is used to encrypt it. Digital signatures encrypt an MD5 digest value for data to ensure integrity.

Certificates

Authentication is carried out using certificates of authority. Certificates are held by both the browser and the web server, uniquely identifying both parties in a secure transmission, and verifying that they are who they say they are. Certificates are signed by an independent certificate authority such as VeriSign, verifying that they are valid.

A certificate contains:

• The public key of the server or browser that it is given to
• The digital signature of the certificate authority
• Identity information such as the name of the user or company running the server or browser.

SSL Session

An SSL session is set up using a handshake sequence:

1. The server and browser exchange certificates
2. A cipher is agreed upon to to encrypt the transmissions
3. The digest integrity check is chosen
4. The type of public-key encryption, usually RSA or DSA, is chosen
5. A unique session key is set up that is used by both the browser and the server.

HTTP Request and Response Overview

HTTP (Hyper-Text Transfer Protocol) is probably the most important of the various protocols are used for communication over the web. When a web address is typed into a web browser, the browser requests the web page using HTTP. The browser is an HTTP Client and the server is an HTTP Server. HTTP defines a set of rules regarding how messages and other data should be formatted between servers and browsers.

The HTTP protocol can be likened to a series of questions and answers, referred to respectively as HTTP Requests and HTTP Responses.

The HTTP Request

Once the connection to the server has been opened, the HTTP client transmits a request as follows:

• An opening line
• A number of header lines (optional)
• A blank line
• A message body (optional)

The opening line is generally split into three parts

1. The name of the method (POST, GET, PUT, DELETE or HEAD). Method names are always uppercase.
2. The path to the required server resource. The path is the part of the URL after the host name, also called the request URI (a URI is like a URL, but more general).
3. The HTTP version being used. The HTTP version is always in uppercase and takes the form HTTP/x.x.

For example:

GET /path/to/file/index.html HTTP/1.0

This tells the server that it is receiving an HTTP request of type GET, using HTTP version 1.0, and the server resource we are using, including it's local path, is /path/to/file/index.html.

Header lines are used to send information about the request, or about the data being sent in the message body. One parameter and value pair is sent per line, separated by a colon.

For example, a header line sent from IE9 might look like:
User-Agent: Mozilla /5.0 (Compatible MSIE 9.0;Windows NT 6.1;WOW64; Trident/5.0)
Another example of  a common request header is the Accept: header, which states what sort of information will be acceptable in the server's response.

For example
Accept: text/plain, text/html
This header informs the server that the sending application can accept only plain text or HTML responses.

If the HTTP request includes a message body, the header lines describe the content of the body.

For example
Content-Type: text/plain

This header informs the server that the message body contains text

.

In an HTTP request, the message body is where user-entered data or uploaded files are sent to the server.

The HTTP Response

The server issues an HTTP response in answer to an HTTP request.

The first line of the HTTP response i known as the status line. This consists of three parts separated by spaces:

1. the HTTP version. This is in the same format as for the HTTP request.
2. a three digit integer, called the HTTP response status code, that gives the result of the request, and
3. a short message known as the reason phrase describing the status code. 

Typical status lines are:
HTTP/1.0 200 OK or
HTTP/1.0 404 Not Found
The response status code and the reason phrase are intended as computer- and human-readable versions of the same message. The reason phrase may vary from server to server.

The first digit of the status code identifies the general category of response. Some commonly encountered HTTP Response Status codes are:

Status Code	Explanation
1** indicates an informational message only
100 - Continue	The client should send the remainder of the request
101 - Switching Protocols	The server will switch protocols to those defined following header
2** indicates success of some kind
200 - OK	The request succeeded
204 - No Content	The document contains no data
3** redirects the client to another URL
301 - Moved Permanently	The resource has permanently moved to a different URI
4** indicates an error on the client's part
401 - Not Authorized	The request needs user authorization
403 - Forbidden	The server has refused to fulfil the request
404 - Not Found	The requested resource does not exist on the server
408 - Request Timeout	The client failed to send a request in the time allowed by the server
5** indicates an error on the server's part
500 - Server Error	Due to a malfunctioning script, server configuration error or similar

The response may also contain header lines each containing a header and value pair similar to those of the HTTP request but generally containing information about the server and/or the resource being returned.

For example:
Server: Apache/2.4.5
Last Modified: Wed, 20 Nov 2013 13:33:59 GMT