Configure Squid to control web access

Squid a proxy server and web cache daemon. It has a wide variety of uses, from speeding up a web server by caching repeated requests, to caching web, DNS and other computer network lookups for a group of people sharing network resources, to aiding security by filtering traffic. Squid is primarily used for HTTP and FTP and it includes limited support for several other protocols such as TLS, SSL, Internet Gopher and HTTPS and the development version of Squid includes IPv6 and ICAP support too.

In this article I’m not going to cover the installation process of Squid-cache. My focus will be on the access control based configuration of Squid-cache for various requirements and also I’ll be covering how to fine tune the other applications to work with Squid, such as the firewall. In other words I’m gonna talk about access-controls (ACLs) in squid.conf and some post configurations.

The “/etc/squid/squid.conf” file

The main Squid configuration file is squid.conf, and, like most Linux applications, Squid needs to be restarted for changes to the configuration file can take effect.

Squid will fail to start if you don’t give your server a hostname. You can set this with the visible_hostname parameter. Here, the hostname is set to the real name of the server ‘myhost’.

visible_hostname myhost

You can limit users’ ability to browse the Internet with access control lists (ACLs). Each ACL line defines a particular type of activity, such as an access time or source network, they are then linked to an http_access statement that tells Squid whether or not to deny or allow traffic that matches the ACL.

Squid matches each Web access request it receives by checking the http_access list from top to bottom. If it finds a match, it enforces the allow or deny statement and stops reading further. You have to be careful not to place a deny statement in the list that blocks a similar allow statement below it.

NOTE: The final http_access statement denies everything, so it is best to place new http_access statements above that statement.

Squid has a minimum required set of ACL statements in the ACCESS_CONTROL section of the squid.conf file. It is best to put new customized entries right after this list to improve the readability.

Restricting web access by time

You can create access control lists with time parameters. For example, you can allow only business hour access from the home network, while always restricting access to host 192.168.1.10.

#
# Add this to the bottom of the ACL section of squid.conf
#
acl home_network src 192.168.1.0/24
acl business_hours time M T W H F 9:00-17:00
acl RestrictedHost src 192.168.1.10

#
# Add this at the top of the http_access section of squid.conf
#
http_access deny RestrictedHost
http_access allow home_network business_hours

Or, you can allow morning access only:

#
# Add this to the bottom of the ACL section of squid.conf
#
acl morning_hours time 08:00-12:00

#
# Add this at the top of the http_access section of squid.conf
#
http_access allow morning_hours

Restricting access to specific URLs

Squid is also capable of reading files containing lists of web sites and/or domains for use in ACLs. In this example we create to lists in files named /etc/squid/allowed-sites.acl and /etc/squid/restricted-sites.acl

# File: /etc/squid/allowed-sites.acl
www.gnu.org
mysite.com

# File: /etc/squid/restricted-sites.acl
www.restricted.com
illegal.com

These can then be used to always block the restricted sites and permit the allowed sites during working hours. This can be illustrated by expanding our previous example slightly.

#
# Add this to the bottom of the ACL section of squid.conf
#
acl home_network src 192.168.1.0/24
acl business_hours time M T W H F 9:00-17:00
acl GoodSites dstdomain "/etc/allowed-sites.acl"
acl BadSites  dstdomain "/etc/restricted-sites.acl"

#
# Add this at the top of the http_access section of squid.conf
#
http_access deny BadSites
http_access allow home_network business_hours GoodSites

Restricting web access by IP address

You can create an access control list that restricts web access to users on certain networks. In this case, it’s an ACL that defines a home network of 192.168.1.0.

#
# Add this to the bottom of the ACL section of squid.conf
#
acl home_network src 192.168.1.0/255.255.255.0

You also have to add a corresponding http_access statement that allows traffic that matches the ACL:

#
# Add this at the top of the http_access section of squid.conf
#
http_access allow home_network

Password based authentication using NCSA

You can configure Squid to prompt users for a username and password when they are browsing any URLs. Squid comes with a program called ncsa_auth that reads any NCSA-compliant encrypted password file. You can use the htpasswd program that comes installed with Apache to create your passwords. Here is how it’s done:

First you need to create the password file. Here the name of the password file should be /etc/squid/squid_passwd, and you need to make sure that it’s universally readable.

[root]# touch /etc/squid/squid_passwd
[root]# chmod o+r /etc/squid/squid_passwd

Then use the htpasswd program to add users to the password file. You can add users at anytime without having to restart Squid. In this case, you add a username called ‘test_user’:

[root]# htpasswd /etc/squid/squid_passwd test_user
New password:
Re-type new password:
Adding password for user test_user

Now you have to locate the ncsa_auth file.

[root]# locate ncsa_auth
/usr/lib/squid/ncsa_auth

Edit squid.conf; specifically, you need to define the authentication program in squid.conf, which is in this case ncsa_auth. Next, create an ACL named ncsa_users with the REQUIRED keyword that forces Squid to use the NCSA auth_param method you defined previously. Finally, create an http_access entry that allows traffic that matches the ncsa_users ACL entry. Here’s a simple user authentication example; the order of the statements are important:

#
# Add this to the auth_param section of squid.conf
#
auth_param basic program /usr/lib/squid/ncsa_auth /etc/squid/squid_passwd

#
# Add this to the bottom of the ACL section of squid.conf
#
acl ncsa_users proxy_auth REQUIRED

#
# Add this at the top of the http_access section of squid.conf
#
http_access allow ncsa_users

This will enable the password based authentication and allows access only during business hours. Once again, the order of the statements is important:

#
# Add this to the auth_param section of squid.conf
#
auth_param basic program /usr/lib/squid/ncsa_auth /etc/squid/squid_passwd

#
# Add this to the bottom of the ACL section of squid.conf
#
acl ncsa_users proxy_auth REQUIRED
acl business_hours time M T W H F 9:00-17:00

#
# Add this at the top of the http_access section of squid.conf
#
http_access allow ncsa_users business_hours

Remember to restart Squid for the changes to take effect.

Forcing users to use your Squid Server

If you are using access controls on Squid, you may also want to configure your firewall to allow only HTTP Internet access to only the Squid server. This forces your users to browse the Web through the Squid proxy. Also it is possible to limit HTTP Internet access to only the Squid server without having to modify the browser settings on your client PCs. This called a transparent proxy configuration. It is usually achieved by configuring a firewall between the client PCs and the WAN to redirect all HTTP (TCP port 80) traffic to the Squid server on TCP port 3128, which is the Squid server’s default TCP port.

Squid transparent proxy configuration

Your first step will be to modify your squid.conf to create a transparent proxy. The procedure is different depending on your version of Squid. In older versions of Squid ( <>squid.conf would be as follows:

httpd_accel_host virtual
httpd_accel_port 80
httpd_accel_with_proxy on
httpd_accel_uses_host_header on

Newer versions of Squid simply require you to add the word “transparent” to the default “http_port 3128″ statement. In this example, Squid not only listens on TCP port 3128 for proxy connections, but will also do so in transparent mode.

http_port 3128 transparent

Configuring iptables to support the Squid tansparent proxy

In this example, assuming the Squid server and firewall are in the same server, all HTTP traffic from the home network is redirecting to the firewall itself on the Squid port of 3128 and then only the firewall itself has access the Internet on port 80.

iptables -t nat -A PREROUTING -i eth1 -p tcp --dport 80 -j REDIRECT --to-port 3128
iptables -A INPUT -j ACCEPT -m state --state NEW,ESTABLISHED,RELATED -i eth1 -p tcp --dport 3128
iptables -A OUTPUT -j ACCEPT -m state --state ESTABLISHED,RELATED -o eth1 -p tcp --sport 80

Note: This example is specific to HTTP traffic. You won’t be able to adapt this example to support HTTPS web browsing on TCP port 443, as that protocol specifically doesn’t allow the insertion of a “man in the middle” server for security purposes. One solution is to add IP masquerading statements for port 443, or any other important traffic, immediately after the code snippet. This will allow non HTTP traffic to access the Internet without being cached by Squid.

Bash Tips and Tricks

Bash, or the Bourne Again Shell, is the default shell in most Linux distributions. The popularity of the bash shell amongst Linux and UNIX users is no accident. It has many features to enhance user-friendliness and productivity. Unfortunately, you can’t take advantage of those features unless you know they exist.

When I first started using Linux, the only bash feature I took advantage of was going back through the command history using the up arrow. I soon learned additional features by watching others and asking questions. In this article, I’d like to share some bash tricks I’ve learned over the years.

This article isn’t meant to cover all of the features of the bash shell; that would require a book, and plenty of books are available that cover this topic. Instead, this article is a summary of the bash tricks I use most often and would be lost without.

Brace Expansion

One of my favorite bash tricks is brace expansion. Brace expansion takes a list of strings separated by commas and expands those strings into separate arguments for you. The list is enclosed by braces, the symbols { and }, and there should be no spaces around the commas. For example:

$ echo {one,two,red,blue}
one two red blue

Using brace expansion as illustrated in this simple example doesn’t offer too much to the user. In fact, the above example requires typing two more characters than simply typing:

echo one two red blue

which produces the same result. However, brace expansion becomes quite useful when the brace-enclosed list occurs immediately before, after or inside another string:

$ echo {one,two,red,blue}baloon
onebaloon twobaloon redbaloon bluebaloon

$ echo fish{one,two,red,blue}
fishone fishtwo fishred fishblue

$ echo fi{one,two,red,blue}sh
fionesh fitwosh firedsh fibluesh

Notice that there are no spaces inside the brackets or between the brackets and the adjoining strings. If you include spaces, it breaks things:

$ echo {one, two, red, blue }fi
{one, two, red, blue }fi

$ echo "{one,two,red,blue} fi"
{one,two,red,blue} fi

However, you can use spaces if they’re enclosed in quotes outside the braces or within an item in the comma-separated list:

$ echo {"one ","two ","red ","blue "}fish
one fish two fish red fish blue fish

$ echo {one,two,red,blue}" fish"
one fish two fish red fish blue fish

You also can nest braces, but you must use some caution here too:

$ echo {{1,2,3},1,2,3}
1 2 3 1 2 3

$ echo {{1,2,3}1,2,3}
11 21 31 2 3

Now, after all these examples, you might be thinking to yourself, “Those are great parlor tricks, but why should I care about brace expansion?”

Brace expansion becomes useful when you need to make a backup of a file. This is why it’s my favorite shell trick. I use it almost every day when I need to make a backup of a config file before changing it. For example, if I’m making a change to my Apache configuration, I can do the following and save some typing:

$ cp /etc/apache2/apache2.conf{,.bak}

Notice that there is no character between the opening brace and the first comma. It’s perfectly acceptable to do this and is useful when adding characters to an existing filename or when one argument is a substring of the other. Then, if I need to see what changes I made later in the day, I use the diff command and reverse the order of the strings inside the braces:

$ diff /etc/apache2/apache2.conf{.bak,}
1050a1051
> # I added this comment earlier

Redirecting Standard Error

Have you ever looked for a file using the find command, only to learn the file you were looking for is lost in a sea of permission denied error messages that quickly fill your terminal window?

If you are the administrator of the system, you can become root and execute find again as root. Because root can read any file, you don’t get that error anymore. Unfortunately, not everyone has root access on the system being used. Besides, it’s bad practice to be root unless it’s absolutely necessary. So what can you do?

One thing you can do is redirect your output to a file. Basic output redirection should be nothing new to anyone who has spent a reasonable amount of time using any UNIX or Linux shell, so I won’t go into detail regarding the basics of output redirection. To save the useful output from the find command, you can redirect the output to a file:

$ find / -name foo > output.txt

You still see the error messages on the screen but not the path of the file you’re looking for. Instead, that is placed in the file output.txt. When the find command completes, you can cat the file output.txt to get the location(s) of the file(s) you want.

That’s an acceptable solution, but there’s a better way. Instead of redirecting the standard output to a file, you can redirect the error messages to a file. This can be done by placing a 2 directly in front of the redirection angle bracket. If you are not interested in the error messages, you simply can send them to /dev/null:

$ find / -name foo 2> /dev/null

This shows you the location of file foo, if it exists, without those pesky permission denied error messages. I almost always invoke the find command in this way.

The number 2 represents the standard error output stream. Standard error is where most commands send their error messages. Normal (non-error) output is sent to standard output, which can be represented by the number 1. Because most redirected output is the standard output, output redirection works only on the standard output stream by default. This makes the following two commands equivalent:

$ find / -name foo > output.txt
$ find / -name foo 1> output.txt

Sometimes you might want to save both the error messages and the standard output to file. This often is done with cron jobs, when you want to save all the output to a log file. This also can be done by directing both output streams to the same file:

$ find / -name foo > output.txt 2> output.txt

This works, but again, there’s a better way to do it. You can tie the standard error stream to the standard output stream using an ampersand. Once you do this, the error messages goes to wherever you redirect the standard output:

$ find / -name foo > output.txt 2>&1

One caveat about doing this is that the tying operation goes at the end of the command generating the output. This is important if piping the output to another command. This line works as expected:

find -name test.sh 2>&1 | tee /tmp/output2.txt

but this line doesn’t:

find -name test.sh | tee /tmp/output2.txt 2>&1

and neither does this one:

find -name test.sh 2>&1 > /tmp/output.txt

I started this discussion on output redirection using the find command as an example, and all the examples used the find command. This discussion isn’t limited to the output of find, however. Many other commands can generate enough error messages to obscure the one or two lines of output you need.

Output redirection isn’t limited to bash, either. All UNIX/Linux shells support output redirection using the same syntax.

Using Loops from the Command Line

One last tip I’d like to offer is using loops from the command line. The command line is not the place to write complicated scripts that include multiple loops or branching. For small loops, though, it can be a great time saver. Unfortunately, I don’t see many people taking advantage of this. Instead, I frequently see people use the up arrow key to go back in the command history and modify the previous command for each iteration.

If you are not familiar with creating for loops or other types of loops, many good books on shell scripting discuss this topic. A discussion on for loops in general is an article in itself.

You can write loops interactively in two ways. The first way, and the method I prefer, is to separate each line with a semicolon. A simple loop to make a backup copy of all the files in a directory would look like this:

$ for file in * ; do cp $file $file.bak; done

Another way to write loops is to press Enter after each line instead of inserting a semicolon. bash recognizes that you are creating a loop from the use of the for keyword, and it prompts you for the next line with a secondary prompt. It knows you are done when you enter the keyword done, signifying that your loop is complete:

$ for file in *
> do cp $file $file.bak
> done

And Now for Something Completely Different

When I originally conceived this article, I was going to name it “Stupid bash Tricks”, and show off some unusual, esoteric bash commands I’ve learned. The tone of the article has changed since then, but there is one stupid bash trick I’d like to share.

About 2 years ago, a Linux system I was responsible for ran out of memory. Even simple commands, such as ls, failed with an insufficient memory error. The obvious solution to this problem was simply to reboot. One of the other system administrators wanted to look at a file that may have held clues to the problem, but he couldn’t remember the exact name of the file. We could switch to different directories, because the cd command is part of bash, but we couldn’t get a list of the files, because even ls would fail. To get around this problem, the other system administrator created a simple loop to show us the files in the directory:

$ for file in *; do echo $file; done

This worked when ls wouldn’t, because echo is a part of the bash shell, so it already is loaded into memory. It’s an interesting solution to an unusual problem. Now, can anyone suggest a way to display the contents of a file using only bash built-ins?

Conclusion

Bash has many great features to make life easier for its users. I hope this summary of bash tricks I like to use has shown you some new ways to take advantage of the power bash has to offer.


Tags : Bash, Bash find, Bash loops, CLI, Command line, Redirecting errors, shell, Tips, Tricks
Categories : HOW TO