Documents
Resources
Learning Center
Upload
Plans & pricing Sign in
Sign Out

Caching

VIEWS: 6 PAGES: 52

									                            Caching
• Caching uses faster hardware to save information
  (code or data) that you have used recently so that, if
  you need it again, it takes less time to access
   – for processing a program, caching takes place in cache
     memory, which is either stored on the CPU, or on the
     motherboard
      • storage is typically for a very brief time period (fractions of a
        second)
   – for secondary storage, caching is stored in a buffer on the
     hard disk
      • storage is typically until there are new hard disk accesses
   – for web access, caching is stored on the hard disk itself
      • storage is typically for about a month if the information being
        stored is static (dynamic web content is usually not cached)
Controlling Browser Caches from Apache
• Why wouldn’t you want your pages cached in the browsers
  of the users who visit your website?
   – if the content is being modified often
   – if the content is dynamic
      • typically browser caches will not cache dynamic content – if the file
        extension includes .php, .dhtml, or .cgi but this is not always the case
   – if the web page causes cookies to be created or set
   – if information is sent to a specific users, for instance a page
     created as a result of entering data into a form
• From apache, you can control how long items are cached
  by using the mod_expires module by using the HTTP
  Expires header
   – if the content is being modified, you can set an expiration date
     of the last modification
   – alternatively, you might want to set an expiration date relative
     to the date the items is sent (e.g., 3 days from now)
                      Expires Directives
• ExpiresActive – controls whether you will send an expiration in
  the header or not (values on or off)
   – just because you set this to on does not necessarily mean an
     expiration will be sent with the header, see below
• ExpiresByType type/encoding “<base> [plus] {<num>
  <type>}*”
   – this is an algorithm for computing expiration time for a particular file
     type
• ExpiresDefault “<base> [plus] {<num> <type>}*”
   – this is an algorithm for computing the expiration time for all other
     file types
       •   <base> is either access or modification
       •   plus is merely a keyword, optional
       •   num is a number
       •   type is the type of time unit (e.g., month, week, day, hour)
   – examples:
       • ExpiresDefault “access plus 2 weeks”
       • ExpiresType text/html “access plus 1 day”
       • ExpiresType image/gif “access plus 1 week 3 days 6 hours”
                Expiration Contexts
• These directives can be placed in any context
   – in a container, the directives only impact files of that
     container (e.g., that directory)
   – in an .htaccess, the directives only impact files of that
     directory (or lower)
   – ExpiresType and ExpiresDefault can override earlier
     definitions if placed in a lower context
      • e.g., if placed in <Directory /var/web/htdocs> and a later one is
        placed in <Directory /var/web/htdocs/pub> then the later one
        overrides the earlier one, and if there is an .htaccess file in
        /var/web/htdocs/pub/foo, it overrides the earlier ones
• If you do not use a default and a file does not match
  the give type in an ExpiresType, then no expiration
  would be sent
   – in order to ensure that a document is resent every time, set
     the expiration to 1 second (the least amount of time)
      • ExpiresDefault “access plus 1 second”
                        SSI Caching
• Recall that SSI is used to generate and/or insert dynamic
  content into your web pages
   – Apache will not place a last modification date or content-
     length in the HTTP header of any SSI page because these are
     difficult for Apache to determine, so by default, there is no
     date to compare against with respect to whether a document
     has expired (and thus, it will be assumed to have expired)
• However, if you want to permit SSI caching, it is possible
  because SSI can also be used to create the outlines for a
  page (e.g., using #include to include the navigation bar
  and footer)
   – two ways to do this:
      • use the XBitHack Full directive which tells Apache to determine the
        last modification date of the SSI file, not any included files
      • use the mod_expires and set the directives of the previous slides to a
        specified time for these particular files using a <Directory> or <Files>
        or <Location> container
                     Proxy Caches
• Browser caches are useful for users who frequently
  view the same web sites over and over
  – but for an organization, the browser caches cannot help
     • that is, a browser cache is local to a single computer and not
       shared among multiple clients of the same site
     • so the organization needs a cache that can extend across multiple
       users so that users who view the same web sites can obtain pages
       from the shared cache instead of having to wait for content to
       come across the Internet
• A proxy cache is one that extends across users of an
  organization
  – a proxy cache is part of a proxy server
  – the proxy server offers a cache for all users so that
    commonly accessed content can be retrieved across the
    Internet once and then shared, improving network usage
                      Proxy Servers
• A proxy server serves at least two functions
   – it offers an extended cache to the local users so that
     multiple users who access the same pages get a savings
   – it offers control over what material can be brought into the
     organization’s network and thus on to the clients
      • for instance, it can filter material for viruses
      • it can also filter material to disallow access to pornography, etc
   – other functions that it can serve include
      • an authentication server
      • performing SSL operations like encryption and decryption
      • collecting statistics on web traffic and usage
   – additionally, the proxy server can offer an added degree of
     anonymity in that it is the proxy server that places
     requests of remote hosts, not an individual’s computer
      • thus, the IP addresses sent to servers is that of the proxy server
        not of the client
        Forward vs Reverse Proxies
• The typical form of proxy server is the forward proxy
   – a collection of browsers (on the same LAN, or within an
     organization) share the same proxy server
   – all client requests go to the proxy server
      • the server looks in its cache to see if the material is available
      • if not, the server looks to make sure that the request can be fulfilled
        (does not violate any access rules), and sends the request over the
        Internet
      • once a response is received, the server caches it and responds to the
        client
• A reverse proxy server is used at the server end of the
  Internet
   – responses from the Internet come into the proxy server which
     then determines which web server to route the request on to
   – this might be used to balance the load of many requests for a
     company that runs multiple servers
   – it also allows the proxy server to cache information and
     respond directly if the requested page is in its cache
      • we’ll consider reverse proxy servers in a bit
  Using Apache as a Forward Server
• You can use your Apache web server as a forward proxy
  server with little enhancement
   – you might do this if you already have a web server and your
     organization uses it often (say for 50% of all web traffic)
• Use the mod_proxy module and the <proxy> container
   – ProxyRequests on (or off – the default)
   – ProxyTimeout x (x is in seconds)
   – <Proxy URL>
      • access directives here such as Deny from all, Access from
        172.31.5.0/24 and/or authorization directives that require log ins
   – </Proxy>
      • the URL can be a regular expression and matches based on the
        outgoing URL, use * for “all http requests”
• NOTE: to use Apache as a forward server, you must also
  configure your browsers to work with a proxy server
   – see pages 332-337
                       Other Directives
• NoProxy valuelist
   – This directive specifies items (domains, subnets, IP addresses,
     hostnames) that should be handled by this server rather than proxied
   – These items are separated by spaces as in 172.31.0.0/24 192
     www.google.com
       • any URL that would be sent to one of these locations is instead redirected to
         the location specified by a ProxyRemote directive
       • ProxyRemote * http://firewall.nku.edu
• ProxyBlock valuelist
   – Unlike NoProxy, the valuelist here can also include words or *,
     words mean that if they appear anywhere in the URL, they are
     blocked
• ProxyVia on, off – the default, full, block)
   – If on, then any request that is redirected by proxy will have a “Via:”
     line added to the header to indicate how the request was serviced
   – If full, then the server’s version is added to each Via line
   – If block, then remove all Via lines
       • this differs from off because off only controls this server, other servers
         whose ProxyVia is on will still insert their own Via line, if set to block, all
         via headers are removed by this server
               Additional Modules
• Aside from mod_proxy, you might use any of these
  – mod_proxy_ajp – AJP support for mod_proxy
     • AJP is the Apache Jserv Protocol which enhances performance
       by using a new binary format for packets, and adds SSL security
  – mod_proxy_balancer – extension for load balancing
     • Your Apache proxy server can issue requests to back-end servers
       (i.e., in a reverse proxy setting)
     • There are 3 load balancing algorithms:
         – Request Counting – each server gets an equal number of requests
         – Weighted Traffic Counting – requests are distributed based on byte
           count of the amount of work each server has recently handled
         – Pending Request Counting – distributed based on how many requests
           are currently waiting for each server
  – mod_proxy_connect – used for CONNECT request
    handling (connect is an HTTP method)
  – mod_proxy_ftp, mod_proxy_http – ftp and http support
    for the proxy server
  – mod_cache – as we discussed earlier in these notes
               Reverse Proxy Uses
• The reverse proxy works at the server end
   – One of its capabilities is to perform load balancing
• Additional features of the reverse proxy server are
   – Scrubbing – verification of incoming http requests to make
     sure that each request is syntactically valid
   – Fault tolerance – as part of load balancing, if a server goes
     down, the reverse proxy server can continue to maintain the
     incoming requests by reallocating the requests that the
     server was supposed to handle, and rebalancing the load to
     the remaining available servers
   – HTTPS support – if the back-end web servers do not have
     the capability
   – Redeployment – if a request requires a web application
     (e.g., execution of perl code), the request can be sent to a
     separate server that runs code
   – Central repository – to cache static data for quick response
     time
The Reverse Proxy Server
  Apache as a Reverse Proxy Server
• By default, Apache is configured to serve as a
  reverse proxy server
  – the forward proxy server is controlled through
    ProxyRequests
     • if you set it to off, Apache will function as a reverse proxy
       server but not a forward proxy server
  – the directives ProxyPass and ProxyPassReverse map
    incoming URLs to a new location no matter if that
    incoming URL is coming internally (from a client of
    this site that the Apache server is serving as a proxy)
    or externally (for a reverse proxy mapping)
     • ProxyPass /foo http://foo.example.com/bar
     • ProxyPassReverse /foo http://foo.example.com/bar
        – now, any request received by this server for anything under
          directory /foo is sent (redirected) to foo.example.com/bar
                                Squid
• Apache is not the best proxy server
   – its main use is for a web server only
   – it does not provide the types of security and access control that
     Squid has
• Therefore, we will concentrate on Squid
   – note: the textbook also discussed Pound but we will skip that
• Squid is an open source proxy server
   – its main use is as a forward proxy server but it can also be set up as
     a reverse proxy server
   – its genesis is back with the original CERN HTTP server from 1994
     which had a caching module
   – the caching module was separated and it has evolved over time into
     Squid
• In these notes, we will look at installing and running a basic
  configuration for Squid along with setting up access control
  list directives to control access and content
   – we are going to skip over a lot of detail on Squid as there is not
     sufficient time to cover it
                Installing Squid
• Squid source code for Linux or Windows can be
  found at http://www.squid-cache.org/Download/
  – installing from the source code allows you to
    configure Squid (much like we did with Apache)
• Before compiling, we will want to “tune” our
  kernel
  – recall that Linux limits the number of descriptors
    available by your software (possibly to 64)
  – this wasn’t too critical in Apache unless we were
    going to use it to run a lot of VHs
  – but it is important to Squid because Squid uses a file
    descriptor for each request, so we will want to
    increase the number of descriptors available to Squid
         Increasing File Descriptors
• In most Unix systems, its easy, just use ulimit -n
   – ulimit –n unlimited
   – ulimit –n 8192 (or some other number)
      • some Unix systems will use different commands or editing a
        config file
• In Linux its more complex
   – edit the file /usr/include/bits/typesizes.h and change the
     entry #define __FD_SETSIZE 1024 to a larger number
     such as 4096 or 8192
   – next, place that number in the file /proc/sys/fs/file-max
     (instead of editing that file, you could do echo 8192 >
     /proc/sys/fs/file-max)
   – now, you can use ulimit with –Hn as in ulimit –Hn 8192
      • make sure the number you use is consistent in all three operations
      • when done, you do not have to reboot Linux, now you can
        configure and compile Squid from source code
                 Configure Options
• Similar to Apache, you can change many defaults in
  Squid through the ./configure command
  – --prefix – same as in Apache
  – --sysconfdir, --localstatedir
     • change the location of the configuration (from prefix/etc) and var
       (from prefix/var) directories, the var directory stores Squid’s log
       files and disk cache
  – --enable-x
     • allows you to enable Squid-specific modules including
         – gnuregex
         – carp (Cache Array Routing Protocol useful for forwarding cache
           misses to an array)
         – pthreads
         – storeio (storage modules)
         – removal-policies
         – ssl, openssl
     • full list is available at http://wiki.squid-
       cache.org/SquidFaq/CompilingSquid#configure_options
               Squid Configuration
• Once compiled and installed, running Squid is fairly
  simple if you don’t want to make any changes to the
  configuration
  – the config file is squid.conf
     • like httpd.conf, the file contains comments and directives
     • directives are similar to httpd.conf directives in that they all start
       with the directive and are followed by zero or more arguments
       which can include for instance on/off, times (e.g., 2 minutes), IP
       addresses, filenames/paths, keywords such as deny, UNGET, etc
  – we will study some of the configuration directives later
  – as with Apache, changing the conf file requires that you
    restart Squid so that the file can be reread
     • although in Squid, you can keep Squid running and still have it
       reread this file
  – unlike Apache, Squid directives and values are case
    sensitive!
                Initializing the Cache
• Before running Squid, and whenever you want to add a
  new cache directory, you must first initialize the cache
  directory(ies)
   – squid –z
   – this initializes all of the directories listed in the Cache variable
     cache_dir
• For this command to work successfully
   – you must make sure that the owner that Squid runs under
     (probably squid) has read and write permission for each of the
     directories under cache_dir
       • when these directories are created, make sure they are either owned by
         squid or that squid is in the same group as the owner
   – the name of the owner of these directories is established using
     the cache_effective_user directive in squid.conf
   – you should start squid using the command su – squid
       • this tells Squid to switch from root to squid as soon as it can (after
         dealing with root-only tasks)
                         Running Squid
• You start Squid from the command line and control it much like
  apachectl, but there are a lot of possible options, here we look
  at the most important
   – -a port – start Squid but have it listen to the port supplied rather than
     the default port (3128), this also overrides any port specified in
     squid.conf using the http_port directive
   – -f file – specify alternative conf file
   – -k function – perform administrative function such as reconfigure,
     rotate, shutdown, debug or parse
       • parse causes Squid to read the conf file to test it for errors without reading it
         to configure itself, this is useful for debugging your conf file
   – -s – enables logging to the syslog daemon
   – -z – initializes cache directories
   – -D – disable initial DNS test
       • squid usually tests the DNS before starting
   – -N – keep Squid in the foreground instead of as a background
     daemon
       • you might do this when first testing Squid so that you can see immediate
         feedback printed to the terminal window, once debugged, kill Squid and
         rerun it without this option
                            Comments
• If you want to run Squid upon booting
   – you might add the start-up command to a script in rc.d, init.d or
     inittab
• Many people do not like running Squid in the main OS
  environment
   – for security purposes, just as you might not want to run apache in the
     main OS environment, therefore they create a chroot environment
   – this is a new root filesystem directory separate from the remainder of
     the filesystem
   – anyone who hacks into squid will not be able to damage your file
     system, only the chroot environment
• The safest way to shut down Squid is through
   – squid –k shutdown
       • do not use kill
• To reconfigure Squid after changing squid.conf
   – run squid –k reconfigure, this prevents you from having to
     stop/restart squid
• To rotate Squid log files, use squid –k rotate
   – put this in a crontab to rotate the files every so often (e.g., once a day)
                       ACLs in Squid
• Since apache can be used as a proxy server, you might
  wonder why use squid?
   – squid allows you to define access control lists (acls) which in
     turn can then be used to specify rules for access
      • who should be able to access web pages via squid?
      • what pages should be accessible? are there restrictions based on file
        name? web server? web page content or size?
      • what pages should be cached?
      • what pages can be redirected?
   – such rules are defined in two portions
      • acl definition (similar to what we saw when defining accessors in bind)
      • followed by an access statement (allow or deny statements)
   – Squid offers a variety of acl definition types
      •   IP addresses
      •   IP aliases
      •   URLs
      •   User names (requiring authentication)
      •   file types
        Defining ACLs and Rules
• Define access in two steps
  – first, define your ACL statements
     • simple definitions of a name to a specification
        – such as calling a particular IP address “home” or using a regular
          expression to match against URLs and calling them
          “homenetwork”
     • each acl contains a type that specifies what type of
       information you are using as a comparison, e.g., IP address,
       IP alias, user name, filename, port address, regular
       expression
  – second, define a rule for how the ACL(s) is to be used
     • the rule will typically specify if this acl can or cannot gain
       access through squid, for instance, if foo is a previously
       defined acl, then the following allows access
        – http_access allow foo
  – you must define an acl before you use it in any rule
                             Example
• The most common form of acl is to define and permit
  access to specific clients
   – we will define some src (source IP address) acls
      • typically with src, we define specific IP addresses or subnetworks
        (rather than IP aliases)
   – acl src localhost 127.0.0.1
      • here, we define the source acl “localhost” to be the IP address 127.0.0.1
   – acl src mynet 10.2/16
      • this could also be 10.2.0.0/16
• Now we use our acls to allow and deny access
   – http_access allow localhost
   – http_access allow mynet
   – http_access deny all
      • here, we are allowing access only from localhost and those on “mynet”,
        everyone else is denied
      • order of the allow and deny statements is critical, we will explore this
        next time
                    Types of ACLs
• Aside from src, you can also specify ACLs based on
  – dst – the URL of the web server (destination)
  – srcdomain and dstdomain – same as src and dst except
    that these permit IP aliases
  – srcdom_regex and dstdom_regex – same as srcdomain
    and dstdomain except that the IP aliases can be denoted
    using regular expressions
  – time – specify the times and days of the week that the
    proxy server allows or denies access
  – port, method, proto – specify the port(s) that the proxy
    server permits access, the HTTP methods allowable (or
    denied) and the protocal(s) allowable (or denied)
  – rep_mime_type – allow or deny access based on the type
    of file being returned
     • we will study these (and others) in detail next time
                 Types of ACLs
• src – the IP address of the user (client) whose
  requests are going from their browser to the squid
  proxy server
• srcdomain – the IP alias of the user
• dst – the IP address of the requested URL on the
  Internet
• dstdomain – the IP alias of the request
• myip – same as src, but it is the internal IP address
  rather than (possibly) an external IP address
• srcdom_regex, dstdom_regex – same as srcdomain
  and dstdomain except that regular expressions are
  permissible
• arp – access controlled based on the MAC address
                       Comments
• You can specify an IP alias using src or dst, but this
  requires that squid use a reverse DNS lookup
   – it is best to use srcdomain/dstdomain if you want to
     specify aliases instead of addresses
• When using srcdomain and dstdomain, you can
  specify part of the domain, such as .nku.edu or .edu
  instead of www.nku.edu
   – this is not true if you specify IP aliases using src and dst
• If you use src/dst, then after doing the reverse
  lookup one time, the value is cached
   – if the IP address were to change, squid would not be able
     to find the computer in the future
                     More ACLs
• port – specify one or more port numbers
  – ranges separated by – as in 8000-8010
  – multiple ports are separated by spaces or on separate
    definitions
  – typically, you will define “safe” ports and then disallow
    access to any port that is not safe, for example:
     • acl port safe_ports 80 443 8080 3128
     • http_access deny !safe_ports
• method – permissible HTTP method
  – GET, POST, PUT, HEAD, OPTIONS, TRACE, DELETE
  – squid also knows additional methods including
    PROPFIND, PROPPATCH, MKCOL, COPY, MOVE,
    LOCK, UNLOCK, CONNECT and PURGE
     • acl method allowable_method GET HEAD OPTIONS
     • http_access deny !allowable_method
                    More ACLs
• proto – permissible protocol(s)
  – http, https, ftp, gopher, whois, urn and cache_object
     • ex: acl proto myprotos HTTP HTTPS FTP
• proxy_auth – requires user login and a
  file/database of username/passwords
  – you specify the allowable user names here, such as
     • acl proxy_auth legal_users foxr zappaf newellg
• maxconn – maximum connections
  – you can control access based on a maximum number
    of server connections
  – this limitation is per IP address, so for instance you
    could limit users to 25 accesses, once the number is
    exceeded, that particular IP address gets “shut out”
                        Time ACLs
• To control when users can access the proxy server, based
  on either days of the week, or times (or both)
   – S, M, T, W, H, F, A for Sunday – Saturday, D for weekdays
   – time specified as a range, hh:mm – hh:mm in military time
• The format is acl name time [day(s)] [hh:mm - hh:mm]
   – example: to specify weekdays from 9 am to 5 pm:
      • acl weekdays time D 09:00 – 17:00
   – example: to specify Saturday and Sunday:
      • acl weekend time SA
• The first time must be less than the second
   – if you want to indicate a time that wraps around midnight,
     such as 9:30 pm to 5:30 am, you have to divide this into two
     definitions (9:30 pm – 11:59 pm, and 12:00 am – 5:30 am)
   – if days have different times, you need to separate them into
     multiple statements, such as wanting to define a time for M 3-7
     and W 3-8 would require two definitions
Regular Expressions and more ACLs
• As stated earlier, you can specify regular
  expressions in srcdom_regex and dstdom_regex
• There are also regex versions to build rules for
  the URL
  – url_regex and urlpath_regex
     • for the full URL and the path (directory) portion of the
       URL respectively
        – you might use this to find URLs that contain certain words, such
          as paths that include “bin”, or paths/filenames that include words
          like “porn”
  – ident_regex
     • to apply regular expressions to user names after the squid
       server performs authentication
     User Names & Authentication
• The ident acl can be used to match user names
• The proxy_auth acl can specify either REQUIRED or
  specific users by name that then require that a user log in
   – authentication requires that the user must perform a
     username/password authentication before Squid can continue
      • any request that must be authenticated is postponed until authentication
        can be completed
   – although authentication itself adds time, using ident or
     proxy_auth also adds time after authentication has taken place
     because Squid must still look up the user’s name among the
     authentication records to see if the name has been
     authenticated
• Squid itself does not come with its own authentication
  mechanisms, so we have to add them as modules much
  like with apache
                 Other ACL Types
• req_mime_type and rep_mime_type
  – test content-type in either the request or response header
  – it only makes sense to use req_mime_type when
    uploading a file via POST or PUT
  – example: acl badImage rep_mime_type image/jpeg
• Browsers
  – restrict what type(s) of browser can make a request
• External ACLs
  – this allows Squid to sort of “pass the buck” by requesting
    that some outside process(es) get involved to determine if
    a request should be fulfilled or not
     • external ACLs can include factors such as cache access time,
       number of children processes available, login or ident name, and
       many of the ACLs we have already covered, but now handled by
       some other server
            Matching Against ACLs
• As we have seen, a single ACL can contain multiple items
  to match against
• ACL lists are “ORed” items – the ACL is true if there is a
  match among any item in the list
   – to establish if an ACL is true, Squid works down the list of
     items looking for the first match, or the end of the list
      • if a match is found, the ACL is established as true, otherwise that ACL
        is established as false
   – for example: acl Simpsons ident Lisa Bart Marge Homer
      • Squid will attempt to confirm that the user’s identity, as previously
        established via authentication, matches any one of the items
   – if you have a lot of ACLs and/or lengthy lists in ACLs, it is
     worthwhile ordering the entries based on most common to least
     common
      • imagine that Homer is the most common user, then move Homer’s
        name to be first in the list, and if Bart is the least common user, move
        his name to the end
                       Types of Rules
• The most common rule is the http_access rule
   – access is either allow or deny
   – if allow and the acl matches, then you are allowing the client to
     have access, if deny and the acl matches, you are disallowing
     the client to have access
• You can also use http_reply_access
   – this allows the retrieved item to be let through the proxy server
     back to the client, again you can use allow or deny
      • this rule allows you to supply definitions that can disallow items being
        returned based on content (type, size, etc)
• You can control whether an item is cached or not using
  no_cache rules
   – here, the word “no” in a rule means “do not cache”
      • it looks like a double negative: no_cache someACL no
   – you would use this to ensure certain pages do not get cached
     (e.g., they have dynamic content, they aren’t worth caching,
     they are too large)
                    Matching Rules
• Imagine an access rule says
   – http_access allow A B C D
      • this means that all of A, B, C and D must be true for access to be
        allowed
      • Squid will stop searching this rule after the first mismatch, so
        again, you might order these in this case from the least likely to
        the most likely to be more efficient (if A is usually true but C is
        seldom true, put C first)
   – http_access deny A B C D
      • all must be true to deny access, if any are untrue, the rule is
        skipped
• To create OR access rules, list each access rule
  sequentially as in
   – http_access allow A
   – http_access allow B
      • now, if either A or B are true, access is allowed
              Allow vs Deny Order
• In Apache, you specified the order that allow and deny are
  enforced using the Order directive
   – in Squid, the order is based strictly on the order of the rules as
     they appear in your conf file
• In Apache, you would specify “deny from all” first and
  then override this with more specific “allow” statements
• In Squid, you do this in the opposite way
   – place an allow statement first, if the rule is true, then the
     remainder of the rules are skipped
   – add a deny all type statement at the end to act as a default or
     “fall through” case
   – you might define ALL to be everyone (e.g., IP address 0/0)
   – the deny all will look like this: http_access deny ALL
• You can specify multiple sets of rules, typically each set
  will contain allow statements and end with a deny ALL
                 Rule Organization
• You have to place your allow and deny rules in a logical
  manner for them to work
   – for instance, you would not do http_access deny All as the first
     rule because it would be true of everyone and no other rules
     would be checked
• You will want to do is organize rules generally like this:
   – specific denial rules
   – specific acceptance rules
   – http_access deny All
• In this way, if a particular situation fits both a denial and
  acceptance rule, the access is denied
   – for instance, a request may be acceptable because it has the
     proper src IP address, but it is during the wrong time of day, so
     it should ultimately be denied
   – by reversing the order of the denial and acceptance rules, the
     request would be fulfilled because as soon as it is accepted,
     access is allowed an no further rules are considered
                Common Scenarios
• Allowing only local clients
  –   acl ALL src 0/0
  –   acl MyNetwork src 172.31/16
  –   http_access allow MyNetwork
  –   http_access deny ALL
• Blocking a few clients (assume ALL and
  MyNetwork are as defined above)
  –   acl ProblemHosts 172.31.1.5 172.31.1.6 172.31.4/24
  –   http_access deny ProblemHosts
  –   http_access allow MyNetwork
  –   http_access deny ALL
       • notice the ordering here, since MyNetwork is more general than
         ProblemHosts, we first deny anyone specifically in
         ProblemHosts, then we allow access to those in MyNetwork that
         were not in ProblemHosts
                               More
• Denying access to any URL that looks like it might
  contain pornography
   – acl PornSites url_regex –i porn nude sex [add more terms
     here]
   – http_access deny PornSites
   – http_access allow ALL
        • here we allow anyone access if the URL does not include the list
          of PornSite words
• We might want to add to this a refusal to accept
  replies that contain movie or image files
   –   acl Movies rep_mime_type video/*
   –   acl Images rep_mime_type image/*
   –   http_reply_access deny Movies
   –   http_reply_access deny Images
   –   http_reply_access allow ALL
                        And More
• Here, we restrict access to be working hours and our
  own site (disallow access to URLs off site)
   –   acl WorkHours D 08:30-17:30
   –   acl OurLocation dstdomain “/usr/local/squid/etc/ourURLS”
   –   http_access allow WorkHours OurLocation
   –   http_access deny ALL
• And here is an example to permit only specific port
  accesses
   – acl SafePorts port 80 21 443 563 70 210 280 488 591 777
     1025-65535
   – acl SSLPorts port 443 563
   – acl CONNECT method CONNECT
   – http_access deny !SafePorts
   – http_access deny CONNECT !SSLPorts
   – http_access allow ALL
                          Redirectors
• A redirector is similar to the rewrite rules and redirection
  used by apache
   – here however, we are redirecting an internal request before it
     leaves the proxy server
      • in apache, we redirect an external request to either a new internal
        location/file or to a new external resource
   – this permits
      •   access control
      •   the remove of advertisement
      •   local mirroring of resources
      •   working around browser bugs
   – with access control, you can even send the user to a page that
     explains why they were rerouted
• A redirector is just a program that reads a URI (along with
  other information) and creates a new URI as output
   – redirectors are often written in Perl or Python, or possibly C
           How to Use a Redirector
• To apply a redirector in squid, issue one or more of
  the following directives
   – redirect_program specifies the external program to run
     (the redirector)
   – redirect_children specifies how many redirector processes
     Squid should start
   – redirect_rewrite_host_header will update a request header
     to specify that a redirector is being used
   – redirect_access allows you to specify rules that decide
     which requests to send to redirectors
      • without this, every request to Squid is sent to a redirector to
        check to see if it should be redirected
   – redirector_bypass will bypass a redirector if all spawned
     redirectors are currently busy, otherwise the requests
     begin to stack up and wait
           How to Write a Redirector
• A redirector receives four pieces of input
   –   request URI, including any query terms (after the ?)
   –   client IP address (and optionally domain name)
   –   user’s name or proxy authentication
   –   HTTP request method
• Your redirector code will consist of rules that
   – investigate parts of the input to see if any of the redirector
     rules match to the input
        • if a rule matches, then the redirector code will produce output
          which will be a new URI, redirecting the request
        • an example might be to search for any URL being sent to an IP
          address in China, and rewrite the query to a mirror site that exists
          in Taiwan
        • another example is to search the URI for certain “bad words” and
          if any are found, redirect the request to a local page that explains
          why any such requests are not being allowed
                    Redirector Code
• Aside from building a new URI for the request, it
  can also alter components of a response header
• The redirector code may involve
  –   database queries
  –   searching the URI for specified regular expressions
  –   complex computations
  –   invoking other programs
       • thus, a redirector can take a long time to respond, this would
         slow squid’s processing down
       • this is one reason why the bypass directive is available, you
         don’t necessarily want to penalize everyone because of
         redirections taking too much time
• Redirector code is commonly written in Perl but it
  can be written in other scripting languages
           Authentication Helpers
• As with Apache, Squid does not have built-in
  mechanisms for handling password files
  – so Squid turns to authentication helpers
  – Squid supports three forms of authentication, the first two
    are similar to Apache
     • Basic, Digest, NTLM (this is an MS authentication protocol)
  – for each of these, you have to download the software and
    compile it and then configure Squid to use the helper
     • we already visited how to write ACL and http_access directives
       that use authentication, so we skip it here
  – basic authentication helpers come with Squid, you can
    use NCSA (simple), LDAP, MSNT (for MS NT
    databases), NTLM, PAM, SASL (which includes SSL),
    winbind or others
                         Log Files
• As with Apache, Squid uses log files to store
  messages of importance and to maintain access and
  error logs
  – however, one additional log that Squid has that Apache
    does not is a cache log in order to record what files are
    cached
  – there are also optional log files available
     • useragent.log and referer.log which contain information about
       user agent headers and web referers for every access
     • swap.state and netdb_statestore information regarding the disk
       and network performance of Squid
  – you can control the names of the log files and which of
    these optional log files are used through directives in your
    conf file
  – because there are so many logs and they can generate a lot
    of content, there are log rotation tools available just as
    with Apache
                       cache.log
• This log contains
   – configuration information
   – warnings about performance problems
   – errors
• Entries are of the form
   – date time | message
• Configuration messages might include such things as
   – process ID of a starting squid process
   – successful (or failed) tests to the DNS and the DNS IP
     address (as obtained from resolv.conf)
   – starting helper programs
• The remaining cache entries are made based on a
  specified debug level that dictate which types of
  operations should be logged here
   – normal information, warnings, errors, emergencies, etc
                             access.log
• Much like Apache’s access log, Squid’s access log will store
  every request received
   – each entry contains 10 pieces of information
       • timestamp
       • response time
       • client address
       • status code of request
       • size of file transferred
       • HTTP method
       • URI
       • client identity (if available)
       • how requests were fulfilled on a cache miss (that is, where we had to go to
         get the file)
       • content type
   – status codes differ from Apache as they indicate cache access as well
     as server status codes, and include these:
       • TCP_HIT, TCP_MISS, TCP_REFRESH_HIT, TCP_REF_FAIL_HIT,
         TCP_REFRESH_MISS, TCP_CLIENT_REFRESH_MISS,
         TCP_IMS_HIT, TCP_SWAPFAIL_MISS, TCP_NEGATIVE_HIT,
         TCP_MEM_HIT, TCP_DENIED, TCP_OFFLINE_HIT, TCP_REDIRECT
         and NONE
           Directives for access.log
• log_icp_queries – default is enabled, allows you to control
  whether ICP (Internet Cache Protocol) requests are logged or
  not
• emulate_http_log – whether to use the same format as http
  server access logs (that is, match Apache’s server log) or use
  Squid’s native format which contains more information
• log_mime_hdrs – if set to on, Squid will add HTTP request and
  response headers to each log entry (this adds two more fields to
  each entry)
• log_fqdn – this toggles whether Squid records requests by
  destination IP address or hostname – if hostname, then Squid
  has to do a reverse DNS lookup which takes more time
• log_ip_on_direct – same as above except whether to log client’s
  (requestor’s) IP address or hostname
• strip_query_terms, uri_whitespace – whether to remove the
  query terms from an URL and whether to strip, chop, or encode
  white space in a URL (if any)
                         Store.log
• The store.log file stores decisions to store and remove
  objects from the Squid cache
   – if an object is cached, the entry includes where it was
     cached and when
   – if an object is uncacheable, then the entry indicates why
     the object was uncacheable
   – if a cache is full, a replacement strategy is used to decide
     what to remove, and any such action is logged here
• The store log contains the following fields:
   – timestamp, action (SWAPOUT, RELESE, SO_FAIL),
     directory number (which cache), file number, cache key
     (the hash value of the object), status code, date,
     last_modified from the HTTP response header, expires,
     content-type, content-length/size, HTTP method and URI

								
To top