Pre-release checklist:
- perform compile test on hp testdrive systems.
- update version in configure.ac.
- grep source for XXX.
- make HTML documentation from docbook.
- tag CVS.
- include version of archive directory.

possible bugs:
- this showed up in the logfile:
  Sat 03 06:17:18 [21496] cache: /var/cache/mman/3b/http:\\www.bulkmarketing.net:80(null) doesn't appear to be a valid cache file

needs testing before next release:
- transfer limiting (test buffered and unbuffered transfers).
- forwarding (add own proxy to forward entries and ensure it doesn't forward to it).
- dns blacklist


cleanup tasks:
- add URL struct to header type and get rid of proto, host, file, and port members.
- utilize new methods in Socket class (i.e. instead of net_connect + new Socket, use new Socket + Socket->Connect)
- for every Section which can cause buffering to occur, add a check_match method to check if it should be buffered.
- remove 'size' option from all buffering sections.. it's broken
- move DNSBL checking code into dnsbl_section class.

short term TODO:
- have master track live slaves and report them to other slaves when they sync... and have every proxy in the cluster
  track which peers are alive.

  when a slave syncs, add a header to let the master know the sync interval.. have the master consider it dead if 
  it doesn't sync again after that interval.

  when a slave syncs with a master, have the master send a header containing a list of alive and dead slaves.

  add PeerTrack class with the following methods:
  alive(string host); - set peer status to alive.
  dead(string host); - set peer status to dead.
  isalive(string host); - return TRUE if peer is alive.
  add(string host); - add known peer (we've seen activity from the peer).
  remove(string host); - remove known peer (it's been way too long since the peer was active).
  known(string host); - return TRUE if peer is known.
  timeout(string host, time_t timeout); - set activity timeout until peer is considered dead (only if it's syncing).
  retry(string host, time_t timeout); - set timeout after peer is considered dead to give it another try.
  activity(string host); - activity seen by peer.

- add option to add filtered url's to prefetch queue so banner is downloaded (generating revenue for website).

- cleanup the code of main.c and protocol.c in order to be more readable and more object oriented. All classes inhertited from Section
  should have only public methods that properly lock/unlock the section, and all other methods and variables must be private. For example, 
  see AccessSection.

- I'd like to move more stuff over to C++ classes, the following should be fairly easy:

  html stream parser (for prefetching)
  regular expressions
  connection pool

- make outgoing socket connections bind on the same address as the client connection was accepted on.

- replace all *_new functions for struct initializers into constructors.

- clean up code for transfer limiting (hacks still remain in network code).

long term TODO:

- better statistics gathering... develope protocol to have slave proxies report statistics to master, and have master
  aggregate them and provide overall statistics for a proxy cluster.

- access control based on X-Forwarded-For header.

- support ranged requests.
  for now range requests are passed directly to the webserver... eventually I'd like for it to work with cache and ftp.

- write some regressions tests to automate testing before releases.
  automate the enabling/disabling of features via the web interface and perform various
  tests to ensure those settings are working correctly.

  preferably, these tests would automatically set/unset configuration settings by making requests
  to the web interface and systematically test each setting to ensure it works as expected.

- benchmarking.
  some comprehensive benchmarks of cache performance and how it compares to other proxy servers are needed
  so I have a better idea of what needs improvement. Unfortunately, there's not many tools fit for the task..
  web polygraph looks alright but is a bitch to setup.

- ipv6 support.

- make it possible to add/delete headers within an external parser.

- make it possible to select the order content modification features are applied.

  add a new section that allows the order Keyword scoring, rewriting, external parsing, and HTML stream parsing are
  applied to be configured. Each entry would contain an ordered list of these features (a new field type is needed) 
  as well as the standard comment/profiles options.

- add support for ICAP (Internet Content Adaptation Protocol) protocol.

  ICAP is a protocol that allows an external server (ICAP server) to make modifications to requests and responses.

  There are 4 types of actions an ICAP server may make:

  - Request modification: Client request header is sent to ICAP server, it is then sent back by the ICAP server, possibly
    modifified.

  - Request satisfaction: Client request header is sent to ICAP server, the ICAP server then fulfills the request.

  - Response modification: Origin server response (including body) is sent to ICAP server, ICAP server sends back
    possibly modified version which is then delivered to the client.

  - Result modification: Origin server response (including body) is sent to ICAP server, ICAP server sends possibly
    modified version directly to the client.

   ICAP transfers are always chunked.

   for faster processing, send header/body to ICAP server in socket Read callback.

   hop-to-hop headers are not sent to ICAP server (Connection and Proxy-Connection headers)

   An ICAP request looks like this:

   -------
   (REQMOD|RESPMOD) icap://$host:$port/$file ICAP/1.0
   User-Agent: Middleman $version
   Encapsulated: (req-hdr|res-hdr)=0, (null-body|req-body|res-body)=$header_length
   Connection: (Keep-Alive|Close)
   \r\n
   REQUEST OR RESPONSE HEADER
   \r\n
   REQUEST OR RESPONSE BODY
   -------

   An ICAP response looks like this:

   -------
   ICAP/1.0 $code $message
   Encapsulated: (req-hdr|res-hdr)=0, (null-body|req-body|res-body)=$header_length
   \r\n
   REQUEST OR RESPONSE HEADER
   \r\n
   REQUEST OR RESPONSE BODY
   -------

 
- implement protocol to allow a group of proxy servers to syncronize their configuration.

  this will be useful to ease management when using several proxy servers together with CARP or ICP to improve scalability.
  the slave proxy servers will syncronize off the master, all changes to configuration will be broadcast
  across the socket. 

  slave proxy server will be invoked with the command line arguments -S (to indicate it's a slave), 
  -M (master host:port), -s (for section.xml), and -c (only general section is loaded)

  web interface will be inaccessible from slave proxy servers.

  general section is not syncronized.

  if connection to master is lost, have it attempt to reconnect at an interval and do a full resyncronization if successful.

  syncronization protocol:

  when slave proxy is started, it will connect to the master and perform a handshake (ensure versions match, etc).

  in each section's load method, instead of retrieving the configuration from the XML file the proxy will request
  all the global options and subsection entries from the master.

  when any configuration option is changed in the master, it will broadcast a message to all the slave's notifying them to 
  syncronize.

  use class derived from IOMultiplex to handle IO to slaves.

  transfer section's XML file, use Parse method in TiXmlDocument to parse it then pass it to the section's Load method.

  use HTTP to transfer section's XML file through the web interface.

- create an 'IOMultiplex' base class which manages a list of SOCKET objects and takes care of polling for input/output,
  use for sync master.

- add following members to SOCKET class:
  SOCKET *Accept() - accept (for listening sockets)
  int Listen(struct in_addr addr, int port) - make SOCKET a listening socket.
