TITLE
*****

This manual is for `ulog-acctd', version 0.4

   Copyright  2002 Hilko Bengen

     This program is free software; you can redistribute it and/or
     modify it under the terms of the GNU General Public License as
     published by the Free Software Foundation; either version 2 of the
     License, or (at your option) any later version.

     This program is distributed in the hope that it will be useful,
     but WITHOUT ANY WARRANTY; without even the implied warranty of
     MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
     General Public License for more details.

     You should have received a copy of the GNU General Public License
     along with this program; if not, write to the Free Software
     Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA
     02111-1307  USA

1 Overview
**********

`ulog-acctd' is a userspace network accounting daemon which generates
log files of network traffic for accounting purposes.

1.1 Features
============

`ulog-acctd' collects headers of IP packets fhat travel through the
Linux 2.4+ Netfilter framework. It writes accounting information to a
log which can include

   * protocol type,

   * source and destination address,

   * port information,

   * byte and packet count,

   * incoming and outgoing network interfaces.

   The log file format is configurable. For instance, it should be
possible to directly generate files that contain SQL statements so that
the raw accounting data can be fed into a DBMS.

1.2 Nonfeatures
===============

`ulog-acctd' only collects, aggregates, and logs the all data it is fed
by netfilter. If only traffic for specific networks needs to be
collected, selection for this should be done in the netfilter rulesets,
as `ulog-acctd' has no way to ignore some packets and log others.

   `ulog-acctd' does not analyze them nor paint pretty pictures for
traffic visualization. In the "one tool for one job" spirit of UNIX,
Other tools should be used for these tasks.

1.3 Others
==========

1.3.1 net-acct
--------------

``net--acct'' (`http://exorsus.net/projects/net--acct/') was developed
by Ulrich Callmeier. The Mekafile hints that it can be used on Solaris
(using libpcap). The Linux version uses the now obsolete packet
interface at protocol level. (see `ip(7)').

   Information for net-acct is available at
`http://www.exorsus.net/net--acct/'.

   I had tried to implement support for the newer packet interface at
device level (supported since Linux kernel versions 2.2, see
`packet(7)'), but I found that it lost packets in high load situations.
Patches against `net--acct' are available on request.

   `ulog-acctd' is based on `net--acct' and still shares some portions
of the code. `net--acct' does not seem to be actively maintained any
longer.

1.3.2 ipcad
-----------

`ipcad' stands for IP Cisco Accounting Daemon. It runs in background
and listens traffic on the specified interfaces. Statistics are
available through a Cisco-compatible rsh interface. `ipcad' can use the
BPF interface or libpcap, which on Linux probably leads to packet less
under high load, as described above.

   `ipcad' is available from `http://sourceforge.net/projects/ipcad'.

   It is certainly possible to set up a similar scenario with
`ulog-acctd' with a small rsh daemon (or maybe just a shell script)
which would emulate the relevant parts of Cisco command line syntax.

1.3.3 traff
-----------

`traff' (`http://sourceforge.net/projects/traff/') is a program that
attaches itself to one or more network interfaces, sniffs all
IP-packages passing at it and accounts each packets size. The collected
information can be dumped to an SQL database (PostgreSQL or MySQL),
syslog or a file.

   Like `ipcad', it uses libpcap.

1.3.4 ipac-ng
-------------

`ipac--ng' regularly reads byte and packet counters associated with the
Linux kernel's packet filter rules and stores those numbers in a
database. This works well for a small number of IP addresses / ports,
but it does not scale well since one rule in the packet filter is needed
for each possible entry.

1.3.5 ulogd
-----------

As the name suggests, `ulogd' uses the same approach for collecting IP
packets as `ulog-acctd' does. `ulogd' can be extended through binary
plugins for adding packet interpreters and output-targets (e.g. for
logging into databases, user defined filetypes, etc.). It can be
downloaded from `http://www.gnumonks.org/projects/ulogd'.

2 Installation
**************

  1. Compile the sources: `cd src; make'

  2. Generate the documentation from the Texinfo source (optional): `cd
     doc; make ulog-acctd.info ulog-acctd.html ulog-acctd.dvi'

  3. Install the `ulog-acctd' binary into appropriate location, i.e.
     `/usr/local/sbin'.

  4. Install the configuration file `src/ulog-acctd.conf' to `/etc' and
     edit it to your needs.

  5. Create the directories for logging that are referenced in the
     configuration file. (In the default configuration, this is
     `/var/log/ulog-acctd/'.)

3 Configuration
***************

3.1 Netfilter
=============

`ulog-acctd' relies on the kernel (more specific: Netfilter) to pipe
the packet headers to user space. This has to be enabled by adding
rules with the ULOG target to packet filtering tables.

   The following example causes all packets that successfully pass the
`FORWARD' table to be logged to `ulog-acctd'.

     iptables --append FORWARD -j ULOG --ulog-nlgroup 1 --ulog-cprange 48 --ulog-qthreshold 50 --ulog-prefix "FORWARD"

   Explaination:

   * `-j ULOG' causes the packet to passed to userspace in the first
     place.

   * `--ulog-nlgroup GROUP' specifies thet the packet be sent to
     multicast group GROUP. `ulog-acctd' has to be configured to listen
     to this group.

   * `--ulog-cprange SIZE' means that only the first SIZE bytes are
     copied to userspace. The default is to copy the whole packet,
     limiting the number of bytes should increase performance. However,
     when IP headers carry a lot of option fields, this may lead to the
     case that TCP, UDP, or ICMP headers are out of the copied range.
     If the `ulog-acctd' will log a warning "Short IP header. Increase
     copy range to RANGE" to to the system log as well as the debug log
     in this case.

     If the copy range is too small to even hold the IP address, an
     additional error message "copy range is too short to even capture
     IP headers. ALL IP PACKETS WILL BE DROPPED!" is logged.

   * `--ulog-qthreshold COUNT' makes the kernel collect COUNT packets
     before sending the message containing these packets to userspace.

   * `--ulog-prefix PREFIX' causes a prefix string (max. 32 characters)
     to be passed along with the packet. This can for example be used
     for including hints for billing systems.

   For further information, see `iptables(8)'.

3.2 ulog-acctd.conf
===================

Per default, configuration is done via the file `/etc/ulog-acctd.conf'.
The following keywords are recognized:

   * `multicast groups = LIST': LIST is a comma-separated list of
     multicast groups that `ulog-acctd' should listen to. Possible
     values are 1-32.

   * `socket receive buffer = SIZE': The kernel's socket receive buffer
     will be set to SIZE bytes instead of the `net/core/rmem_default'
     kernel parameter's value. However, it can not be set to a value
     higher than the `net/core/rmem_max' kernel parameter. (Both of
     these kernel parameters are set to 64kBytes by the kernel, but can
     be canged with `sysctl(8)'.

   * `hash table size = SIZE': `ulog-acctd' uses a hash table and
     linked lists to store the collected accounting data in memory.
     SIZE should be set to a power of 2. The default value is 65536.

     This has changed in version 0.4.2

   * `hash init value = VALUE': This value is used as a "salt" to the
     hash function. Only set this if you know what you are doing. The
     default value is 0.

   * `logger nice value = VALUE': `ulog-acctd' forks off separate
     processes for writing accounting data to disk. The nice value of
     these processes are set to VALUE. It is probably not advisable to
     set VALUE to a negative number. By default, the nice value is not
     changed.

   * `accounting file = FILENAME', `dump file FILENAME', `debug file =
     FILENAME', `pid file = FILENAME': Locations for the accounting
     file, dump file, debug log, pid file.

   * `debug = LIST': LIST is a comma-spearated list of debug message
     types. Valid debug message types are: `config', `state',
     `syscall', `misc', `statistics', `signal', `error', `annoying',
     `error-packet'.

   * `accounting format = STRING': STRING is a format string with
     somewhat similar syntax to `printf(3)'. The following directives
     are recognized and replaced in the output file.

    `%h'
          hostname

    `%x'
          since (when the first packet of a chunk was recorded)

    `%t'
          till (when the last packet of a ckunk was recorded)

    `%p'
          protocol

    `%s'
          source IP

    `%S'
          source port

    `%d'
          destination IP

    `%D'
          destination port

    `%P'
          packets

    `%b'
          bytes

    `%i'
          incoming interface

    `%o'
          outgoing interface

    `%r'
          prefix

    `%u'
          Local user that owns a TCP connection

    `%m'
          MAC address, if any.

    `%X'
          formatted date string (since)

    `%Z'
          formatted date string (till), see below for defining the
          format. The time when the output is written is used for this.

    `\t'
          tab

    `\n'
          newline

    `\\'
          Literal \

    `\"'
          Literal "

    `\%'
          Literal %

    `%%'
          Literal %

   * `date format = STRING' STRING is a format string for turning the
     date into a string. All the  directives that are known to
     `strftime(3)' on your system are recognized.  _This option has
     changed in version 0.4 of `ulog-acctd'._

   * `empty interface = STRING': If no incoming or outgoing interface
     is passed for a packet, insert STRING instead.

   * `empty prefix = STRING': If no ULOG-prefix is passed for a packet,
     insert STRING instead.

   * `flush = TIME': The data that has been accumulated in a in-memory
     structure is written to the accounting file every TIME seconds.

   * `fdelay = TIME': A certain record of traffic information may be
     written out after TIME seconds. This helps making the logfiles
     smaller since only one output record will be generated for related
     traffic

   Example:
     multicast groups=1
     accounting file = /var/log/ulog-acctd/account.log
     dump file =       /var/log/ulog-acctd/dump
     debug file =      /var/log/ulog-acctd/debug.log
     debug = error
     accounting format = "%h\t%x\t%t\t%p\t%s\t%S\t%d\t%D\t%P\t%b\t\"%i\"\t\"%o\"\t\"%f\"\n"
     flush = 30
     fdelay = 30

   The supplied accounting format string generates output records in the
following form (all in one line, with a linefeed at the end):
judith	1032521450	1032521454	6	192.168.42.11	110	192.168.215.71	1546	12	250113	"eth0"	"eth1"	"prefix"

   The following format string will generate CISCO "ip accounting
output-packets"-style output lines:
     accounting format = "%s %d %P %b\n"
   The lines will look like this:
     192.168.42.11 192.168.215.71 12 250113

   When not all types of information are collected, `ulog-acctd' will
have to do fewer comparisions _for each packet_ while collecting data.
Records are smaller and likely fewer records will be written out. This
will be the case especially if port information are not collected.

4 Running
*********

4.1 Invokation
==============

`ulog-acctd' recognizes the following command-line parameters:

   * `-d' Switch on debugging

   * `-c' Specify alternative configuration file

   * `-D' Don't detach from terminal

4.2 Signals
===========

The following signals are recognized by  `ulog-acctd':

`SIGHUP'
     causes the config file to be re-read.

`SIGTSTP'
     causes `ulog-acctd' to stop writing to the accounting logfile

`SIGCONT'
     causes `ulog-acctd' to re-open both debug and log file and resume
     writing to the account log.

`SIGIOT'
     causes `ulog-acctd' to re-open the capture socket.

`SIGINT'

`SIGTERM'
     These signals stop the daemon, gracefully. The PID file is cleaned
     up and the IP data that is resident in memory is flushed to the
     accounting file.

`SIGKILL'
     These signals stop the daemon.

`SIGALRM'
     This is used internally. Do not send `ulog-acctd' this signal.

4.3 Rotating Log files
======================

To cleanly rotate accounting log files, send `SIGTSTP' to the daemon,
then move the logfiles out of the way, and finally send a `SIGCONT' to
re-enable logging accounting data.

   Be sure _not_ to compress the log files immediately after rotating.
When the accounting log file is written to, the writing process will
create a hard link to the accounting file and `gzip' and `bzip2' will
not compress files with more than 1 link.

4.4 Hints
=========

4.4.1 High CPU load and "No buffer space available" messages
------------------------------------------------------------

This message in the debug log means that the kernel is delivering
packets faster than `ulog-acctd' can process them. This happens most
likely when `ulog-acctd' writes its accounting log to disk.

   There are a few steps that can be taken both in kernel space and in
`ulog-acctd'.
  1. Increase `socket receive buffer'. This requires that the kernel
     parameter `net/core/rmem_max' is set with `sysctl(8)' to allow for
     a larger socket receive buffer.

  2. Increase `hash table size'. However, be sure that the size of the
     hash table is a prime number.

  3. Set a nice value for the child process that writes the account log.

  4. Increase the `--ulog-qthreshold' value, if approtpiate

  5. Decrease the `--ulog-cprange' value, if approtpiate. Experience
     shows that 24 to 32 bytes are enough for normal IP traffic.

4.4.2 fdelay
------------

The `fdelay' configuration parameter can be used to reduce log file
size where `ulog-acctd' is used to account many long-lasting
connections and where it is not strictly necessary that the time values
are absolutely correct.

4.4.3 The dump file
-------------------

When `fdelay' is set to a non-zero value, `ulog-acctd' writes an
additional dump file every time it writes to the accounting log. This
dump file contains those records that have been kept in memory. If
`ulog-acctd' crashes before writing to the accounting log file, these
entries can still be recovered by the system administrator.

5 Credits
*********

`ulog-acctd' was written by Hilko Bengen
<bengen+ulog-acctd@hilluzination.de>. Source tarballs and public CVS
are accessible through
`http://alioth.debian.org/projects/pkg-ulog-acctd'.

Thanks go to
============

   * Marc Haber, for many useful ideas, and for pointing me in the right
     direction at times.

   * Fredrik Vang, for his code for formatted dates in log files.

   * Ulrich Callmeier, for `net-acct' which gave me a good base to
     start my work on. If `ulog-acctd' does not work for you, don't
     blame Ulrich or any other contributors mentioned in the source
     files, though.

   * Stefan Neufeind, for his useful ideas and data from operating
     `ulog-acctd' in environments with high network load. (traffic from
     game servers).

   * Patrick Schaaf, for coming up with a quick solution to
     synchronization issues, in the `de.comp.os.unix.programing'
     newsgroup.

   * Radek Podgorny, for his "since" patch, for logging the start of a
     flow.

   Initial work (up to version 0.4.1) on this program was funded by
toplink-plannet GmbH, Karlsruhe, Germany
`http://www.toplink-plannet.de/'.

