                              Snowlist 1.0 Readme
          (Last update: $Id: README 293 2004-10-30 15:18:00Z kiza $)

Snowlist is a collection of Perl scripts that can generate an RDF (RSS 1.0)
feed from a mailing list. The resulting feeds will contain one news item per 
thread and all postings that belong to a thread in a bulletin board like 
view. See this short example.

Let's say there is a discussion with the subject "Snowlist is cool" on the 
mailing list you want to use. From this discussion Snowlist will generate 
one news item with the following content:

------
Subject:  Snowlist is cool

From:     Snowlist evangelist
Date:     Some date

It's cool!

=====================================================

From:     Non believer
Date:     Later

> It's cool!

And what if I don't believe you?

=====================================================

From:     Snowlist evangelist
Date:     Later still

But you must believe me! I wrote it and I must know!

Oh, and did I mention that snownews is the perfect
RSS reader to use it with?

=====================================================
------

So what you get is a chronological list of postings for each single 
thread. If you want you can configure the maximum number of threads in the 
feed and the maximum number of items per thread. If either of these values 
are exceeded by the number of messages the oldest are dropped from the feed.


System requirements
===================

PostgreSQL
You need to install PostgreSQL. It's the better database anyway. ;)
(http://www.postgresql.org/)

You can use MySQL if you change the SQL in both scripts to work with 
MySQL. Same goes for any other database.

Snowlist requires a (moderate) number of Perl modules from CPAN. They are 
all required, sorry.

DBI
Common database module (http://search.cpan.org/search?module=DBI)

XML::LibXML
LibXML access (http://search.cpan.org/search?module=XML::LibXML)

MIME::Tools
MIME decoding (http://search.cpan.org/search?module=MIME::Tools)

Text::Iconv
Iconv access (http://search.cpan.org/search?module=Text::Iconv)

Digest::MD5
MD5 message digest support (http://search.cpan.org/search?module=Digest::MD5)

Depending on how you want to feed mails into the database you need 
procmail or another mail processor.


Installation & Configuration
============================

I'd recommend adding a dedicated user on your system that receives all the 
mailing list mail to make processing easier. I'll assume you're using 
the user "snowlist" through the rest of this manual.

Create the database table now. You can use the included script 
createdb.DBNAME.sql to create the table Snowlist uses, or you can use
the script as a reference and create the table manually. Either way, in 
the end you must have a table that looks like in the sql example file and 
a user that can access it.

Before running the script edit it and change $YOUR_DB_USER to the desired 
value!

If you make it work on a database other than the included creation scripts 
it would be nice if you could send me a createdb.DBNAME.sql script. Right 
now there are scripts for: PostgreSQL.

Put the three scripts 'snowlistDB', 'snowlistOutput' and 'SnowlistConfig.pm' 
into ~snowlist/bin (or wherever you want them). Create a config directory 
~snowlist/.snowlist/ and create the main config file, named 'config' 
there. See the included file 'config.example' how this should look like. 
The section names do not have any meaning to snowlist.

compressoutput      generated feeds will be compressed with gzip. Useful to 
                    lower server load and bandwidth consumption. Value is 
                    "yes" or "no".
compressionlevel    zlib compression level: 0 (fastest) - 8 (highest)

dbusername          username for database login

dbpassword          password for database login

maxthread           maximum number of threads the RSS feed will contain

maxitemsperthread   maximum number of mails per thread

msg_expire          maximum age of messages kept. See section Database
                    cleanup below.


If you have created this file you can start to add your first mailing 
list. I'd recommend to subscribe to the list, before you actually set up 
everything so you'll get the sent mails delivered to the user snowlist's 
standard mailbox and can verify that everything works well.

I assume that the mailinglist now forwards mails to the user snowlist and 
everything works fine.

Create a config for the list in ~snowlist/.snowlist/. You will need to
choose a name for it that you need to use in various places from now on. I
will call this ListID. We choose the name "testlist" for now. Create a file
named 'testlist.config'. See the included file 'examplefeed.config' to get
an idea how it should look like. 

title          the title of the resulting feed

link           the homepage of this feed. You can set this to the mailing 
               list's home page for example.

description    description of this feed the RSS reader will show

outputfile     where to write the resulting feed. This location must be 
               writable to the user "snowlist" (or whatever user you chose).

maxthreads            (optional) You can override the default settings
maxitemsperthreas     in the main configuration for this specific feed.


You now need to redirect the list mail to the script 'snowlistDB'. The 
most convenient way to do this is procmail I would say. Add a .forward 
file for the user "snowlist" and put "|/path/to/procmail" in there. Add a 
.procmailrc that contains a filter for the mailing list you subscribed. 
This filter must pipe its mail to "~snowlist/bin/snowlistDB testfeed". The 
first (and only) argument given to "snowlistDB" must be the ListID of this 
feed. It's the same as the name of the config file for this feed. If you 
mix them up it will not work!

If you did everything right, mail that will arrive at snowlist@your.domain 
will now be written into the database. You can now generate the RSS feed.

Run "~snowlist/bin/snowlistOutput testfeed" (once again the first and only 
argument is the ListID) and the script will write the feed to the location 
you gave in the config file "testfeed.config".

You should add the last command to the user's crontab to regenerate the 
feed hourly or in a desired interval. If you specify "-" as the 
outputfile in the config "snowlistOutput" will write the feed to standard 
output for further processing with other programs.

That should be all that is needed to get everything running. Please read 
the next section! It contains various things to consider before running a 
publicly accessable feed.


Notes & Warnings
================

Make sure the list owners allow public archiving of their list. If it's a 
private list you might run into trouble otherwise. :)

All email addresses are stripped from the output and replaced by 
[EMAIL REMOVED]. Don't change this! Well, if you really know what you are 
doing you can find the relevant lines that do this in "snowlistDB".

Be aware that list software like Mailman sends password reminders. Either 
disable this after you've subscribed or hack up a procmail filter that 
removes these mails from the usual list mail.

It will be difficult for the list administrator to get in touch with you. 
It would be a good idea to choose an email address that tells the list 
admin that this might be a machine that is subscribed. Something like 
snowlist+lkml_relay@your.domain or something. Though this is difficult to 
achieve I guess. Try not to annoy list admins too much and act quickly if 
messages are bouncing!

!!! IMPORTANT !!!
If you use procmail put the option "DELIVERED=YES" into the .procmailrc of
the user who is running snowlist. With this option procmail will always
return success to the MTA and never cause a bounce when a script in the
chain fails. Worst thing that will happen is a list mail becoming lost.
Still it's better than to cause bounces. 

Put the directive "MAILTO=[your username]" into snowlist's crontab. "Your 
username" being the user that receives your mail on the system. If the 
script that generates the feed fails you will receive the output. 
Otherwise, depending on your mail filtering configuration, the failure 
notice mail may show up in the RSS feed.


Database cleanup
================

After a while the database table will be filled with a lot of messages. You
should run "snowlistExpire" via cron daily/weekly/monthly to clean up old
messages. The maximum age at where messages are deleted from the database
is 30 days. You can configure this age in the main config file with the
directive "msg_expire". The Syntax is PostgreSQL syntax.

Self explanatory examples:
30 days
2 months
1 year 6 months 3 weeks 5 days 10 hours 4 minutes 42 seconds


AUTHORS
=======

Snowlist was written and designed by Oliver Feiler <kiza@gmx.net> with the 
help of Ren Puls <rpuls@kcore.de>. If you need to contact the maintainer, 
write Oliver. As the time of this writing he spoke English and German. 


Have fun!

And don't forget to http://kiza.kcore.de/software/donate and try out 
http://snownews.kcore.de with Snowlist's generated feeds!
