NewsCache

Due to serious time constraints, I unfortunately have no longer time to work on NewsCache. Thanks to Herbert Straub <h.straub at aon.at>, however, further development has been picked up again and can be found here.

Usenet News provides a global distributed blackboard on top of other networks. It consists of a set of hierarchical newsgroups which are dedicated to specific topics. Articles (messages) are submitted (posted) to one or more newsgroups and are replicated to all Usenet sites holding one of the newsgroups the article was posted to. The newsgroups that are stored on a news server and thus provided to its clients are defined by the news server's administrator.

The world-wide set of cooperating news servers makes up the distribution infrastructure of the News system. Articles are distributed among news servers using the Network News Transfer Protocol (NNTP) which is defined in RFC977. In recent years several extensions have been applied to NNTP. These are currently available as an Internet draft (draft-ietf-nntpext-base) which will supersede RFC977 within the next few months. The format of Usenet messages is defined in RFC1036.

For a detailed discussion of Usenet, please have a look at Chip Salzenberg, "What is Usenet?".

Problems

Usenet's main problem is its growth. The volume of news articles that has to be distributed doubles every 1.5 years (currently, about 15GB/day). This causes network load and a high I/O-load on news servers.

The result is that many organizations can no longer afford to run their own news server with a full news distribution. One solution to this problem is the introduction of caching.

For a deeper explanation, please consult Thomas Gschwind and Manfred Hauswirth, "A Cache Architecture for Modernizing the Usenet Infrastructure".

NewsCache, A Solution

The main idea of NewsCache is to separate the news access infrastructure from the distribution infrastructure.

NewsCache can be used in the following configurations:

As a Cache Server: NewsCache can be used in combination with a news server. In this setup news reading clients connect to NewsCache instead of to the news server. In a small test at our university, we got a hit rate of over 30%, although the interests of the people accessing NewsCache was relatively wide spread. This figure can be mapped onto a reduction of the load at the news server.
Instead of a Leaf Node News Server: NewsCache can replace a leaf node news server. Usually, news readers access only a small fraction of the articles available at the news server. NewsCache only stores those newsgroups and articles being accessed frequently, while still providing a virtual full feed. This reduces the network bandwidth between the former news server and the news feed and also hardware requirements (both in terms of disk space and CPU performance). Less CPU performance is sufficient because no local article numbering has to be performed and a smaller database has to be searched.
Multiplexing Between Different News Servers: NewsCache allows to retrieve news from different news servers. This allows you to merge the feed of multiple news servers into one virtual feed and thus your users do not have to configure multiple news servers in their news clients (which is not always possible). This is of especial interest of you frequently access vendor specific news servers.
Partitioning the Spool Among News Server: In case you need to provide a full news service, you might be benefit from NewsCache by partitioning the spool among different news servers and let NewsCache multiplex between them. This allows you to replace one powerful news server by several less powerful, but more cost effective news servers.
Provision of Local Newsgroups: NewsCache can also be used for the provision of local newsgroups without setting up and maintaining a news server with a full feed. To you use this feature, you simply set up a small news server that just holds the local newsgroups and let NewsCache multiplex between the local and the upstream news server.

Acknowledgements

We received a grant from USENIX (The Advanced Computing Systems Association) for improving NewsCache and its database organization.