Package Tracking System
Hacking's guide

By Raphaël Hertzog <hertzog@debian.org>

September 2005 in Darmstadt

Debian-QA mini-conference

Plan

Introduction

Where to get the sources

Directory overview

The mail part

The web part

What's left ? What you can do ...

Introduction

The Package Tracking System lets you follow everything related to the life of a package. It's of interest for co-maintainers, advanced users, QA members, ...

With the mail interface you can subscribe to “packages” and receive in real-time the same information than the maintainer (bugs, upload notification, cvs/svn commits, ...).

The web interface displays the actual status of the package and some historical information.
Example: http://packages.qa.debian.org/dpkg

Some figures

Almost 4 years old

3300 source packages with subscribers

6300 subscriptions (with 3000 unique emails)

Several hundreds of megabytes of HTML pages

The sources

The sources are hosted on cvs.debian.org

export CVSROOT=:pserver:anonymous@cvs.debian.org/cvs/qa

cvs login

cvs co pts

Web interface: http://cvs.debian.org/pts/?cvsroot=qa

Live on master.debian.org:/org/packages.qa.debian.org/

Directory overview

/bin: Perl scripts for the mail interface

/control.pl: 360 lines to handle pts@qa.debian.org

/dispatch.pl: 200 lines to handle *@packages.qa.debian.org

/etc: Configuration (a template for the confirmation mail)

/perl: Common code (350 more lines)

/mail: Exim configuration for p.q.d.o

/www: Web interface

Mail interface

Two berkeley databases are used to store the subscriptions (/db/subscriptions.db and /db/tags.db)

Script dispatch.pl

Receives mails from several sources (BTS, katie, users, ...), identifies them (“keyword/tag”), associates them to a source package and forwards them to the subscribers

Script control.pl

Identify commands within the mail and execute them

Handle separate confirmation mails (stores “secret” hashes in /spool)

Script dump.pl, dump-tags.pl, count.pl.

Display a textual version of the database and some statistics

Nice features of the mail interface

Mails sent to the PTS are classified so that one can subscribe only to the information that is of interest for him

The user can filter everything with the help of the X-PTS-Package and X-PTS-Keyword header.

We can always send more info to the PTS by introducing new “keywords/tags”

What's lacking ?

VERP Bounce handling

owner@p.q.d.o (me) is receiving all the bounces and they are either not treated, or treated manually (which is time consuming)

An automatic (monthly or quarterly) mail giving a summary of the state of the package

Number of bugs opened/closed since last summary

List of actual problems

...

Web interface

Directory overview (/www)

/bin: Python scripts (1400 lines in a dozen files)

/incoming: Raw data after download

/base: Reference data generated by the scripts (mostly XML and emails)

/xsl/pts.xsl: XSLT stylesheet

/web: Generated HTML files

/cgi-bin: 404 handling for redirection, online subscription form

/etc: mhonarc config file (for html conversion of emails)

Architecture

Raw data

Directory “incoming”

downloaded by

update_incoming.sh

XML + Emails

Directory “base”

*_to_xml.py

HTML

Directory “www”

generate_html.sh (XSLT)

received by MTA (exim)

receive_auto.py

receive_news.py

sources_to_xml.py

excuses_to_html.py

other_to_xml.py

Everything launched 3 times/day by bin/do_all.sh

Nice features of the web interface

You can put your own content on the web page in the “Latest news” section or in a “Static news” section.

Put a link to the upstream web site

Indicate where the CVS/SVN is

This is done by sending a mail to pts-news@qa.debian.org or pts-static-news@qa.debian.org

Check the developers reference for details !

Ideas for the future

The PTS is meant to be a real TODO list for the packages maintainers

Any “know problem” or any “possible amelioration” should be listed :

lack of debtags information

other examples, recently added:

availability of Ubuntu patches

availability of patches in the BTS

Conclusion - Questions

The PTS has become in 4 years an important part of Debian's infrastructure

It's not a very complex software but very few people looked at it.

We need to have more people who are able to maintain it.

spread the knowledge

add some (human) redundancy

share the workload

implement new ideas

2005-06-29T10:24:57