Buxy rêve tout haut

On m’a récemment demandé s’il existait un outil Debian pour vérifier qu’un ensemble de paquets dérivés de Debian intègrent bien les dernières mises à jour de sécurité. L’idée est d’avoir un fichier listant l’ensemble des paquets personnalisés avec les versions qui ont servi de base aux paquets personnalisés et d’avoir un outil qui va vérifier automatiquement security.debian.org pour voir si de nouvelles versions sont disponibles.

Ma première réponse était de regarder famke un petit script python écrit par James Troup pour les besoins de l’équipe DSA de Debian: il prévient par mail lorsque des mises à jour sont possibles et essaient de les rapprocher avec le fil RDF des mises à jour de sécurité.

J’ai donc repris le principe et réalisé un petit script python un peu plus souple dans la mesure où il ne vérifie pas si les paquets installés peuvent être mis à jour mais si les paquets listés dans un fichier (que j’ai appelé watchlist) peuvent être mis à jour à partir d’un ensemble de dépôts APT que l’on consigne dans un fichier sources.list personnel.

Il peut servir d’exemples à ceux qui cherchent à employer l’interface “python-apt”. Le script est disponible dans un dépôt Git.

Installation et usage par l’exemple :

$ git clone git://git.debian.org/~hertzog/hacks/checkupdate.git
$ cd checkupdate
$ vim checkupdate.conf
# On change les répertoires pour pointer vers `pwd`
$ vim watchlist
# On indique les paquets à surveiller et leur version courante
$ ./checkupdate checkupdate.conf
An update of the package 'rsync' is available: 2.6.9-2etch1
Origin: [component: 'main' archive: 'stable' origin: 'Debian'
label: 'Debian-Security' site 'security.debian.org' isTrusted: 'True']

Following my previous article, I had some interesting comments introducing me to git-filter-branch (which is a new function coming from cogito’s cg-admin-rewritehist). This command is really designed to rewrite the history and you can do much more changes… it enabled me to fix the dates/authors/committers/logs of all the commits that were created with git_load_dirs. It can also be used to add one or more “parent commits” to any commit.

In parallel I discovered some problems with the git repository that I created: the tags were no more pointing to my master branch. This is because git rebase won’t convert them while rewriting history.

This lead me to redo everything from scratch. This time I used git-filter-branch instead. The man page even gives an example of how to link two branches together as if one was the predecessor of the other. Here’s how you can do it: let’s bind together “old” and “new”… the resulting branch will be “new-rewritten”.

$ git rev-parse old
0975870bb1631379f2da798fa78736a4fe32960a
$ git checkout new
$ git-filter-branch --tag-name-filter=cat --parent-filter \
"sed -e 's/^$/-p 0975870bb1631379f2da798fa78736a4fe32960a/'" \
new-rewritten
[...]
Rewritten history saved to the new-rewritten branch

Short explanation: the only commit without a parent commit (thus matching the empty regex “^$”) is the root commit and this one is changed to have a parent (-p) which is the last commit of the branch “old”.

At the end, you remove all the temporary branches, keep only what’s needed and repack everything to save space:


$ git branch -D old new
$ git prune
$ git repack -a -d


The dpkg team has a nice history of changing VCS over time. At the beginning, Ian Jackson simply uploaded new tarballs, then CVS was used during a few years, then Arch got used and up to now Subversion was used. When the subversion repository got created, the arch history has not been integrated as somehow the conversion tools didn’t work.

Now we’re likely to move over git for various reasons and we wanted to get back the various bits of history stored in the different VCS. Unfortunately we lost the arch repository. So we have disjoints bits of history and we want to put them all in a single nice git branch… git comes with git-cvsimport, git-archimport and git-svnimport, so converting CVS/SVN/Arch repositories is relatively easy. But you end up with several repositories and several branches.

Git comes with a nice feature called “git rebase” which is able to replay history over another branch, but for this to work you need to have a common ancestor in the branch used for the rebase. That’s not the case… so let’s try to create that common ancestor! Extracting the first tree from the newest branch and committing it on top on the oldest branch will give that common ancestor because two identical trees will have the same identifier. Using git_load_dirs you can easily load a tree in your git repository, and “git archive” will let you extract the first tree too.

In short, let’s see how I attach the “master” branch of my “git-svn” repository to the “master” branch of my “git-cvs” repository:

$ cd git-svn
$ git-rev-list --all | tail -1
0d6ec86c5d05f7e60a484c68d37fb5fc31146c40
$ git-archive --prefix=dpkg-1.13.11/ 0d6ec86c5d05f7e60a484c68d37fb5fc31146c40 | (cd /tmp && tar xf -)
$ cd ../git-cvs
$ git checkout master
$ git_load_dirs -L"Fake commit to link SVN to older CVS history" /tmp/dpkg-1.13.11
[...]
$ git fetch ../git-svn master:svn
$ git checkout svn
$ git rebase master

That’s it, your svn branch now contains the old cvs history. Repeat as many times as necessary…

This is a never ending story for me. The first time I’ve had problems with Linux’s handling of serial UART dates back to 2005 (see my previous blog post on buffer overruns). At that time I could improve the situation by applying two patches (kernel-preempt and low latency).

One year later, I have a situation where the buffer overruns are again easily reproducible at slower baudrate (54 kbauds), arguably there’s more than a serial application running this time, and it looks like the load generated by other processes (mainly watching digital I/O) renders the system less reliable with respect to its handling of serial ports.

This time I follow the advice to try out the 2.6 kernel because many “real-time improvements” (coming from the -rt branch, check its wiki) and “embedded improvements” (coming from linux-tiny) have been merged.

So I tried the stock kernel with very bad result. Results are better than the stock 2.4 but they are worse than the patched 2.4.
So I decide to try the -rt patch on 2.6, but this patch doesn’t work on my CPU card and my bugreport didn’t lead to any fix (nobody responded even though I tried hard to include the necessary information and I was ready to do whatever I would have been asked to try).

At the same time I explain my problem on the linux-kernel mailing list.
The discussion doesn’t answer my questions but still brings two ideas to try out. In the end, with two simple tweaks to the stock 2.6 kernel (mainly configuring the UART to send the interrupt as soon as the first character arrives, instead of waiting for 8 chars to accumulate in the FIFO) I have been able to get something better than the patched 2.4. And it turns out my first choices for the 2.6 kernel configuration have not been very wise so the comparison between stock 2.4 and stock 2.6 above doesn’t mean much.

Unfortunately, even if better than the patched Linux 2.4, it still doesn’t give good results in some conditions. So my primary question remains: is there a way to patch my kernel so that it will handle the serial related tasks (servicing interrupts from the UART mainly) as its primary job ? I don’t mind if such a change impact negatively the speed of the system if I can make sure that my serial exchanges are reliable.

And by reliable I mean of course no buffer overruns, but there’s a second similar problem that has been discovered: when using software handshake, the system can send out between 10 and 25 characters after the partner has sent its XOFF. Sending up to 6 characters after the XOFF is ok, more is asking for troubles because the UART on the other side will probably encounter a buffer overrun…

If you have any idea on how to resolve my problems, by any means, let me know.

J’ai récemment été confronté à la problématique de devoir détecter (dans une application C sous Linux) si une socket employée avait été fermée sans être obligé de lire les données en attente dans celle-ci.

Il n’existe (à ma connaissance) aucun moyen portable de réaliser cela. Mais Linux offre (via l’appel getsockopt) la possibilité de consulter la structure tcp_info du noyau (voir /usr/include/linux/tcp.h). Voici comment le test a été réalisé:

struct tcp_info info;
socklen_t len = sizeof(info);
if (getsockopt(remote_s, SOL_TCP, TCP_INFO, &info, &len) != -1) {
    if (info.tcpi_state == TCP_CLOSE ||
        info.tcpi_state == TCP_CLOSE_WAIT ||
        info.tcpi_state == TCP_CLOSING) {
                [... do what you want here ...]
    }
}

Working with serial lines can sometimes give you big headaches. I have an embedded PC based on 386 SX 40 processor. This PC doesn’t make much but it has programs using the serial line intensively. Things didn’t work as well as expected so I looked carefully what was going on … the beast was loosing bytes ! My information has been promptly confirmed by the /proc/tty/driver/serial entry. If you have “oe: X” (where X is a positive number) there, it means that one of your UART detected overrun errors.

So what’s an overrun ? An overrun happens when the UART receives data while its FIFO buffer is full. Why is the FIFO full ? Because Linux didn’t treat the serial interruption quickly enough. Why is Linux so slow ? Linux is not a real time OS and it doesn’t guarantee any response time to interruption, so Linux is not so slow but my PC really is … what happens is that interruption related to the network are treated before serial interruptions. Furthermore IDE disk interruptions can take too long too. Worst case is of course, you’re treating a disk interruption, then you have to treat the network interruption and only after that you can treat the serial interruption which in fact happened right after the beginning of the disk interruption

So fixing serial overrun is a rather complex problem since it’s really a kernel related problem. Googling on the subject I have found several ideas to explore :

• configure IDE disk to use DMA hdparm -d 1 /dev/hda, use of DMA will shorten the time where IRQ will be masked to the kernel (in my case it doesn’t work since I’m using DiskOnModule which do not support DMA)
• make disk IRQ interruptible with hdparm -u 1 /dev/hda
• use irqtune to re-prioritize the IRQ on the interruption controller. This software is no more maintained and it doesn’t work out of the box on kernel 2.4.x.

Using hdparm -u wasn’t enough to solve my problem… so I continued to look for a solution and I found one ! I recompiled my kernel with the low latency patch and the preemptible kernel patch. Those are usually used for multimedia applications where you need good responsiveness in order to deliver content in real-time but the fact is that they do work for my purpose too !

My serial overruns are completely gone at 9600 bauds. However I can still have some when running at 115200 bauds. Moreover I can create serial overrun by running a find / -type f | grep -v /proc/ | xargs md5sum in the background… I can’t make miracles with this slow processor… if you have more ideas to further improve the situation, I’m willing to try !

Je cherchais un moyen simple d’intégrer un feed RSS dans une page WML (comme Website Meta Language et non pas le langage du WAP) et je ne trouvais rien de prêt à l’emploi. Étant donné que WML emploie perl de manière importante, je me suis dit que je pourrai écrire les quelques lignes de Perl qui vont bien. J’emploie plusieurs modules dont XML::RSS, pensez donc à l’installer au préalable (paquet libxml-rss-perl dans Debian). Découvrez le code dans la suite !

Lire le reste de cet article »

Le thème par défaut de Wordpress 1.5 a de nombreux défauts malgré sa belle apparence. Petit à petit j’apprends à les dépasser.

Le problème qui m’a occupé aujourd’hui est très simple. La liste de pages située sur la droite est triée par défaut selon le titre… or je voulais trier manuellement, j’ai donc changé le numéro d’ordre de chacune des pages dans l’interface d’administration. Puis j’ai cherché dans le thème comment ce bout de HTML est généré. On trouve ainsi une fonction wp_list_pages dans le fichier patron correspondant à la barre de côté (sidebar.php)… en lisant le code source de cette fonction (fichier wp-includes/template-functions-post.php) je me suis aperçu qu’il était possible de passer des paramètres supplémentaires et notamment un champ qui s’appelle sort_column (colonne de tri). Voici donc le petit bout de code à mettre pour trier selon numéro d’ordre croissant :

wp_list_pages('title_li=<h2>' . __('Pages') . '</h2>&sort_column=menu_order' );

Pour trier par ordre décroissant, il serait possible de rajouter un paramètre supplémentaire sort_order=DESC. Et voilà.