Tag Archives: retrospective

grep-dctrl is ten years old

My message Intent to package a Debian control file grepper to WNPP and debian-devel is today a decade old. Apparently, the message predates the invention of the acronym ITP for Intent to Package (the first instance I can find is from May 1999).

The changelog reveals that the first upload was on March 1st, 1999. There is unfortunately no record of when the package hit unstable the first time, since dinstall did not send Installed (or even the later Accepted) mails to an archived mailing list at that time. I suppose I might have it in some old private email archive, but more likely it’s just gone. Similarly, the first fixed bug (#35527) predates BTS archiving and is now lost.

The grep-dctrl program came out of repeated awkward grepping of the dpkg available and status files. Eventually I decided there must be a better way, and failed to find any canned solutions (I was later pointed to sgrep, which didn’t look useful enough, and even later to dpkg-awk, but I had already committed to my own solution by then). I wrote a simple C program that processed these files as a sequence of records and did simple substring searches in each record. I rapidly added support for field selection, regular expressions and output field selection. By version 1.3a of March 2000 (which was released with Debian 2.2 ‘potato’), the program was as good as it was going to get – with one exception.

“Make disjunctive searches possible,” said my TODO file those days. Conjunctive searches (that is, AND-searches) were possible even then by using more than one grep-dctrl command in a pipeline. Disjunctive (OR) was not possible. The problem was not so much that it would be hard to program (although the program’s internal structure wasn’t very good, to be honest, making extensive changes difficult), it was more a problem of coming up with a good command line syntax.

Another thing that bothered me with the old grep-dctrl was how to implement Ben Armstrong’s feature request. Again, the programming part wasn’t the problem, the problem was coming up with a good, clean semantics for the feature.

It was finally the appearance of ara in 2003 that got me moving again. Ara’s author proudly compared eir program to grep-dctrl, claiming that my program did not do disjunctive searches while ara does. Competition being good for the soul, I took it as a challenge. In April 2003 I announced a complete rewrite of grep-dctrl, which was completed in January 2004 (the 2.0 release).

The rewrite changed the way the command line was handled – even though the usual Unix switch style is still used, the command line is regarded as a language with a parser (first an operator-precedence parser, then a recursive-descent one). The command line is transformed into an interpreted stack-based language which drives the actual grepping.

The rewrite also generalised the internal data structures into an internal library which could be used to write other tools. The first such tool was sort-dctrl (introcuded in 2.7, June 2005), which was soon followed by tbl-dctrl (2.8, July 2005). The later appearance of join-dctrl (2.11, August 2007) finally allowed me to close Ben Armstrong’s longstanding feature request mentioned above.

The unpronounceable part of the names, “dctrl”, is an abbreviation for “Debian control”, which I decided to call the file format used by dpkg. Some people call it a RFC-822 format, but that really is a misnomer, since the differences between dctrl and RFC-822 outweigh the mainly superficial similarities (and the historical connection). I did consider calling my program dpkg-grep, but I didn’t feel like I had the right to invade the dpkg namespace. The later rename to dctrl-tools reflects the fact that there are now several tools, grep-dctrl being just the oldest.

I have several plans for the dctrl-tools suite, but my time and energy are mostly claimed by other responsibilities. The suite is currently team-maintained, but unfortunately the team is not very active. I would love it if I weren’t the most active one with my busy schedule! Feel free to pop in on the dctrl-tools-devel mailing list, and to look at the Git repository and the todo list. If you decide to participate, please follow the rules.