Category Archives: English

Alue status report

In May, I posted about the discussion forum software I am writing, Alue. Since then –

What works:

  • The NNTP interface is essentially complete.
  • There is a rudimentary reading and posting HTTPS interface .
  • Users are able to self-register, manage their own accounts and reset lost passwords using an email challenge system.

What is broken:

  • NNTP control messages.
  • MIME message display in HTTPS (including character set conversions)
  • Web design. Although – all HTML is template-generated, so it’s more a problem with the test installation than with the actual software.

What is missing:

  • HTTPS-based administration
  • Moderation
  • Spam control
  • Email distribution of messages
  • Posting by email
  • Packaging (including proper installation and upgrade procedures)

And it would probably use a proper security review of the code.

If you are interested, go check out the test installation. The code and the test installation templates are available through Git. If you are really brave (and are a skilled system administrator), you might try creating your own installation – if you do, let me know.

Eric Flint on copyright and DRM

[Originally posted in June 2006; updated with new links several times, most recently in February 2010]

Eric Flint: A Matter of Principle, Jim Baen’s Universe 1 (1), 2006.
Eric Flint: Copyright: What Are the Proper Terms for the Debate?, Jim Baen’s Universe 1 (2), 2006.
Eric Flint: Copyright: How Long Should It Be?, Jim Baen’s Universe 1 (3), 2006.
Eric Flint: What is Fair Use, Jim Baen’s Universe 1 (4), 2006.
Eric Flint: Lies, and More Lies, Jim Baen’s Universe 1 (5), 2007.
Eric Flint: There Ain’t No Such Thing as a Free Lunch, Jim Baen’s Universe 1 (6), 2007.
Eric Flint: Books: The Opaque Market, Jim Baen’s Universe 2 (1), 2007.
Eric Flint: Spillage: or, The Way Fair Use Works in Favor of Authors and Publishers, Jim Baen’s Universe 2 (2), 2007.
Eric Flint: The Economics of Writing, Jim Baen’s Universe 2 (3), 2007.
Eric Flint: The Pig-in-a-Poke Factor, Jim Baen’s Universe 2 (4), 2007.
Eric Flint: Paper books are not going to be joining the dodo any time soon. If ever., Jim Baen’s Universe 2 (5), 2008
Eric Flint: A Matter of Symbiosis. Jim Baen’s Universe 2 (6), 2008
Eric Flint: The Nature of Transitions. Jim Baen’s Universe 3 (1), 2008
Eric Flint: Adventures with a Search Engine. Jim Baen’s Universe 3 (2), 2008
Eric Flint: The Problem is Legal Scarcity, not Illegal Greed. Jim Baen’s Universe 3 (3), 2008
Eric Flint: Foam and Froth and Mighty (Upside-down) Pyramids. Jim Baen’s Universe 3 (4), 2009
Eric Flint: The Internet is Not a Magic Wand. Jim Baen’s Universe 3 (5), 2009

The column series seems to have not been continued.

Eric Flint is a fairly successful sf author. These columns explore the evils of Digital Restrictions Management (DRM, also known as Don’t Read Me).

This is Alue

I have made a couple of references in my blog to the new software suite I am writing, which I am calling Alue. It is time to explain what it is all about.

Alue will be a discussion forum system providing a web-based forum interface, a NNTP (Netnews) interface and an email interface, all with equal status. What will be unusual compared to most of the competition is that all these interfaces will be coequal views to the same abstract discussion, instead of being primarily one of these things and providing the others as bolted-on gateways. (I am aware of at least one other such system, but it is proprietary and thus not useful to my needs. Besides, I get to learn all kinds of fun things while doing this.)

I have, over several years, come across many times the need for such systems and never found a good, free implementation. I am now building this software for the use of one new discussion site that is being formed (which is graciously willing to serve as my guinea pig), but I hope it will eventually be of use to many other places as well.

I now have the first increment ready for beta testing. Note that this is not even close to being what I described above; it is merely a start. It currently provides a fully functional NNTP interface to a rudimentary (unreliable and unscalable) discussion database.

The NNTP server implements most of RFC 3977 (the base NNTP spec – IHAVE, MODE-READER, NEWNEWS and HDR are missing), all of RFC 4642 (STARTTLS) and a part of RFC 4643 (AUTHINFO USER – the SASL part is missing). The article database is intended to support – with certain deliberate omissions – the upcoming Netnews standards (USEFOR and USEPRO), but currently omits most of the mandatory checks.

There is a test installation at verbosify.org (port 119), which allows anonymous reading but requires identification and authentication for posting. I am currently handing out accounts only by invitation.

Code can be browsed in a Gitweb; git clone requests should be directed to git://git.verbosify.org/git/alue.git/.

There are some tweaks to be done to the NNTP frontend, but after that I expect to be rewriting the message filing system to be at least reliable if not scalable. After that, it is time for a web interface.

Asynchronous transput and gnutls

CC0
To the extent possible under law,
Antti-Juhani Kaijanaho has waived all copyright and related or neighboring rights to
Asynchronous transput and gnutls. This work is published from Finland.

GnuTLS is a wonderful thing. It even has a thick manual – but nevertheless its documentation is severely lacking from the programmer’s point of view (and there doesn’t even seem to be independent howtos floating on the net). My hope is to remedy with this post, in small part, that problem.

I spent the weekend adding STARTTLS support to the NNTP (reading) server component of Alue. Since Alue is written in C++ and uses the Boost ASIO library as its primary concurrency framework, it seemed prudent to use ASIO’s SSL sublibrary. However, the result wasn’t stable and debugging it looked unappetizing. So, I wrote my own TLS layer on top of ASIO, based on gnutls.

Now, the gnutls API looks like it works only with synchronous transput: all TLS network operations are of the form “do this and return when done”; for example gnutls_handshake returns once the handshake is finished. So how does one adapt this to asynchronous transput? Fortunately, there are (badly documented) hooks for this purpose.

An application can tell gnutls to call application-supplied functions instead of the read(2) and write(2) system calls. Thus, when setting up a TLS session but before the handshake, I do the following:

                gnutls_transport_set_ptr(gs, this);
                gnutls_transport_set_push_function(gs, push_static);
                gnutls_transport_set_pull_function(gs, pull_static);
                gnutls_transport_set_lowat(gs, 0);

Here, gs is my private copy of the gnutls session structure, and the push_static and pull_static are static member functions in my sesssion wrapper class. The first line tells gnutls to give the current this pointer (a pointer to the current session wrapper) as the first argument to them. The last line tells gnutls not to try treating the this pointer as a Berkeley socket.

The pull_static static member function just passes control on to a non-static member, for convenience:

ssize_t session::pull_static(void * th, void *b, size_t n)
{
        return static_cast<session *>(th)->pull(b, n);
}

The basic idea of the pull function is to try to return immediately with data from a buffer, and if the buffer is empty, to fail with an error code signalling the absence of data with the possibility that data may become available later (the POSIX EAGAIN code):

class session
{
        [...]
        std::vector<unsigned char> ins;
        size_t ins_low, ins_high;
        [...]
};
ssize_t session::pull(void *b, size_t n_wanted)
{
        unsigned char *cs = static_cast<unsigned char *>(b);
        if (ins_high - ins_low > 0)
        {
                errno = EAGAIN;
                return -1;
        }
        size_t n = ins_high - ins_low < n_wanted 
                ?  ins_high - ins_low 
                :  n_wanted;
        for (size_t i = 0; i < n; i++)
        {
                cs[i] = ins[ins_low+i];
        }
        ins_low += n;
        return n;
}

Here, ins_low is an index to the ins vector specifying the first byte which has not already been passed on to gnutls, while ins_high is an index to the ins vector specifying the first byte that does not contain data read from the network. The assertions 0 <= ins_low, ins_low <= ins_high and ins_high <= ins.size() are obvious invariants in this buffering scheme.

The push case is simpler: all one needs to do is buffer the data that gnutls wants to send, for later transmission:

class session
{
        [...]
        std::vector<unsigned char> outs;
        size_t outs_low;
        [...]
};
ssize_t session::push(const void *b, size_t n)
{
        const unsigned char *cs = static_cast<const unsigned char *>(b);
        for (size_t i = 0; i < n; i++)
        {
                outs.push_back(cs[i]);
        }
        return n;
}

The low water mark outs_low (indicating the first byte that has not yet been sent to the network) is not needed in the push function. It would be possible for the push callback to signal EAGAIN, but it is not necessary in this scheme (assuming that one does not need to establish hard buffer limits).

Once gnutls receives an EAGAIN condition from the pull callback, it suspends the current operation and returns to its caller with the gnutls condition GNUTLS_E_AGAIN. The caller must arrange for more data to become available to the pull callback (in this case by scheduling an asynchronous write of the data in the outs buffer scheme and scheduling an asynchronous read to the ins buffer scheme) and then call the operation again, allowing the operation to resume.

The code so far does not actually perform any network transput. For this, I have written two auxiliary methods:

class session
{
        [...]
        bool read_active, write_active;
        [...]
};
void session::post_write()
{
        if (write_active) return;
        if (outs_low > 0 && outs_low == outs.size())
        {
                outs.clear();
                outs_low = 0;
        }
        else if (outs_low > 4096)
        {
                outs.erase(outs.begin(), outs.begin() + outs_low);
                outs_low = 0;
        }
        if (outs_low < outs.size())
        {
                stream.async_write_some
                        (boost::asio::buffer(outs.data()+outs_low,
                                             outs.size()-outs_low),
                         boost::bind(&session::sent_some,
                                     this, _1, _2));
                write_active = true;
        }
}

void session::post_read()
{
        if (read_active) return;
        if (ins_low > 0 && ins_low == ins.size())
        {
                ins.clear();
                ins_low = 0;
                ins_high = 0;
        }
        else if (ins_low > 4096)
        {
                ins.erase(ins.begin(), ins.begin() + ins_low);
                ins_high -= ins_low;
                ins_low = 0;
        }
        
        if (ins_high + 4096 >= ins.size()) ins.resize(ins_high + 4096);

        stream.async_read_some(boost::asio::buffer(ins.data()+ins_high,
                                                   ins.size()-ins_high),
                               boost::bind(&session::received_some,
                                           this, _1, _2));
        read_active = true;
}

Both helpers prune the buffers when necessary. (I should really remove those magic 4096s and make them a symbolic constant.)

The data members read_active and write_active ensure that at most one asynchronous read and at most one asynchronous write is pending at any given time. My first version did not have this safeguard (instead trying to rely on the ASIO stream reset method to cancel any outstanding asynchronous transput at need), and the code sent some TLS records twice – which is not good: sending the ServerHello twice is guaranteed to confuse the client.

Once ASIO completes an asynchronous transput request, it calls the corresponding handler:

void session::received_some(boost::system::error_code ec, size_t n)
{
        read_active = false;
        if (ec) { pending_error = ec; return; }
        ins_high += n;
        post_pending_actions();
}
void session::sent_some(boost::system::error_code ec, size_t n)
{
        write_active = false;
        if (ec) { pending_error = ec; return; }
        outs_low += n;
        post_pending_actions();
}

Their job is to update the bookkeeping and to trigger the resumption of suspended gnutls operations (which is done by post_pending_actions).

Now we have all the main pieces of the puzzle. The remaining pieces are obvious but rather messy, and I’d rather not repeat them here (not even in a cleaned-up form). But their essential idea goes as follows:

When called by the application code or when resumed by post_pending_actions, an asynchronous wrapper of a gnutls operation first examines the session state for a saved error code. If one is found, it is propagated to the application using the usual ASIO techniques, and the operation is cancelled. Otherwise, the wrapper calls the actual gnutls operation. When it returns, the wrapper examines the return value. If successful completion is indicated, the handler given by the application is posted in the ASIO io_service for later execution. If GNUTLS_E_AGAIN is indicated, post_read and post_write are called to schedule actual network transput, and the wrapper is suspended (by pushing it into a queue of pending actions). If any other kind of failure is indicated, it is propagated to the application using the usual ASIO techniques.

The post_pending_actions merely empties the queue of pending actions and schedules the actions that it found in the queue for resumption.

The code snippets above are not my actual working code. I have mainly removed from them some irrelevant details (mostly certain template parameters, debug logging and mutex handling). I don’t expect the snippets to compile. I expect I will be able to post my actual git repository to the web in a couple of days.

Please note that my (actual) code has received only rudimentary testing. I believe it is correct, but I won’t be surprised to find it contains bugs in the edge cases. I hope this is, still, of some use to somebody :)

Star Trek

It is curious to see that the eleventh movie in a series is the first to bear the series name with no adornment. It is apt, however: Star Trek is a clear attempt at rebooting the universe and basically forgetting most of the decades-heavy baggage. It seems to me that the reboot was fairly well done, too.

The movie opens with the birth of James Tiberius Kirk, and follows his development into the Captain of the Enterprise. Along the way, we also see the growth of Spock from adolescence into Kirk’s trusted sidekick and also into … well. Despite the fact that the action plot macguffins are time travel and planet-killer weaponry, it is mainly a story of personal vengeance, personal tragedy, and personal growth. Curiously enough, although Kirk gets a lot of screen time, it is really the personal story of Spock.

Besides Kirk and Spock, we also get to meet reimagined versions of Uhura (I like!), McCoy, Sulu, Chekov and Scott. And Christopher Pike, the first Captain of the Enterprise. The appearance of Leonard Nimoy as the pre-reboot Spock merits a special mention and a special thanks.

I overheard someone say in the theatre, after the movie ended, that the movie was a ripoff and had nothing to do with anything that had gone before. I respectfully disagree. The old Star Trek continuum had been weighed down by all the history into being a 600-pound elderly man who is unable to leave the couch on his own. This movie provided a clearn reboot, ripping out most of the baggage, retaining the essence of classic Star Trek and giving a healthy, new platform for good new stories. One just hopes Paramount is wise enough not to foul it up again.

It was worth it, I thought.

Anecdote Nr 1 of implementing a NNTP (reading) server

The first version of my work-in-progress NNTP server I tested with a real newsreader had enough code to handle connections, give the greeting and reject every command except QUIT (whose implementation was fully RFC 3977 conforming).

Here’s an excerpt from my server log when giving tin the Treatment:

2009-05-02T01:05:49 nntpd for 127.0.0.1:55432 terminating
2009-05-02T15:24:31 nntpd listening at 0.0.0.0:5555
2009-05-02T15:24:46 nntpd for 127.0.0.1:34844 starts
2009-05-02T15:24:46 nntpd for 127.0.0.1:34844 ==> 200 posting allowed
2009-05-02T15:24:46 nntpd for 127.0.0.1:34844 <== CAPABILITIES
2009-05-02T15:24:46 nntpd for 127.0.0.1:34844 ==> 500 unrecognized command
2009-05-02T15:24:46 nntpd for 127.0.0.1:34844 <== MODE READER
2009-05-02T15:24:46 nntpd for 127.0.0.1:34844 ==> 500 unrecognized command
2009-05-02T15:24:46 nntpd for 127.0.0.1:34844 <== CAPABILITIES
2009-05-02T15:24:46 nntpd for 127.0.0.1:34844 ==> 500 unrecognized command
2009-05-02T15:24:46 nntpd for 127.0.0.1:34844 <== MODE READER
2009-05-02T15:24:46 nntpd for 127.0.0.1:34844 ==> 500 unrecognized command

This went on and on and on for five seconds, until tin crashed (which I believe is a bug – a NNTP client ought to either give up after getting 500 to CAPABILITIES, or ignore the 500 from MODE READER – I’m sure there are functional NNTP servers predating CAPABILITIES that do not implement MODE READER either). My server promptly crashed as well – my logic for handling socket errors was faulty (fixed).

Next victim was slrn. It was much better behaved:

2009-05-02T15:37:07 nntpd for 127.0.0.1:40000 starts
2009-05-02T15:37:07 nntpd for 127.0.0.1:40000 ==> 200 posting allowed
2009-05-02T15:37:07 nntpd for 127.0.0.1:40000 <== MODE READER
2009-05-02T15:37:07 nntpd for 127.0.0.1:40000 ==> 500 unrecognized command
2009-05-02T15:37:07 nntpd for 127.0.0.1:40000 <== XOVER
2009-05-02T15:37:07 nntpd for 127.0.0.1:40000 ==> 500 unrecognized command
2009-05-02T15:37:07 nntpd for 127.0.0.1:40000 <== XHDR Path
2009-05-02T15:37:07 nntpd for 127.0.0.1:40000 ==> 500 unrecognized command
2009-05-02T15:37:07 nntpd for 127.0.0.1:40000 <== NEWGROUPS 071115 214035 GMT
2009-05-02T15:37:07 nntpd for 127.0.0.1:40000 ==> 500 unrecognized command
2009-05-02T15:37:07 nntpd for 127.0.0.1:40000 <== GROUP gmane.comp.shells.dash
2009-05-02T15:37:07 nntpd for 127.0.0.1:40000 ==> 500 unrecognized command
2009-05-02T15:37:07 nntpd for 127.0.0.1:40000 <== GROUP gmane.comp.version-control.packaging
2009-05-02T15:37:07 nntpd for 127.0.0.1:40000 ==> 500 unrecognized command
2009-05-02T15:37:07 nntpd for 127.0.0.1:40000 <== GROUP gmane.linux.drivers.ath5k.user
2009-05-02T15:37:07 nntpd for 127.0.0.1:40000 ==> 500 unrecognized command
2009-05-02T15:37:07 nntpd for 127.0.0.1:40000 <== GROUP gmane.linux.kernel
2009-05-02T15:37:07 nntpd for 127.0.0.1:40000 ==> 500 unrecognized command
2009-05-02T15:37:07 nntpd for 127.0.0.1:40000 <== GROUP gmane.linux.kernel.announce
2009-05-02T15:37:07 nntpd for 127.0.0.1:40000 ==> 500 unrecognized command
2009-05-02T15:37:07 nntpd for 127.0.0.1:40000 <== GROUP gmane.comp.compilers.cyclone.general
2009-05-02T15:37:07 nntpd for 127.0.0.1:40000 ==> 500 unrecognized command
2009-05-02T15:37:07 nntpd for 127.0.0.1:40000 <== GROUP gmane.comp.version-control.git
2009-05-02T15:37:07 nntpd for 127.0.0.1:40000 ==> 500 unrecognized command
2009-05-02T15:37:07 nntpd for 127.0.0.1:40000 <== GROUP gmane.comp.compilers.pcc
2009-05-02T15:37:07 nntpd for 127.0.0.1:40000 ==> 500 unrecognized command
2009-05-02T15:37:21 nntpd for 127.0.0.1:40000 <== QUIT
2009-05-02T15:37:21 nntpd for 127.0.0.1:40000 ==> 205 bye
2009-05-02T15:37:21 nntpd for 127.0.0.1:40000 terminating

Unfortunately, slrn went and deleted my subscriptions to those groups. The bastard. :)

dpkg tip

If your dpkg runs seem to take a long time in the “reading database” step, try this:

Step One: Clear the available file dpkg --clear-avail

Step Two: Forget old unavailable packages dpkg --forget-old-unavail

Step Three: If you use grep-available or other tools that rely on a useful available file, update the available file using sync-available (in the dctrl-tools package).

The few times I’ve tried it (all situations where the “reading database” step seemed to take ages), it has always sped the process up dramatically. There probably are situations where it won’t make much difference, but I haven’t run into them.

Initramfs problems with the new kernel-package, and a solution

I’ve been using Manoj’s new kernel-package for some weeks now, and used it to compile two kernels (a reconfigured 2.6.29.1 and the new 2.6.29.2). Both times I’ve had trouble with initrd.

As the documentation says, kernel-package kernel packages no longer do their own initramfs generation. One must copy the example scripts at /usr/share/doc/kernel-package/examples/etc.kernel/postinst.d/initramfs and /usr/share/doc/kernel-package/examples/etc.kernel/postrm.d/initramfs to the appropriate subdirectories of /etc/kernel/. However, this is not enough.

My /etc/kernel-img.conf file had the usual posthook and prehook lines calling update-grub. Unfortunately, those hooks are called before the postinst.d hooks, and so update-grub never saw my initramfs images.

Fix? I removed those lines from /etc/kernel-img.conf and created a very simple postinst.d and postrm.d script:

#!/bin/sh

update-grub

I call the script zzz-grub-local, to ensure that it runs last.

I have inhibitions

There is a particular issue about which I am incapable of expressing my opinion. I do have an opinion about it, and I have developed it quite a bit in my own head over the years. However, if I try to voice it, something in me tells me not to.

There is another issue about which I am capable of expressing my opinion, but I choose not to. It is a highly controversial topic, and it is very likely that if I expressed my opinion about it, it would be interpreted as being another opinion, which I do not hold.

I do not know whether to call that Wittgenstein’s curse or not.

Some things to avoid when triaging other people’s bugs

DO NOT send your query for more information only to nnn@bugs.debian.org. That address (almost) never reaches the submitter. (The correct address is nnn-submitter@bugs.debian.org – or you can CC the submitter directly.)

DO NOT close a bug just because your query “can you still reproduce it” has not been promptly answered.

And, actually:

DO NOT close a bug if you do not have the maintainer’s permission to do so. You may, if you wish, state in the bug logs that you think the bug should be closed.

This ends today’s public service announcement. Thank you for your attention.