A note on Planet Haskell policy

I quote from the Planet Haskell FAQ:

A common misunderstanding about Planet Haskell is that it republishes only Haskell content. That is not its mission. A Planet shows what is happening in the community, what people are thinking about or doing. Thus Planets tend to contain a fair bit of “off-topic” material. Think of it as a feature, not a bug.

As such, it is my policy, as a Planet Haskell editor, to encourage people to give us their general feed, unless there are specific reasons to do otherwise (see the FAQ for discussion). People who do that have no selective control over what posts are republished by Planet Haskell – what they post, we republish.

The reason behind this policy is given in the quote. To spell it out: A Planet is not a scholarly journal, nor is it a discussion forum. It is a way for people to read the blogs of like-minded people. Blogs are their authors’ podiums for pontificating on whatever they like; I as a Planet Haskell editor only care about whether they are Haskell people, not (in general) about the content of their specific posts.

If you have issues with that policy, talk to us at planet@community.haskell.org. Please do not harass individual posters with comments like “Why did you post this on Planet Haskell?”

New Netnews (Usenet) RFCs

The RFC editor has finally released

  • K. Murchison, Ed., C. Lindsey, D. Kohn: Netnews Article Format. RFC 5536, November 2009.
  • R. Allbery, Ed., C. Lindsey: Netnews Architecture and Protocols. RFC 5537, November 2009.

They obsolete the venerable RFC 1036 (Standard for Interchange of USENET Messages) from December 1987.

The new RFCs are the work of the IETF working group USEFOR, chartered November 1997 and concluded March 2009. I’ve heard it was not quite the longest lived IETF working group ever. (I personally missed the group by a couple of months, since I started following Netnews and NNTP standardization in April, due to Alue.)

Both RFCs are on the Standards Track, currenlty as Proposed Standards. In my opinion, they are a huge improvement over both RFC 1036 and son-of-1036 (which will probably be published as a Historic RFC real soon now).

Socialized vaccination – a narrative

Such was the scene I arrived at on Wednesday last week at the municipal health center at Kyllö, Jyväskylä, Finland. A queue extended a hundred meters beyond the door. It was not hard to guess what it was about, as it had been announced that the pandemic influenza A/H1N1 vaccine would be administered there to people in specific risk groups from 10 am to 3:30 pm.

I should explain here the Finnish health care setup. There are three parallel systems – a comprehensive public health care system maintained by the municipalities to standards set by the state, a network of private for-profit health care providers, and a national foundation dedicated to university student health care. Employers are required by law to provide a minimal level of health care to their employees, and most of them also provide, free of charge to the employees, access to near-comprehensive general-practice-level care; most employers buy this service from the for-profit providers. The public system suffers from resource starvation at the general-practice level but provides excellent care in the central hospitals that handle the difficult cases.

Anyway, the H1N1 vaccine is only available through the public system and through the foundation – free of charge if you qualify for the vaccine, and no amount of money buys it for you in this country if you don’t. Thus, I and many others, normally cared by the employee services, found ourselves queuing up at a public health care institute. And clearly, the public health care system was overwhelmed on that first day.

When I entered the queue, its tail was a traffic hazard. Fortunately, the queue moved faster than new people arrived thereafter, and the hazard ended. The queue moved surprisingly fast – it took me one hour to advance the 100 meters to the door. Even so, this was a failure in the system – there is no good reason to have people with underlying illnesses (and we all had them, to qualify for the vaccine) stand around in freezing cold weather for an hour!

Once, a nurse came and asked us if any of us was 63 years old or older. Apparently no-one was, since no-one was asked to leave (they will be vaccinated later). Later, another nurse asked everyone in the queue outside to show their KELA cards – KELA is the national health insurance system, and its card carries information about extra coverage one qualifies for due to underlying illnesses.

Eventually, I reached the door. Two guards stopped anyone who tried to enter directly, sidestepping the queue, and let in only those who had legitimate business other than vaccination. The main hall was full of people, and I quickly realized that the queue took a full circle inside the hall to reach the multiplexing point. It took me another hour to slowly advance my way though the hall. At the multiplexing point, I was asked to wait a bit, and then I was assigned a vaccination room and a waiting number.

Some twenty minutes later I was called in. The vaccination room I was assigned to was a nurse’s office. Two nurses were there, one who would administer the vaccine, and another at the computer, to keep a record. I gave them my KELA card, and shed my coat and my outer shirt, and bared my left shoulder. I was quickly given the pandemic vaccine; there was no question I was qualified for it, not with my obesity being obvious.

Then they asked what my diagnosis was. “Prmarily I’m here because of my obesity,” I said. “But I also have paroxysmal atrial fibrillation.”

“That’s not what your KELA card says,” accused the nurse at the computer.

“The diagnosis is so new,” I countered. “There has been no time to do the paperwork for KELA.” (And indeed, I later learned, it would come up in my next checkup in the spring.)

They stared at each other.

“I can show you my prescriptions,” I said, making no move for them.


I stared back.

“Do you want the seasonal vaccine or not?” asked the nurse with the injectors.

I lauged briefly. “I might as well.” It had, honestly, never crossed my mind that I might qualify.

She injected me on my right shoulder. “You should stay in out there for ten minutes.”

I picked up my clothes and found the cafeteria with coffee and pastry.

Then the real fun started. Next day, I woke with horrible upper back pain. The thermometer showed mild fever; but since i didn’t have any respiratory symptoms, I decided to go to work. In the evening, turning in bed was excruciatingly painful. It took days for the pain to subside.

I left the building three hours after I arrived at the end of the queue. What? You think that’s excessive? So do I and many others; queuing feels so Soviet Union! But honestly, while it did take time, it worked. I am vaccinated; are you?

30 Things About My Invisible Illness You May Not Know

Edited on 2009-09-17 to add to the 13th question.

1. The illness I live with is:
There are several, but let’s select the three that affect my life the most: my congenital cerebral palsy, my hearing problem (a sharp drop in my right ear’s ability around the frequencies commonly used in consonants), and my paroxysmal atrial fibrillation.

2. I was diagnosed with it in the year:
The CP was diagnosed some time very early in my life; not at birth, but not many years afterward. I’m sure my parents can give a better estimate in the comments. The hearing problem was diagnosed when I was 16 or so. The afib was diagnosed in the emergency room in January 2008.

3. But I had symptoms since:
The CP is a neurological injury that probably happened during my birth. I’ve had its symptoms ever since. The hearing problem was diagnosed soon after the symptoms were noticed. The afib was diagnosed less than one hour after first onset of symptoms.

4. The biggest adjustment I’ve had to make is:
I always try to position myself so that other people that I expect to converse with are not on my right, since – especially in situations with lots of background noise – I tend to have trouble understanding speech that I hear through my right ear. Sometimes this is a tradeoff, with some people on my right whatever I do.

5. Most people assume:
That my problems in moving around are due to my obesity. Granted, it doesn’t help, but a lot comes from the CP.

6. The hardest part about mornings are:
Nothing relevant to these medical conditions.

7. My favorite medical TV show is:
There isn’t one. They all make me panic.

8. A gadget I couldn’t live without is:
Nothing relevant to these medical conditions.

9. The hardest part about nights are:
Settling down on the bed, feeling relaxation coming and then having the heart freak itself out in an afib episode. Or going to bed a bit too late, falling asleep, and then waking up to a freaked-out heart which then keeps me awake until dawn. Mind, these don’t happen that often – the usual is about three weeks between normal episodes and a couple of months between the “awakens me” ones.

10. Each day I take __ pills & vitamins.
Two – bisoprolol 5 mg and flecainide 100 mg. The flecainide is a new medication for me and hopefully helps me with the previous question.

11. Regarding alternative treatments I:
I doubt them. (I was going to write that I’m skeptical, but that has special connotations in this context.)

12. If I had to choose between an invisible illness or visible I would choose:
To kill the person making me make that choice. Well, at least curse them.

13. Regarding working and career:
I can’t play music well enough due to the CP, though I was enthusiastic in my teens. It was hopeless for me to even try sports for the same reason. I was disqualified from serving in the peacetime military (which is normally compulsory for men in Finland) due to the CP; the hearing problem alone would have given me a reduced fitness rating. Fortunately, none of these affect my current profession.

14. People would be surprised to know:
That I have CP. Mine is mild enough that only experts can tell without me cluing them in. Also, I suppose people would be surprised to know that the afib is not life-threatening (since anything involving the heart is scary, right?).

15. The hardest thing to accept about my new reality has been:
That I do need the scary medicine I was recommended (namely, flecainide).

16. Something I never thought I could do with my illness that I did was:
Can’t think of anything.

17. The commercials about my illness:
There aren’t any.

18. Something I really miss doing since I was diagnosed is:

19. It was really hard to have to give up:

20. A new hobby I have taken up since my diagnosis is:

21. If I could have one day of feeling normal again I would:
“Normal is what everyone else is and you are not.” An unforgettable quote from a forgettable movie, and quite right. I’ve never been, and I never will be, normal. I have no desire to be normal.

22. My illness has taught me:
That not all heart problems are life-threatening.

23. Want to know a secret? One thing people say that gets under my skin is:
“CP”. At one time, it was a contentless hate word.

24. But I love it when people:
Nothing specific to these medical conditions.

25. My favorite motto, scripture, quote that gets me through tough times is:
Nothing specific to these medical conditions.

26. When someone is diagnosed I’d like to tell them:
It isn’t the end of the world.

27. Something that has surprised me about living with an illness is:
Can’t think of a thing.

28. The nicest thing someone did for me when I wasn’t feeling well was:
Reassure me that I wasn’t going to die. That was about two hours after I had been admitted to the emergency room due to my first afib episode.

29. I’m involved with Invisible Illness Week because:
I’m not. I’m doing this because I feel like it and because someone pointed me to another poster who had participated.

30. The fact that you read this list makes me feel:
Intrigued. Why did you?

Alue status report

In May, I posted about the discussion forum software I am writing, Alue. Since then –

What works:

  • The NNTP interface is essentially complete.
  • There is a rudimentary reading and posting HTTPS interface .
  • Users are able to self-register, manage their own accounts and reset lost passwords using an email challenge system.

What is broken:

  • NNTP control messages.
  • MIME message display in HTTPS (including character set conversions)
  • Web design. Although – all HTML is template-generated, so it’s more a problem with the test installation than with the actual software.

What is missing:

  • HTTPS-based administration
  • Moderation
  • Spam control
  • Email distribution of messages
  • Posting by email
  • Packaging (including proper installation and upgrade procedures)

And it would probably use a proper security review of the code.

If you are interested, go check out the test installation. The code and the test installation templates are available through Git. If you are really brave (and are a skilled system administrator), you might try creating your own installation – if you do, let me know.

Eric Flint on copyright and DRM

[Originally posted in June 2006; updated with new links several times, most recently in February 2010]

Eric Flint: A Matter of Principle, Jim Baen’s Universe 1 (1), 2006.
Eric Flint: Copyright: What Are the Proper Terms for the Debate?, Jim Baen’s Universe 1 (2), 2006.
Eric Flint: Copyright: How Long Should It Be?, Jim Baen’s Universe 1 (3), 2006.
Eric Flint: What is Fair Use, Jim Baen’s Universe 1 (4), 2006.
Eric Flint: Lies, and More Lies, Jim Baen’s Universe 1 (5), 2007.
Eric Flint: There Ain’t No Such Thing as a Free Lunch, Jim Baen’s Universe 1 (6), 2007.
Eric Flint: Books: The Opaque Market, Jim Baen’s Universe 2 (1), 2007.
Eric Flint: Spillage: or, The Way Fair Use Works in Favor of Authors and Publishers, Jim Baen’s Universe 2 (2), 2007.
Eric Flint: The Economics of Writing, Jim Baen’s Universe 2 (3), 2007.
Eric Flint: The Pig-in-a-Poke Factor, Jim Baen’s Universe 2 (4), 2007.
Eric Flint: Paper books are not going to be joining the dodo any time soon. If ever., Jim Baen’s Universe 2 (5), 2008
Eric Flint: A Matter of Symbiosis. Jim Baen’s Universe 2 (6), 2008
Eric Flint: The Nature of Transitions. Jim Baen’s Universe 3 (1), 2008
Eric Flint: Adventures with a Search Engine. Jim Baen’s Universe 3 (2), 2008
Eric Flint: The Problem is Legal Scarcity, not Illegal Greed. Jim Baen’s Universe 3 (3), 2008
Eric Flint: Foam and Froth and Mighty (Upside-down) Pyramids. Jim Baen’s Universe 3 (4), 2009
Eric Flint: The Internet is Not a Magic Wand. Jim Baen’s Universe 3 (5), 2009

The column series seems to have not been continued.

Eric Flint is a fairly successful sf author. These columns explore the evils of Digital Restrictions Management (DRM, also known as Don’t Read Me).

This is Alue

I have made a couple of references in my blog to the new software suite I am writing, which I am calling Alue. It is time to explain what it is all about.

Alue will be a discussion forum system providing a web-based forum interface, a NNTP (Netnews) interface and an email interface, all with equal status. What will be unusual compared to most of the competition is that all these interfaces will be coequal views to the same abstract discussion, instead of being primarily one of these things and providing the others as bolted-on gateways. (I am aware of at least one other such system, but it is proprietary and thus not useful to my needs. Besides, I get to learn all kinds of fun things while doing this.)

I have, over several years, come across many times the need for such systems and never found a good, free implementation. I am now building this software for the use of one new discussion site that is being formed (which is graciously willing to serve as my guinea pig), but I hope it will eventually be of use to many other places as well.

I now have the first increment ready for beta testing. Note that this is not even close to being what I described above; it is merely a start. It currently provides a fully functional NNTP interface to a rudimentary (unreliable and unscalable) discussion database.

The NNTP server implements most of RFC 3977 (the base NNTP spec – IHAVE, MODE-READER, NEWNEWS and HDR are missing), all of RFC 4642 (STARTTLS) and a part of RFC 4643 (AUTHINFO USER – the SASL part is missing). The article database is intended to support – with certain deliberate omissions – the upcoming Netnews standards (USEFOR and USEPRO), but currently omits most of the mandatory checks.

There is a test installation at verbosify.org (port 119), which allows anonymous reading but requires identification and authentication for posting. I am currently handing out accounts only by invitation.

Code can be browsed in a Gitweb; git clone requests should be directed to git://git.verbosify.org/git/alue.git/.

There are some tweaks to be done to the NNTP frontend, but after that I expect to be rewriting the message filing system to be at least reliable if not scalable. After that, it is time for a web interface.

Asynchronous transput and gnutls

To the extent possible under law,
Antti-Juhani Kaijanaho has waived all copyright and related or neighboring rights to
Asynchronous transput and gnutls. This work is published from Finland.

GnuTLS is a wonderful thing. It even has a thick manual – but nevertheless its documentation is severely lacking from the programmer’s point of view (and there doesn’t even seem to be independent howtos floating on the net). My hope is to remedy with this post, in small part, that problem.

I spent the weekend adding STARTTLS support to the NNTP (reading) server component of Alue. Since Alue is written in C++ and uses the Boost ASIO library as its primary concurrency framework, it seemed prudent to use ASIO’s SSL sublibrary. However, the result wasn’t stable and debugging it looked unappetizing. So, I wrote my own TLS layer on top of ASIO, based on gnutls.

Now, the gnutls API looks like it works only with synchronous transput: all TLS network operations are of the form “do this and return when done”; for example gnutls_handshake returns once the handshake is finished. So how does one adapt this to asynchronous transput? Fortunately, there are (badly documented) hooks for this purpose.

An application can tell gnutls to call application-supplied functions instead of the read(2) and write(2) system calls. Thus, when setting up a TLS session but before the handshake, I do the following:

                gnutls_transport_set_ptr(gs, this);
                gnutls_transport_set_push_function(gs, push_static);
                gnutls_transport_set_pull_function(gs, pull_static);
                gnutls_transport_set_lowat(gs, 0);

Here, gs is my private copy of the gnutls session structure, and the push_static and pull_static are static member functions in my sesssion wrapper class. The first line tells gnutls to give the current this pointer (a pointer to the current session wrapper) as the first argument to them. The last line tells gnutls not to try treating the this pointer as a Berkeley socket.

The pull_static static member function just passes control on to a non-static member, for convenience:

ssize_t session::pull_static(void * th, void *b, size_t n)
        return static_cast<session *>(th)->pull(b, n);

The basic idea of the pull function is to try to return immediately with data from a buffer, and if the buffer is empty, to fail with an error code signalling the absence of data with the possibility that data may become available later (the POSIX EAGAIN code):

class session
        std::vector<unsigned char> ins;
        size_t ins_low, ins_high;
ssize_t session::pull(void *b, size_t n_wanted)
        unsigned char *cs = static_cast<unsigned char *>(b);
        if (ins_high - ins_low > 0)
                errno = EAGAIN;
                return -1;
        size_t n = ins_high - ins_low < n_wanted 
                ?  ins_high - ins_low 
                :  n_wanted;
        for (size_t i = 0; i < n; i++)
                cs[i] = ins[ins_low+i];
        ins_low += n;
        return n;

Here, ins_low is an index to the ins vector specifying the first byte which has not already been passed on to gnutls, while ins_high is an index to the ins vector specifying the first byte that does not contain data read from the network. The assertions 0 <= ins_low, ins_low <= ins_high and ins_high <= ins.size() are obvious invariants in this buffering scheme.

The push case is simpler: all one needs to do is buffer the data that gnutls wants to send, for later transmission:

class session
        std::vector<unsigned char> outs;
        size_t outs_low;
ssize_t session::push(const void *b, size_t n)
        const unsigned char *cs = static_cast<const unsigned char *>(b);
        for (size_t i = 0; i < n; i++)
        return n;

The low water mark outs_low (indicating the first byte that has not yet been sent to the network) is not needed in the push function. It would be possible for the push callback to signal EAGAIN, but it is not necessary in this scheme (assuming that one does not need to establish hard buffer limits).

Once gnutls receives an EAGAIN condition from the pull callback, it suspends the current operation and returns to its caller with the gnutls condition GNUTLS_E_AGAIN. The caller must arrange for more data to become available to the pull callback (in this case by scheduling an asynchronous write of the data in the outs buffer scheme and scheduling an asynchronous read to the ins buffer scheme) and then call the operation again, allowing the operation to resume.

The code so far does not actually perform any network transput. For this, I have written two auxiliary methods:

class session
        bool read_active, write_active;
void session::post_write()
        if (write_active) return;
        if (outs_low > 0 && outs_low == outs.size())
                outs_low = 0;
        else if (outs_low > 4096)
                outs.erase(outs.begin(), outs.begin() + outs_low);
                outs_low = 0;
        if (outs_low < outs.size())
                                     this, _1, _2));
                write_active = true;

void session::post_read()
        if (read_active) return;
        if (ins_low > 0 && ins_low == ins.size())
                ins_low = 0;
                ins_high = 0;
        else if (ins_low > 4096)
                ins.erase(ins.begin(), ins.begin() + ins_low);
                ins_high -= ins_low;
                ins_low = 0;
        if (ins_high + 4096 >= ins.size()) ins.resize(ins_high + 4096);

                                           this, _1, _2));
        read_active = true;

Both helpers prune the buffers when necessary. (I should really remove those magic 4096s and make them a symbolic constant.)

The data members read_active and write_active ensure that at most one asynchronous read and at most one asynchronous write is pending at any given time. My first version did not have this safeguard (instead trying to rely on the ASIO stream reset method to cancel any outstanding asynchronous transput at need), and the code sent some TLS records twice – which is not good: sending the ServerHello twice is guaranteed to confuse the client.

Once ASIO completes an asynchronous transput request, it calls the corresponding handler:

void session::received_some(boost::system::error_code ec, size_t n)
        read_active = false;
        if (ec) { pending_error = ec; return; }
        ins_high += n;
void session::sent_some(boost::system::error_code ec, size_t n)
        write_active = false;
        if (ec) { pending_error = ec; return; }
        outs_low += n;

Their job is to update the bookkeeping and to trigger the resumption of suspended gnutls operations (which is done by post_pending_actions).

Now we have all the main pieces of the puzzle. The remaining pieces are obvious but rather messy, and I’d rather not repeat them here (not even in a cleaned-up form). But their essential idea goes as follows:

When called by the application code or when resumed by post_pending_actions, an asynchronous wrapper of a gnutls operation first examines the session state for a saved error code. If one is found, it is propagated to the application using the usual ASIO techniques, and the operation is cancelled. Otherwise, the wrapper calls the actual gnutls operation. When it returns, the wrapper examines the return value. If successful completion is indicated, the handler given by the application is posted in the ASIO io_service for later execution. If GNUTLS_E_AGAIN is indicated, post_read and post_write are called to schedule actual network transput, and the wrapper is suspended (by pushing it into a queue of pending actions). If any other kind of failure is indicated, it is propagated to the application using the usual ASIO techniques.

The post_pending_actions merely empties the queue of pending actions and schedules the actions that it found in the queue for resumption.

The code snippets above are not my actual working code. I have mainly removed from them some irrelevant details (mostly certain template parameters, debug logging and mutex handling). I don’t expect the snippets to compile. I expect I will be able to post my actual git repository to the web in a couple of days.

Please note that my (actual) code has received only rudimentary testing. I believe it is correct, but I won’t be surprised to find it contains bugs in the edge cases. I hope this is, still, of some use to somebody :)