I had one for about 4 years in my mid twenties. Drove it from London to my parents in Chichester (about 70 miles) every other weekend pretty much, all year round, in any weather, with no roof, doors or windscreen. Just wore a lot of clothes in the winter! Took it to Le Mans too, which was fairly intense, but not uncomfortable.
I found the fibreglass seats very comfortable - possibly the only car not to give me back ache on a long journey - visibility is unrestricted, more power than you need, and so small you can overtake in places you wouldn't in most cars (think about those bikes leapfrogging their way past slow traffic one car at a time).
They are unbelievably involving. About half way through my ownership, after a particularly wet trip, I test drove an an S1 Exige and found it quite boring in comparison. At the time the Caterham had about 140bhp, but I went on to have the engine rebuilt to 200bhp or so which made it fairly quick.
I'd have another one if my wife would let me, but unfortunately that one ended up upside down so she has taken against them for some reason...
Despite that ending, I'd say the advantage over a bike is that it's still going to be safer on the whole. And you can take a bit of luggage with you.