Thursday, June 28, 2012

Not Building Erlang Apps

So I've spent the past few days playing around with various ways of building Erlang projects, and it's taken me from mild frustration to fuck-everything-about-this mode. Here is the synopsis of ways that you should not build an app, despite what you may have heard to the contrary.


Like I said last time, every single tutorial that I've found so far has run me up against an error when the time comes to actually generate a working system. I've had no help so far, and without that generation step, rebar is essentially a poor-man's make script fused with a poor-man's quickproject-for-Erlang. This is true both of the build in their repos, and of the one in their downloads.

I'm not saying "it doesn't work", because I've successfully used the rebar application called nitrogen, I'm saying it has yet to work for me, despite the fact that I've tried following five separate pieces of allegedly correct documentation for it. It may work for you, but I'm not inclined to bet on it.


If you take a look at what rebar is actually supposed to do, you'll find that a lot of it can be done from within a running Erlang process. systools and release_handler ostensibly help you put together a production build of your environment and deploy it. And the word "ostensibly" in that sentence should tell you how that went.

Here's the process you're supposed to follow:

  1. arrange your project in the OTP style (with src, ebin, priv and rel directories at minimum, an app file in src to describe your application, and your compiled beams all going into ebin)
  2. compile your project
  3. create a rel file
  4. use systools:make_script/2 to generate the script and boot files for local running (test those, if you like)
  5. create an appup file to tell Erlang what needs to change between some previous and this one
  6. use systools:make_relup/3 to generate a relup file (this oddly requires an unpacked copy of both your previous version and this version; they both need to be named <yourproject>.app, so be prepared to do some directory trickery)
  7. use systools:make_tar/1 to generate a tar.gz file of your entire project
  8. copy that tar file up to your server
  9. if this is your first release, just untar it and run erl -boot releases/<release-name>/start, otherwise use release_handler:unpack_release/1, release_handler:install_release/1 and release_handler:make_permanent/1 to perform a running upgrade

This process gave me some trouble around steps 3, 5 and 6, and finally errored outright at step 9. The problem I was having with the earlier pieces involved what I've come to think of as The Erlang Bureaucracy. That's when a piece of the process requires you, the human, to formally, manually and accurately type out a whole bunch of list-based information that the machine has access to. The rel file was a particularly annoying example. Here's what one looks like


It should also be named example-1.1.rel, in case you thought you had any choice on that front. Like I said, it's implied in the documentation that you ought to be writing this down, but in fact, the template is very straightforward

   {"<application name>_rel", "<application version>"},
   {erts, "<erts version>"},
   [<list of {appname, "version"} for each required application>]}

and every single piece of information there can be automatically generated by an Erlang node that's already running your system. If you wanted to define a shortcut for yourself, you'd do it like so:

make_rel_file(AppnameAtom) ->
    AppnameStr = atom_to_list(AppnameAtom),
    {_, _, V} = lists:keyfind(AppnameAtom, 1, 
    ActiveApps = lists:map(fun ({App, _Description, Ver}) -> {App, Ver} end, 
    Rel = {release, {AppnameStr ++ "_rel", V}, 
           {erts, erlang:system_info(version)}, 
    RelFilename = lists:append(["rel/", AppnameStr, "-", V, ".rel"])
    file:write_file(RelFilename, io_lib:format("~p.", [Rel])).

Granted, this means you need to settle for version names like "1.1" rather than "Porkchop Sandwiches", but that's still a damn sight easier than typing it out yourself every time.

appups are sort of understandable, until you realize that if the system had access to your git repository, it could easily tell what had changed since last time, leaving you to merely specify the tricky manual parts. If you follow, you may also be beginning to suspect that the Smalltalk guys had the right of it, but I digress.

After getting past the bureaucratic rings Erlang sets up, I generated a system using systools:make_tar/1, unpacked it and tested the fucker out. And it worked! It started up my application along with sasl just by doing erl -boot rel/example-1.1/releases/1.1/start! And I could upgrade it on the fly, and it was wonderful with fucking rainbows everywhere!!

And then I copied the tar file up to my remote server, took the same steps on the same release of Erlang/OTP+erts, and got a stack dump. Ho hum. Well, at least it worked on my machine, amirite?

What worked

What ended up working was just copying my application folder up (minus the src and rel subfolders, just to save space), and then running

erl -pa ebin -pa include -eval 'lists:map(fun (App) -> application:load(App), application:start(App) end, [required_app1, required_app2, example]).'

It's simple, it's stupid, and it won't make it easy for me to upgrade later, but it worked, and that's more than I could say for the documented approaches. So that's that. I fucking give up. I'm sure someone out there would be shaking their head if they saw this, but "automatic" release management is not worth the kinds of headaches that this has been causing me. It seems like doing anything other than the simplest possible thing with the Erlang system forces you to keep track of all kind of semi-transparent, undocumented internal state which is improperly set by default. State Is Hard, at the best of times, so I'm going to go ahead and avoid it until I get another masochistic urge to lose another six hours or so.

Sunday, June 24, 2012

Rebar Frustrations and LFE

I'm trying to get my head around Rebar at the moment, and failing pretty badly. It's honestly unclear whether I'm being an idiot here, but I've gone through all of four different tutorials on the subject, each purports to be a simple step-by step guide that'll get you up and running quickly, and each one hands you an error at around step four if you follow it as written.

The only piece of documentation I've been able to follow in its entirety without a crash dump is the basho Getting Started page. Except that it only gets me as far as starting and compiling a project, showcasing none of Rebar's dependency building, distribution creation or release handling. Which makes it marginally less useful than Make.

Following the upgrade_project option works fine on the dummy project provided as part of the rebar repo, but that project is shaped differently, and uses different config options than the ones rebar generates. Naturally, there is no documentation on what steps I have to take to get from what's generated to what works.

Ugh. Sorry, I had to vent for a bit there. Fuck this, I'm going back to Makefiles for the time being. I'll try rebar out again when my blood pressure lowers a bit.

Was that all?

Huh? Oh. No, I guess. I did also find this, which looks pretty badass. It's not exactly a Common Lisp or a Scheme. It's a purely functional, Lisp-2 with unhygenic macros that runs on top of the Erlang VM. It's about as officially supported as you can get, being made by Robert Virding, and it gets rid of quite a few things I don't like about Erlang.

Now, it's not a full Common Lisp, so don't expect quicklisp (sigh) or loop (though in theory, there doesn't seem to be a reason that you couldn't implement that one, if you wanted it badly enough), or CLOS, but it does hit the big ones. Namely full prefix notation, homoiconicity, and macros. And it retains the things about Erlang that I do find interesting, namely massive concurrency, the pure-functional approach, and standardized cross-process/cross-machine communication.

I'm going to keep playing with it, but I like it so far.

Friday, June 22, 2012

Authentication Part Four - Logging into Websites with RSA Keys









-export([start/0, stop/0]).
-export([init/1, handle_call/3, handle_cast/2, handle_info/2,
         terminate/2, code_change/3]).

-export([gen_secret/2, verify/3, new_key/2]).
-export([create/0, clear/0, recreate/0, find/1, now_to_seconds/1]).

-record(pubkey, {user_id, pubkey}).
-record(secret, {timestamp, user_id, ip, plaintext}).

%%% API
new_key(UserId, Pubkey) -> gen_server:call(?MODULE, {new_key, UserId, Pubkey}).
gen_secret(UserId, IP) -> gen_server:call(?MODULE, {gen_secret, UserId, IP}).
verify(UserId, IP, Sig) -> gen_server:call(?MODULE, {verify, UserId, IP, Sig}).

handle_call({gen_secret, UserId, IP}, _From, State) -> 
    Pubkey = find({key, UserId}),
    P = binary_to_hex(crypto:sha(crypto:rand_bytes(32))),
    Ciphertext = binary_to_list(m2crypto:encrypt(Pubkey, P)),
    Secret = #secret{timestamp=now(), user_id=UserId, ip=IP, plaintext=P},
    db:transaction(fun() -> mnesia:write(Secret) end),
    {reply, Ciphertext, State};
handle_call({verify, UserId, IP, Sig}, _From, State) ->
    Pubkey = find({key, UserId}),
    Secrets = find({secrets, UserId, IP}),
    Res = lists:any(
            fun({T, S}) -> verify_key({T, S}, Pubkey, Sig) end, 
    {reply, Res, State};
handle_call({new_key, UserId, Pubkey}, _From, State) -> 
    Res = case exists_p(UserId) of
              false -> Fname = make_tempname("/tmp"),
                       file:write_file(Fname, Pubkey),
                       K = m2crypto:split_key(Fname),
                       Rec = #pubkey{user_id=UserId, pubkey=K},
                       ok = db:transaction(fun() -> mnesia:write(Rec) end),
              true -> already_exists
    {reply, Res, State}.

%%% rsa_auth-specific utility
verify_key({T, S}, Pubkey, Sig) ->
    case old_secret_p(T) of
        true -> revoke_secret(T),
        _ -> case m2crypto:verify(Pubkey, S, Sig) of
                 true -> revoke_secret(T),
                 _ -> false

revoke_secret(T) ->
    db:transaction(fun() -> mnesia:delete({secret, T}) end).

old_secret_p(T) -> 
    %% it's old if the timestamp is older than 5 minutes
    300 < (now_to_seconds(now()) - now_to_seconds(T)).

exists_p(UserId) -> 
        find({key, UserId})
        error:_ -> false

%%% DB related
find({key, UserId}) -> 
    [Rec] = db:do(qlc:q([X#pubkey.pubkey || X <- mnesia:table(pubkey), X#pubkey.user_id =:= UserId])),
find({secrets, UserId, IP}) -> 
    db:do(qlc:q([{X#secret.timestamp, X#secret.plaintext} || 
                    X <- mnesia:table(secret), 
                    X#secret.user_id =:= UserId,
                    X#secret.ip =:= IP])).

create() ->
    mnesia:create_table(pubkey, [{type, ordered_set}, {disc_copies, [node()]}, {attributes, record_info(fields, pubkey)}]),
    mnesia:create_table(secret, [{type, ordered_set}, {disc_copies, [node()]}, {attributes, record_info(fields, secret)}]).

clear() ->

recreate() ->

%%% general utility
now_to_seconds(Now) ->

make_tempname() ->
    {A, B, C} = now(),
    [D, E, F] = lists:map(fun integer_to_list/1, [A, B, C]),
    lists:append(["tmp.", D, ".", E, ".", F]).
make_tempname(TargetDir) ->
    filename:absname_join(TargetDir, make_tempname()).

binary_to_hex(Bin) ->
    lists:flatten([io_lib:format("~2.16.0B", [X]) ||
                      X <- binary_to_list(Bin)]).

%%%%%%%%%%%%%%%%%%%% generic actions
start() -> gen_server:start_link({local, ?MODULE}, ?MODULE, [], []).
stop() -> gen_server:call(?MODULE, stop).

%%%%%%%%%%%%%%%%%%%% gen_server handlers
init([]) -> {ok, []}.
handle_cast(_Msg, State) -> {noreply, State}.
handle_info(_Info, State) -> {noreply, State}.
terminate(_Reason, _State) -> ok.
code_change(_OldVsn, State, _Extra) -> {ok, State}.

Actually, that's way too intimidating, given what this thing does. Lets break that shit down, and strip the gen_server/mnesia-related boilerplate. Chunklet the first is the meatiest:

%%% API
new_key(UserId, Pubkey) -> gen_server:call(?MODULE, {new_key, UserId, Pubkey}).
gen_secret(UserId, IP) -> gen_server:call(?MODULE, {gen_secret, UserId, IP}).
verify(UserId, IP, Sig) -> gen_server:call(?MODULE, {verify, UserId, IP, Sig}).

handle_call({gen_secret, UserId, IP}, _From, State) -> 
    Pubkey = find({key, UserId}),
    P = binary_to_hex(crypto:sha(crypto:rand_bytes(32))),
    Ciphertext = binary_to_list(m2crypto:encrypt(Pubkey, P)),
    Secret = #secret{timestamp=now(), user_id=UserId, ip=IP, plaintext=P},
    db:transaction(fun() -> mnesia:write(Secret) end),
    {reply, Ciphertext, State};
handle_call({verify, UserId, IP, Sig}, _From, State) ->
    Pubkey = find({key, UserId}),
    Secrets = find({secrets, UserId, IP}),
    Res = lists:any(
            fun({T, S}) -> verify_key({T, S}, Pubkey, Sig) end, 
    {reply, Res, State};
handle_call({new_key, UserId, Pubkey}, _From, State) -> 
    Res = case exists_p(UserId) of
              false -> Fname = make_tempname("/tmp"),
                       file:write_file(Fname, Pubkey),
                       K = m2crypto:split_key(Fname),
                       Rec = #pubkey{user_id=UserId, pubkey=K},
                       ok = db:transaction(fun() -> mnesia:write(Rec) end),
              true -> already_exists
    {reply, Res, State}.

That's essentially the entire external API for this style of authentication[1].

The exported functions are self-explanatory, so lets focus in on the handle_call/3 clauses. I mentioned last week that Erlang's own crypto functions don't provide a way to generate keys, and were having trouble importing any RSA 4096 keypairs I tried to work with, pretty much regardless of source. So I decided to call out to python for the actual encryption (more on that later). gen_secret needs to be accompanied by a UserId[2] and an IP[3]. The output is a random string, encrypted with the key of the given user, and associated with the given IP (if we wanted bi-directional authentication, we'd also have the server sign it and send the signature along).

verifying a signature requires the same two pieces of information, as well as the Signature. We select the set of secrets on file for the given user coming from the given IP, select the appropriate key, and then try to verify against each available secret. Verification happens in python too. In fact, lets take a quick look at that Erlang-side verification steps before we move on to handling the new_key message.

%%% rsa_auth-specific utility
verify_key({T, S}, Pubkey, Sig) ->
    case old_secret_p(T) of
        true -> revoke_secret(T),
        _ -> case m2crypto:verify(Pubkey, S, Sig) of
                 true -> revoke_secret(T),
                 _ -> false

revoke_secret(T) ->
    db:transaction(fun() -> mnesia:delete({secret, T}) end).

old_secret_p(T) -> 
    %% it's old if the timestamp is older than 5 minutes
    300 < (now_to_seconds(now()) - now_to_seconds(T)).

That seems reasonably self-explanatory too[4]. We check whether a given secret is too old, revoking it without granting access if it is, then calling out to python for the actual verification step (coming soon, I promise). If it succeeds, we revoke it and grant access. Note that by the time we've gotten to this point, the keys have already been verified for a matching IP. Right, back to the last clause in handle_call/3

handle_call({new_key, UserId, Pubkey}, _From, State) -> 
    Res = case exists_p(UserId) of
              false -> Fname = make_tempname("/tmp"),
                       file:write_file(Fname, Pubkey),
                       K = m2crypto:split_key(Fname),
                       Rec = #pubkey{user_id=UserId, pubkey=K},
                       ok = db:transaction(fun() -> mnesia:write(Rec) end),
              true -> already_exists
    {reply, Res, State}.

The process for storing a new key might take some explaining. We're expecting the contents of a PEM key file, rather than a file name because doing otherwise would force us to put this module on the same machine as the caller[5]. However, that raises a bit of a problem; M2Crypto can't import a pubkey from a string. It can do so for a keypair, but not if you don't have the private key on hand, which we won't. Ever. So what we need to do is create a temporary file somewhere, write the key out to it, then point M2Crypto at that to get the components back in a more digestible format. After that, it's just a matter of storing the key and cleaning up.

Ok, it's Python time


-export([start/0, stop/0]).
-export([init/1, handle_call/3, handle_cast/2, handle_info/2,
         terminate/2, code_change/3]).

-export([encrypt/2, verify/3, split_key/1]).

encrypt({E, N}, Message) -> gen_server:call(?MODULE, {encrypt, E, N, Message}).
verify({E, N}, Message, Signature) -> gen_server:call(?MODULE, {verify, E, N, Message, Signature}).
split_key(Filename) -> gen_server:call(?MODULE, {split_key, Filename}).

handle_call({'EXIT', _Port, Reason}, _From, _State) ->
    exit({port_terminated, Reason});
handle_call(Message, _From, Port) ->
    port_command(Port, term_to_binary(Message)),
        {State, {data, Data}} -> 
            {reply, binary_to_term(Data), State}
    after 3000 -> 

%%%%%%%%%%%%%%%%%%%% generic actions
start() -> gen_server:start_link({local, ?MODULE}, ?MODULE, [], []).
stop() -> gen_server:call(?MODULE, stop).

%%%%%%%%%%%%%%%%%%%% gen_server handlers
init([]) -> {ok, open_port({spawn, "python -u"}, [{packet, 4}, binary, use_stdio])}.
handle_cast(_Msg, State) -> {noreply, State}.
handle_info(_Info, State) -> {noreply, State}.
terminate(_Reason, State) -> State ! {self(), close}, ok.
code_change(_OldVsn, State, _Extra) -> {ok, State}.
from erlport import Port, Protocol, String
import M2Crypto
class M2cryptoProtocol(Protocol):
    def handle_split_key(self, filename):
        pubkey = M2Crypto.RSA.load_pub_key(String(filename))
        return (pubkey.e, pubkey.n)
    def handle_encrypt(self, e, n, message):
        pubkey = M2Crypto.RSA.new_pub_key((str(e), str(n)))
        ciphertext = pubkey.public_encrypt(String(message), M2Crypto.RSA.pkcs1_oaep_padding)
        return ciphertext.encode('base64')
    def handle_verify(self, e, n, message, sig):
        pubkey = M2Crypto.RSA.new_pub_key((str(e), str(n)))
        if pubkey.verify_rsassa_pss(String(message), String(sig).decode('base64')):
            return True
            return False

if __name__ == "__main__":
    M2cryptoProtocol().run(Port(packet=4, use_stdio=True))

Wordy crap on the Erlang side aside, this is reasonably simple. split_key breaks a Public Key PEM into its N and exponent. The reason I bother doing that is that, as I mentioned, while M2Crypto can't import a pubkey from a PEM, it can stitch one back together from its components. In other words, we're only storing keys as {E, N} tuples for the sake of letting M2Crypto do the work without intermediate files. encrypt and verify should be entirely self-explanatory.

And, that's that. Well, ok, on the backend anyway. Which means we've got two more pieces to go through. Here's what the user-facing Nitrogen module looks like

-module (rsa_auth).

main() -> #template { file="./site/templates/bare.html" }.

title() -> "Manual RSA Auth".

body() -> 
     #label {text="Username: "},
     #textbox { id=username, next=sendButton },
     #button { id=sendButton, text="Request Secret", postback=send_user },

     #panel { id=auth_token }

event(send_user) ->
    Token = rpc:call('trivial_user@', rsa_auth, gen_secret, [wf:q(username), wf:peer_ip()]),
              #panel { body=[ #span{text=Token} ]},
              #textarea { id=auth_response },
              #button { id=send_signed, text="Send Signed", postback=send_signed }
event(send_signed) ->
    Args = [wf:q(username), wf:peer_ip(), 
            re:replace(wf:q(auth_response), "\\\\n", "\n", [global, {return, list}])],
    Res = rpc:call('trivial_user@', rsa_auth, verify, Args),
    case Res of
        true -> wf:update(auth_token, [ #span { text="Yay! You're in!"} ]);
        _ -> wf:update(auth_token, [ #span {text="Halt, criminal scum!" } ])
event(_) -> ok.

That should be puzzle-out-able based on what we've been talking about too. Note that this expects to find a running instance of trivial_user at 'trivial_user@'. The only other thing I'll note is the bit that goes

re:replace(wf:q(auth_response), "\\\\n", "\n", [global, {return, list}])

That's necessary because of the way the string "\n" reacts to being dumped into a textarea. if you don't do that, shit will go oddly wrong and you won't be able to figure it out until it's late enough that I'm literally being kept awake by caffeine and strangely hypnotic music.

And that's bad.

The last remaining piece of this little system is the signing component, and here it is.


import M2Crypto, hashlib

def gimme_pw(*args):
    return "your passphrase goes here if you trust your computer"
    ### Ideally, you'd daemonize (not as scary as it looks) this script, have it prompt for a password and cache it

def sign(message, Privkey=M2Crypto.RSA.load_key("/path/to/your/rsa.pem", gimme_pw)):
    plaintext = Privkey.private_decrypt(message.decode('base64'), M2Crypto.RSA.pkcs1_oaep_padding)
    sig = Privkey.sign_rsassa_pss(plaintext)
    return sig.encode('base64')


That's the code down. The interaction, once you've registered and if you're going to be doing this manually, is

  1. Input your name and request a secret
  2. Copy the block of text the server sends you, and run the above signing script on it
  3. Copy the result into the newly formed textarea and click "Send Signed"

Assuming it was done correctly, you should then be logged in. The automatic version is going to have to wait for some sleep.

How Is This Better Than Passwords?

I don't fucking know, something. Oh, wait, yeah it is. In three specific ways.

  • Bi-directional authentication; If we implement that note I mentioned earlier, it lets you authenticate to your server and authenticate the server to you, without an intermediary[6]
  • No critical information is exchanged; even if someone is watching your entire transaction, they never get enough information to impersonate you, whether you're dealing with SSL or not.
  • No critical information is present on the server; even if your service provider is an utter dumbass that keeps their user database in plaintext with a little note taped to it reading "Plz dont steals", you don't care. Unlike a password system, where your password is effectively as secure as the weakest service you use it on, your RSA key is as secure as your personal machine. Granted, that may still not be very secure, but it's a step up.

I'm also convinced that once this is properly automated, it will be easier to deal with than password authentication from the user perspective, but I haven't built it yet, so I won't count that. I'm basing this conviction on the fact that I've stopped using SSH without RSA keys. I encourage you to try it if you haven't already.

Ok, that's it. Automated version coming soon, and good night.


1 - [back] - Ok, that's not true; we're missing two pieces, both critical in practice but borderline irrelevant for the theory.

The first one is bi-directional authentication. That would be pretty simple to implement from our perspective; all we'd need to do is sign the secret as it's being sent out. Doing so would let our user verify that they're talking to the server they expect rather than an eavesdropper or phisher. This overlaps slightly with SSL, but doesn't prevent a site from using both, and is so straightforward if you're already using this model that you may as well.

The second one is a way to revoke keys. That's more or less an open problem. For the purposes of this project, anyway. We could do something like hand our users a revocation phrase, or we could ask them to generate a second keypair which would be used to send a revocation message, or we could handle this through a second channel (which we should probably implement in any case, if we're serious about security). That second method sounds more secure, but really just recurses on the problem; what happens if your revocation key gets compromised? And how do you expect a user to store them?

Assigning a pass-phrase might seem like it's defeating the purpose, but remember that this one only comes out when you need to change keys (rather than at every login), and that lets us get a bit fancier with the sort of infrastructure we want to provide for it. For instance, I could imagine a provider mailing out actual plastic cards that people could stash in their wallets.

The third option is a lot more interesting, but I intend to write a piece on that by itself, so I won't waste much more time on it today. Sufficed to say that redundancy and isolation are key to build reliable systems, as Erlang has clearly demonstrated. And if you want a reliable channel for authentication, you really need to make it multiple independent channels. Slightly more annoying for your users, but exponentially more annoying for anyone trying to impersonate them.

Anyway, that's all beyond the scope of this piece, so I'm going to tactfully ignore it for the rest of the night.

2 - [back] - So that we know whose key to encrypt the secret with.

3 - [back] - Just as a security precaution against some types of sneakiness.

4 - [back] - Except that Erlang doesn't like the idea of durations for some reason, so I had to bring myself to write a comment.

5 - [back] - Which we wouldn't necessarily want to do, even if it didn't go against Erlang's grain.

6 - [back] - I'll save the rant about why having centralized signing authorities is stupid for when my eyelids aren't trying to sabotage me.

Tuesday, June 19, 2012

On Commanding Lines

Still working away on the authentication system. I'm basically at the point where I can use RSA keys to sign in to my demo webapp. Manually. As long as the keys are in PEM format. And crypto:verify is in a good mood.

This isn't about that though.

I've been slowly moving towards more and more command-line oriented interfaces. It's not a recent trend, in fact it started pretty much when I first discovered Emacs. Ever since doing away with my desktop environment a little while ago, it's been more of a necessity than idle speculation. The good news is that there's almost nothing I wanted to do in X windows that I can't do via command line.

Command Line MVPs

Let me draw your attention to some command line programs that I honestly wouldn't want to go without anymore. Not counting obvious necessities like ssh/rsync/find/grep/tail.

I've already written a bit about wicd-curses, the very good, simple command line network manager. After you set up a wireless device with Shift+p, and set up your connection keys, it'll make sure you're as plugged in as you can possibly be with no need for a network widget. You don't even need to run it unless you're connecting to a new network; the daemon starts up with the rest of your machine.

htop isn't anything new, if you've been paying attention. It's an improvement over the regular top in that it gives you more information and prettier colors. That's reason enough for me to use it.

acpi does quite a few things relating to cooling, power, and battery. Really, I just use it as the replacement for the gnome/xfce battery widget.

screen is something I've been using forever. My first time firing it up was to deploy a Hunchentoot application. Since then, I've used it as a way of managing multiple terminals, and kicked its tires as a full-on window manager.

mplayer is another piece that I've been using for a long time. Even while bumping around GNOME, I preferred this to VLC (YMMV). It's worth a read through the documentation if you're having a slow day; the program does various crazy things in addition to music/video playback, including bitmap frame outputs, format conversion and some timeline-based edits.

pacpl is an audio chopping tool. As of the latest version in the Debian repos, it can directly extract music from videos. As you can see by the website there, it can convert to and from pretty much any audio format you care to name, though I mostly use it to convert things to oggs.

imagemagick is a command-line image chopping program with so many options that you'd really better just read the docs. It's actually composed of a bunch of different utilities, of which I mostly use convert, mogrify and identify.

get_flash_videos is about the only way I get to see most videos, given a) how crappy flash support is when you're even half-way dedicated to the idea of Free software and b) how few sites other than YouTube provide an HTML5 based video player.

transmission-cli is the command line interface to my favorite torrent client. Granted, I don't torrent much since I got out of the habit of downloading the massive install CDs, but still.

gtypist is a curses-based typing tutor that has effectively replaced klavaro for me. It's mildly more entertaining to run typing drills on surrealist, minimal poetry than it is to type out old newspaper articles. The only thing about it that rustles my jimmies is that it enforces hitting space twice after a period. Which is a thing I guess? Honestly it sounds like an anachronistic behavior that used to make sense back when actual humans used actual typewriters. Luckily, the lessons are contained in a set of conf files, so I'll be able to do something about this.

EDIT: Aaaaand bam. Enjoy. Wed, 20 Jun, 2012

canto is a command-line based RSS feed reader. I complained about liferea earlier for its complexity, and having taken a look at a number of feed readers (both GUI and CLI), that doesn't seem to be an uncommon feature. canto, by contrast is ridiculously simple; set up your conf file, and it'll track those feeds, pulling when you tell it to (every 5 minutes by default). The example config up at the project site is pretty extensive, but I've gotten on fine with a much more minimal setup:

from canto.extra import *
import os

link_handler("lynx \"%u\"", text=True)
image_handler("feh \"%u\"", fetch=True)

keys['y'] = yank ## requires xclip

filters=[show_unread, None]


The one quirk that I have to highlight is that by default, its update doesn't fetch, it just updates from the local pool. In order to fetch, you actually need to run canto-fetch somehow. You can throw it in your crontab, but given how I use an RSS reader, it made more sense for me to just bind that to a StumpWM key.

feh is an extremely lightweight command-line imageviewer with options to browse folders, delete files, do slideshows and other assorted goodness. I didn't find this looking for an imageviewer, I found it looking for a way to get a background picture directly in Stump. It turns out that this does it:

(defun set-background (bg-image)
  (run-shell-command (format nil "feh --bg-scale ~a" bg-image)))

lynx is something I don't use on a regular basis anymore, but it is quite useful when I need to check a discussion or two without booting up X. It happens every once in a while.

Command Line Gaps

There aren't as many as you'd think. In fact, for my purposes, there is exactly one, and it's sort of minor; the lack of good animated gif viewer. There is a concerted effort at putting one together, but it didn't exactly blow me away. mplayer does a half-decent job, but chops when looping and doesn't loop by default (which is sort of helpful when describing haters). feh is awesome for stills, but doesn't display gifs in an animated fashion, and neither does Emacs. At the moment, my workaround is to just use chromium and call it a day.

Shell UI

Ok, so maybe I lied a little in the previous section. The thing I really don't like about some command line programs is their sometimes inscrutable option settings and lack of sensible defaults.

That second one bugged me enough that I whipped up a pair of Ruby scripts to help me out with archiving a little while ago. Yesterday, I ported them to Python; what they do, basically, is provide a sane set of default options for creating and decompressing various archives. Instead of tar -xyzomgwtfbbq foo.tgz, I can just call unpack foo.tgz. pack -t tar.gz foo/ similarly replaces tar -cwhyareyoueventryingtoreadthis foo.tar.gz foo/. I guess I could have done what most of my friends do (memorize the one or two most common combinations and call it a day), but having the machine adapt to humanware seems like the better idea to me.

That's also what caused me to sit down earlier and whip up a first draft at my first curses-based program. I was trying to pull out sections of some movies into animated gifs, and using mplayer/feh/convert manually proved to be laborious and repetitive. So, I did this.

I call it with a movie file, and the filename I want the result saved to. The program uses a curses interface to

  1. lets me pick a part of the movie to pull, using mplayer options -ss and -endpos
  2. has mplayer output the chosen section as a series of JPGs in a temp folder
  3. opens the folder with feh, giving me the opportunity to delete some frames as desired
  4. once I quit out of feh, stitches the remaining frames together into an animated gif

Honestly, I'm not sure how often I'll want to do that again, but hey. I've got the process formalized now, so it should be Pie next time. And, now I know how to curse in Python.

Thursday, June 14, 2012

Authentication Part Three - RSA Basics

I've been researching and prototyping for the past day or so, and all it's done is given me the desire to go out and murder some cryptographers.

Ok, ok, to be fair I did also get an excellent idea of why this isn't really widely used as an authentication strategy yet. The "UI" is ... intimidating. And that's coming from someone who isn't the least bit intimidated by learning a programming language or three in spare moments between actual programming projects. Something tells me the average user doesn't care enough about security to go through that kind of hoop.

It turns out that there are several distinct standards for public/private key storage, none of the popular Linux RSA-using applications use the same one by default, and it's possible but annoying to convert between them. Further, even though they're all RSA keys, they don't all let you do the same thing. OpenSSH uses PEM for private keys, a custom format for public keys, and a slightly different custom format for the known_hosts file; it doesn't provide facilities for anything other than SSH auth and key generation. GnuPG uses ascii-armored format[1] for exported private and public keys, though it would really prefer never to show you your private key; it lets you sign and encrypt, but is a bit awkward to import/export in other formats. OpenSSL technically works with PEM and several binary formats, but its default is the X.509 certificate standard; OpenSSL lets you verify, encrypt, decrypt and convert between key formats, but is a bit snippy about the format in which it outputs/verifies signatures.

The various language facilities available aren't exactly complete either. Erlang's crypto and public_key modules claim they can make and verify signatures, handle encryption/decryption with RSA, DSA and AES keys, as well as perform a number of cryptographic hashes. But they can't generate keys, and I've yet to get a working import of a 4096 bit RSA keypair, whether it's generated by GnuPG, OpenSSL or SSH. Python has RSA, native crypto, and OpenSSL-wrapper modules available. They sort of do enough things properly, if imperatively, enough that I could see putting together a half-way sane system with them.

I'm not even getting into signature formats, which are ... interesting. In a no-fun-at-all kind of way. To the point that I couldn't reliably verify an OpenSSL signature with anything other than, ostensibly, OpenSSL[2].

Here's a list of things I've tried putting together that didn't work:

  • Using GPG to generate keys, reading them with Erlang and verifying incoming GPG signatures.
  • Using SSH to generate keys, reading them with Erlang and verifying incoming GPG/OpenSSL-generated signatures.
  • Using SSH to generate keys, converting them to PEM format and calling OpenSSL to sign and verify.
  • Using OpenSSL to generate keys, then using Erlang to sign and verify[3]
  • Using OpenSSL to generate keys, and calling OpenSSL to sign and verify[4].
  • Using OpenSSL to generate keys, using OpenSSL to sign messages and M2Crypto to verify[5].

The options that did work:

  • Using OpenSSL to generate keys, then using M2Crypto to sign and verify[6].
  • Using OpenSSH to generate keys, then using M2Crypto to sign and verify
  • Using M2Crypto to generate keys, sign and verify messages[7]
  • Using GPG to generate keys, sign and verify messages[8].

Just as an example, here's how to use M2Crypto to make a round-trip with OpenSSL-generated PEM keys[9].

>>> import M2Crypto
>>> PrivKey = M2Crypto.RSA.load_key("rsa.pem")
Enter passphrase:
>>> Message = "Daring Do and the Quest for the Sapphire Stone"
>>> Signature = PrivKey.sign_rsassa_pss(Message)
>>> Sent = [Message, Signature.encode('base64')]

And at the other end

>>> [Msg, Sig] = Sent
>>> RawSig = Sig.decode('base64')
>>> PubKey = M2Crypto.RSA.load_pub_key("")
>>> PubKey.verify_rsassa_pss(Msg, RawSig)
1 ### 0 for "nope", 1 for "yup", and it might error under certain circumstances

In a real-world situation, we'd actually hash the message using one of the SHA-2 algorithms before sending, but that's the theory in its entirety.

Using gpg is more straightforward, though it does seem to insist on handling public key storage/management for you, so I'm not entirely sure how far this scales in terms of number of users supported. I covered generating gpg keys in a previous post, but here's the signing/verification round trip:

$ echo "Daring Do and the Griffon's Goblet" | gpg --output message.gpg --armor --sign

That generates a file that looks something like

Version: GnuPG v1.4.12 (GNU/Linux)


If you want it sent to standard out for whatever reason, just omit --output message.gpg. Once the server receives the message, it just calls

$ gpg --decrypt message.gpg
Daring Do and the Griffon's Goblet
gpg: Signature made [timestamp] using RSA key ID [GPG ID of the RSA key]
gpg: Good signature from "inaimthi <>"

Note that these are actually slightly different operations. gpg is doing a simultaneous encryption+signing, whereas the M2Crypto code is merely signing a plaintext message.

That's that. I did gloss over the part where we generate messages to send to the client, but other than that, this lays the preliminary groundwork for an RSA-based authentication system with actual web users.


1 - [back] - ascii-armored is distinct from, but similar to PEM, to the point where you can convert one to the other just by changing the start/end tag, and stripping the info entries, blank lines and checksum.

2 - [back] - In practice, I couldn't verify them with OpenSSL either, but that's possibly me being dense.

3 - [back] - Which I'm kind of happy about, since this would involve getting users to install and use Erlang.

4 - [back] - Again, that seems odd, but I'm not sure why yet.

5 - [back] - Which surprised the shit out of me, since M2Crypto is actually just a Python wrapper for OpenSSL.

6 - [back] - Odd that this worked while the above didn't, but I'll take it.

7 - [back] - That's probably the best case scenario anyway, since it means I can make a multi-platform client fairly easily.

8 - [back] - I have a soft spot for GNU, so I'd like to support this if at all possible. The PGP format has pretty thorough metadata attached, so delegating to GnuPG for messages using it will be reasonably straightforward.

9 - [back] - If you want M2Crypto to generate keys too, you'd just need to do

>>> Cert = M2Crypto.RSA.gen_key(4096, 65537)
>>> Cert.save_key("rsa.pem", "passphrase goes here")
>>> Cert.save_pub_key("")

Tuesday, June 12, 2012

Fresh Install

This weekend finally saw enough random free time that I manged a clean install of my laptop, and I think I've gotten it into more-or-less working order.

That's the Lenovo x220 I wrote about a while ago, though it has oddly gone up in price by more than an order of magnitude. I guess Core i3s are in very, very short supply?

Anyhow, this is the first time that I've configured my main machine without a desktop environment. I usually run either XFCE or GNOME under my window manager on my primary, and keep crazy things like Screen-as-WM, and odd bash replacements to my play boxes. Much as I hate to admit it, x-window-system is still a requirement at an office where you need to co-exist with MS users. Mainly for PDF viewing and documents/spreadsheets, but it also helps to be able to do some image editing if the situation calls for it.

Window Manager

I decided to go with StumpWM over XMonad. The practical differences are minute. XMonad uses a workspace-based structure by default, whereas StumpWM treats windows individually[1]. Stump treats all windows equally, where XMonad has the concept of a master window in a given layout. StumpWM assumes C-t as the mod key, and supports Emacs-style chords out of the box. You can use XMonad.Actions.Submap to get some of the functionality back, but there are two places it falls short, and those have annoyed me enough to switch back over to the Lisp-based WM.

The first shortfall is, even though you can technically use the submap feature, there doesn't seem to be a good way of simulating the taken keystroke. That is, if you set your XMonad mod key to C-t, you now have no way of using transpose-chars in Emacs or New Tab in Chrome. StumpWM has a mechanism to let C-t C-t fake the usual C-t keystroke to the focused application, but XMonad has nothing similar that I've found.

The second is that certain XMonad keystrokes are designed to have you hold the mod key, and submap chord keys can't be held. The best example of this is resizing windows. The standard keystrokes are Mod-h and Mod-l, for growing and shrinking the master window respectively. The way these work is that each invocation grows/shrinks the master window by about 5px, so if you want to do any significant resizing, you'll need multiple calls. If you bind Mod to a single key, like Win, you can do a series easily; hold Win + h h h h h. If on the other hand you want a chorded mod, it gets more complicated; C+t h C+t h C+t h C+t h C+t h. That's pretty annoying. StumpWM doesn't let you hold chorded keys either, but because they're the default, Stump keys tend to be designed to account for it. For example, the resizing situation above is solved with a separate interactive-resize setting; you hit C-R, which puts you into a mode where your arrow keys scale the focused window.

I suppose another solution could have been "get used to Win as your mod key", but I don't wanna. I'm working on a laptop, so that key is annoyingly narrow, and hitting it properly with my pinkie requires me to scrunch my left hand up somewhat uncomfortably.

So that's that; keeping my hands on the home row without sacrificing functionality is enough to drive me to another window manager.

Other Installables

Other than the WM, I mentioned this was my first time going desktop-less. That produces one or two challenges, the big one being wireless. I'm using wicd-curses to manage my connections, but that's actually the easy part. The x220 comes with one of three built-in wireless cards, and none of them like Debian very much. What I really ought to do is go out and buy a wifi card that supports free drivers, but in the meantime, this is the one place on my system where I use Debian's contrib non-free repos. I temporarily added them, and dropped them as soon as I was done installing firmware-iwlwifi, firmware-ralink and firmware-realtek.

I honestly don't know which of those did it, but on the next restart, I had wlan0 available.

The other challenge with using a standalone WM is mounting/unmounting USB drives. I don't actually use them very much anymore; scp, git and rsync are much more effective at moving files across machines. The only time a thumbdrive comes out is when I need to do a system install, or when I need to exchange non-emailable files with a non-linux user. For those times, pmount is more than sufficient.

Other than that, I just need to get used to using acpi to check on my battery periodically, and using alsamixer to set up the volume the first time. It goes without saying that caps-lock is an additional ctrl.

The list of things installed on my system at this point is pretty short actually. Here's a script that duplicates most of the install

### with the contrib non-free repos enabled
apt-get install firmware-iwlwifi firmware-ralink firmware-realtek
### disabled again

apt-get install wireless-tools wicd-curses
apt-get install sbcl clisp erlang erlang-doc
apt-get install make screen dmenu htop gnupg git-core gitk emacs stumpwm slime pmount
apt-get install pacpl mplayer alsa imagemagick gimp inkscape conkeror chromium-browser

I haven't even bothered with databases or web servers yet, though I'm sure I'll have to eventually. I did grab a few applications from source just for the hell of it, and set up my usual utility scripts[2], quicklisp, plus 7 or so .*rc files.

Lineup Changes

There are a couple of big things I've changed, that you may have noticed, and one big thing I've changed that you definitely didn't. Most of it is pruning things that I've noticed I don't use. The languages I didn't bother installing this time include haskell[3], smalltalk[4], node.js[5] and ruby[6].

Finally, the latest version of git-core no longer ships with git.el. That's not entirely a bad thing; I've had to hack a lot of additional pieces onto it for my purposes, and I always sort of wished that it just worked out of the box. It turns out I was one of the ~3 people on the planet not using magit. Luckily, the lack of direct git-core has forced me to try it out, and it seems that this mode supports everything I was adding and then some. One or two annoyances, but I haven't run into anything that takes more than a trivial change in my workflow.

So yeah. Net gain, all told.


1 - [back] - You can use groups to approximate the XMonad model, but I haven't played with it much yet.

2 - [back] - Though I will be porting the useful ones away from Ruby very shortly.

3 - [back] - Which I've been playing around with periodically, but haven't used for anything serious.

4 - [back] - Which I'm definitely coming back to at some point, but don't have the time for at the moment.

5 - [back] - Which was enough of a pain in the ass to install properly that I'm avoiding it until I actually decide to use it.

6 - [back] - Which I'm really sad about actually. However, I haven't used it for anything but scripts for the last little while. The number of scripts I've been writing in it has also been going downhill since I started using make actively.

Monday, June 11, 2012

Authentication Part Two

Type your name, click "login",
Now count to three

Come with me and you'll be
In a world of pure authentication
Take a look and you'll see
Into web authorization
We'll commence with a glance
at classic watchword authentication
Where we'll try for complete explanation

If you wish real security
Use RSA to guarantee it...

Ok, that's quite enough of that. There are reasons I don't consider myself a writer, and bullshit like this is right up there.

I've spent the last couple of days doing some quick research on RSA-based, auth systems. This may end up being another one of those things I was severely overconfident about. Since finding out that mathematicians have formalized this thing called Public Key Encryption, I vaguely assumed that someone had put together an authentication system that uses it, and that the only reason such an easy-to-use and secure system was losing out to traditional password auth was... well, I don't know, but it had to be pretty good, right? It turns out that this is only a theoretically solved problem.

Reading material on the subject includes one pretty good statement of the problem, one Enterprise implementation of a similar system, an article musing idly on the possibility, and one or two people wondering the same thing as me: why aren't we already doing this?

At the high-level, there's two ways of implementing such a system (which could, and probably should, be combined):

Prove that you can read this

  1. The server sends the user a random ~64 byte code, encrypted with the users' public key
  2. The user decrypts the key and sends back the plain-text
  3. The server verifies that the plain-text it gets back corresponds to what it sent in step one
    • If it does, that code is revoked and the user is given access.
    • If it doesn't, boot the fucker

Prove that you can sign this

  1. The server sends the user a random ~64 code
  2. The user signs that plain-text and sends the result back
  3. The server verifies that what it got matches the plain-text from step one, and was signed by the user trying to log in
    • If it does and it was, that code is revoked and the user is given access.
    • If it doesn't or it wasn't, boot the fucker (and probably make a note of the break-in attempt)

Like I said, you could combine them at low effort, though I'm not actually sure it would add any kind of security boost over one-or-the-other. The trouble, I assume, is the UI; the simplest possible way to implement this system involves some pretty odd (odd for the average computer user on the interwebs today) steps for the user.

First, logging in becomes a minimum two-step process. Three, really, if you count decrypting/signing as a step. Because the server needs to know who you're trying to log in as before generating and sending your code, you can't identify yourself and send the answer at the same time the way you can with password systems. You need one, then the other.

Second, the user needs to decrypt/sign output from the server. This is non-trivial for most people, or at least, that's the only conclusion I can draw from the fact that most email is not encrypted. Barring trickery, this would need to be done manually; copy the message out of an HTML page, paste it into your PGP/GPG client and have it do its thing.

Third, the user is now effectively tied down to a single computer for their browsing experience, since you can't exactly carry an RSA key around as easily as a passphrase/password. At minimum, you'd need a USB key, and you'd need to trust that computers you were using it with didn't secretly keep a decrypted copy of your key around for nefarious purposes. Good luck with that, I guess.

The forever hack still works, and will continue doing so ... well, forever, but we do gain some real advantages by using public key auth rather than passwords.

  1. No one can sniff your key
  2. Brute-forcing a key is much harder than brute-forcing a password
  3. You don't need to remember passwords (other than the one that encrypts your key)

In other words, if we could solve that UI problem in a semi-automated way, this would be an altogether better way of doing web authentication.

The Plan

I actually intend to build this, because having such a system would be a good thing from my perspective personally, as well as for web security in general. If no one's done it before, I guess I may as well take a crack at it. The steps are already outlined above

  1. [client] requests page requiring auth
  2. [server] asks for a user name/identifier
  3. [client] enters user name/identifier
  4. [server] sends encrypted string, records it, expects signed plain-text
  5. [client] decrypts, signs string, sends it back
  6. [server] compares with what was recorded in 3, authenticates or boots based on comparison

Step 5 can potentially be handled by a browser plugin. Something that would look for an RSA-auth form, and do the right thing. Either by tracking your auth keys itself, or by calling out to OpenSSH or similar on the client side. The server-side actually looks pretty simple, and needs minimal changes from the auth system we put together last time. Really, we'd just need to store a users' public key instead of (or in addition to) their password, and use the Erlang crypto:rsa_*/\d functions to process the specified messages.

Step 5, I'm going to want to think about/research some more. I know there are various RSA implementations in JS, so it's at least plausible to write a browser plugin that can generate a separate key for you for the purposes of website authentication. Another option is to interface with an external program, such as OpenSSH or GnuPG, but that's something with which I have limited experience.

I'll leave those thoughts in the air for now. Feel free to tell me how stupid all this is, and what a better solution would be. I'm off to do some research, before laying down any more words.