Update, and the Finer Points of Quasimodes

I was going to say "this'll just be a quick update", but on reflection, I'm not sure that's true. I want to discuss two specific things I recently implemented, as well as what's been going on lately project-wise.

First things first.

AOSA Chapter

I'm doing a chapter for the upcoming book called 500 Lines or Less. It's the fourth installment in the Architecture of Open Source Applications series. To, I should hope, no ones' surprise, I'm doing a write-up of the House asynchronous server currently serving as the back-end for the Deal project. I've seen fit to simplify the server a bit for entry, since the real live one masses slightly more than 500 lines. You can see the result here; that's the House server with the session and static file mechanisms stripped out, which happens to weigh in at just over 400 lines of fairly readable Common Lisp. The first 1/2 draft of the prose write-up is here.

This process is new to me. Granted, I write a fuck-ton, but I'm usually also the editor and proofreader. Frankly, I don't envy anyone that job. It may not look like it, but I end up chopping a good half of my output before it gets posted anywhere, and then tweak about a third of the result after the fact. Hopefully, involving other eyes will mean a more refined exposition than I usually manage.

2dmacs

I'm still working on that visual editor at work. Internally, we've taken to calling the project "2dmacs", pronounced "two-dee-macs". Hopefully, I can live up to the name. We're going to do some user trials tomorrow, just to see what the target audience of 5 thinks. It's still missing one or two big features, mostly history related and mostly pretty straight-forward to implement. But based on my own experience, it's already more productive than the alternatives for the sorts of things we'll be doing. I'll let you know how it goes.

Now then, moving on to a couple of implementation details. Except that I still can't show you code. Sorry.

Quasimodes

Quasimodes are another thing I picked up from that Raskin book[1]. The idea is to let holding a particular key kick off a different interaction mode and cancel out of it on release. Raskin's example is of the Caps Lock and Shift keys. Basically, Caps Lock switches the user into a mode where standard keyboard keys do something different, whereas Shift activates a quasimode to the same ends. The argument is that the second is better because there's no confusion about what state the system is currently in. If you're holding shift, you're in shout mode. If you're not, you're not. Comparatively, as long as your keyboard has a Caps Lock key, you might be dumped into Shout mode by an accidental keypress. If you've ever tried to type in a password, you know the problems this can cause. Quasimodes neatly sidestep the issue by keeping a mode active only as long as the user deliberately holds a particular key.

Implementing this in Javascript turns out to be non-trivial. Even after you've built up your own little event system around the various input actions a user can take. What you naively need to do is capture the keydown event, activate a given quasimode[2], and clean up when you get a keyup corresponding to the initiating keydown.

That has some implications, though.

First, it requires the ability to optionally route events through particular functions. Which means that you have to have a layer of indirection between the default HTML event handlers and your systems' commands. Luckily, the system I'm building had that already for unrelated reasons, but if you're trying to do this properly in your system, it might mean some architectural changes.

Second, because we're dealing with browser events, we need to be able to cancel out of a quasimode with something other than the initiating keydown event[3]. In my case, I had already wired <esc> as the generic cancel button. It already drops you out of half-completed keystrokes and resets some small pieces of internal state to their original positions. So it made perfect sense to just add "exit any quasimodes" to the list of things it does.

Finally, unless you want to let quasimodes monopolize the keys they'll be bound to, it means having some logic involved that distinguishes between a regular keypress and a quasimode invocation. That ended up being resolved a touch hackily; if you press a quasimode key, but don't trip any of that modes' internal bindings before releasing it, it gets interpreted as a vanilla keypress. Not sure that's the best approach long-term, but I can't think of a simpler one off the top of my head.

Selections

A particular type of quasimode that gave me some pause is area selection. It turns out this is more complicated than you'd think at first glance. If you have a system that models various things as sets of points, and points as well as things are selectable and groupable, then there are a bunch of independent axes that you might be thinking about selections:

additive/subtractive/replacing You might reasonably be looking to add to your existing selection, remove elements from it, or replace your existing selection with the set of elements you're about to specify by area.

intersecting/containing You might want to select any thing whose bounding box touches your selection, or you might want to limit yourself to objects that are entirely contained inside said selection.

element/point You might want to select things, or you might want to select the points that compose them.

top-level/deep You might want to select the top-level groups of elements, or you might want to select the leaf nodes.

The point here is that these are all axes on which the user might choose independently, and most of them are reasonable choices. However, you don't want to have 24 different quasimodes, or even one quasimode with 23 different modifier keys to accomodate them all. After giving it some thought, I made the drag-select interaction default to an additive, intersecting, element, top-level selection. My intuition is that point and subtractive selections are going to be rare and simple enough that we won't need area support for them, that replacing selections are easy enough to counterfeit by hitting the clear-selection keystroke before making a selection, and that I can add modifier keys to distinguish between intersecting/containing and top-level/deep if needed. I'll let you know how that goes too.

---

Footnotes

1 - [back] - Which I thoroughly recommend for anyone involved in interface design of any kind. I'm kind of surprised they never made me read it in college, alongside Design of Everyday Things.

2 - [back] - Which means running its bindings instead of the global ones for the duration.

3 - [back] - An alert, or forced focus switch might take the user away from our window before we get the intended key released.

Which Lisp Should I Learn?

I don't know why this keeps coming up lately, but it does. So, here we go:

My Recommendation

If you want to learn your first Lisp and already know something about the JVM[1], you should learn Clojure[2]. Otherwise, you should learn Scheme. Specifically, I recommend going the route that takes you through Racket[3], and possibly through SICP or HTDP.

If you absolutely, positively must, I guess go ahead and learn Common Lisp.

Why

Clojure the language, as opposed to the current, main, half-fused-with-JVM implementation, is cleaner and more consistent than Common Lisp, which should help you learn it more easily. I've gotten back talk about how there are lots more noobs learning Clojure, and as a result their libraries are in some disarray, and about the fact that the JVM is a sack of donkey balls you have to bite into every time you hit some sort of error[4], and about the general Clojure community pre-disposition to fashion trends. All of which may or may not be true, but I'm specifically talking about the language, not its ecosystem or stalwarts. Now granted, all of Racket, Clojure and Common Lisp

1. are built out of s-expressions
2. have defmacro

so depending on how much work you're willing to put in, you can do whatever the fuck you want in all of them[5]. However, in addition to knowing about prefix notation, and macros, and general Lisp program structure, here's an incomplete list of idiosyncrasies of Common Lisp that you have to commit to memory before you can be effective in it:

• Functions and variables are in different namespaces, and each has constructs that deal with them explicitly. You need to use defvar/defun and let/flet depending on whether you're using functions or variables. If you're passing symbol names around, symbols that denote variables can be sent around as 'foo whereas symbols that denote functions should be sent around as #'foo. If you're trying to pass a function around in the variable foo, you need to (funcall foo arg) or possibly (apply foo args), rather than just (foo arg).
• Most functions that deal with lists are functional, except the standard sort and the default mapcan, both of which destructively modify the list you pass them.
• You can define methods for your classes, but can't easily use certain default names that are bound to top-level functions. Such as length, or the arithmetic primitives. Which is why you frequently see methods like duration-add or matrix-mult.
• There are 7 commonly used equality operators, eq, eql, equal, equalp, string=, char= and =[6], each of which has mildly different, sometimes implementation-specific, behavior. Granted, because CL isn't a pure language, you need at least 2 of those, but 7 is still a bit much to have people memorize.
• There are three different local binding mechanisms that you must decide between depending on whether you want to be able to refer to earlier symbols in the same binding set, or whether you want symbols to be able to refer to themselves. You use let/flet if you don't care, let* for variables where you want later bindings to be able to refer to earlier ones, and labels for functions where you want bindings to be able to refer to themselves or earlier bindings.
• There are many, many implementations of Common Lisp. The popular ones at the moment are SBCL and CCL, but I've personally seen CLISP, ECL and Lispworks around too. More are available, and you might run into a lot of them in the wild. If you want to write portable code, you have to jump through some hoops. The implementation-specifics range from the finer points of equality operator behaviors, to the behavior of handler-case[7], to the types you can specialize on with defmethod, to the presence and behavior of threads, to the contexts in which you can pass streams around, to the names of various extension functions. For a small but representative example, take a look at what it takes to temporarily change your current directory in an implementation-independent way.
• Indexing into different constructs takes different forms. You need nth for lists, aref for arrays and vectors, gethash for hashes and slot-value for CLOS instances[8].
• You can't use the symbol t anywhere. No, not even local scopes. If you try, you'll get warnings or errors[9], because t is the top-level designated Truth symbol, even though anything other than NIL evaluates truthily in a boolean context.
• A hash isn't a list, and a CLOS instance isn't anything like either. One way you'd like them to be similar is when you're traversing them. It seems fairly reasonable to expect map-likes to operate on hashes by treating them as a sequence of key/value pairs, and instances by treating their bound slots as key/value pairs. This is not how things work. If you want to map over instances that way, you need to do something like this and this. If you want to map over hashes, you either use the hilariously mis-named maphash[10] or some idiosyncratic piece of the loop DSL that lets you iterate over hash keys and hash values.
• Common Lisp is case-insensitive. It takes whatever symbol input from you and upcases it internally. So foo-bar and foo-Bar both become FOO-BAR behind the scenes. This is usually not a huge problem, unless you try to interoperate cleanly with newer data standards like CSS or JSON. That leaves you fumbling with strings in situations where symbols and/or keywords really ought to do.

Like I said, this is a small sample. Just the stuff I thought of off the top of my head. I'm sure I could come up with more if I put a day or two into it. And I'm far from the most experienced Lisper out there, others would have more finer points for you, I'm sure. But that's half the problem with little issues like this; experienced Lispers completely forget about them. It's the newbs that have trouble cramming these things into their heads.

When I take a good look at that list, and then imagine the situations that led to each element, it's difficult to conclude that a wrong decision was made at any given point in time. Unfortunately, the sum of all of those potentially correct decisions is a giant system, the inherent rules of which look inconsistent if not outright hostile to human minds.

I don't know if Clojure solves all of them.

I've done very little work with it, for reasons entirely unrelated to the language. For all I know, when you get deep enough into it, you get to inconsistencies and/or restrictions which are worse than anything I've pointed out or could.

But do me a favor, if you're a CL user, either hop over to this web REPL, or install leiningen then hop into your local lein repl and type along here:

user=> (def thing [8 7 6 5 4 3 2 1])
#'user/thing
user=> (thing 0)
8
user=> (thing 3)
5
user=>(map (fn [n] (+ 3 n)) thing)
(11 10 9 8 7 6 5 4)
user=> (def thing {:a 1 :b 2 :c 3})
#'user/thing
user=> (thing :c)
3
user=> (thing :d 6)
6
user=> (thing :a 6)
1
user=> (map (fn [[k v]] (+ v 2)) thing)
(3 4 5)
user=> (def thing #{1 2 3 4 5}) ;; a set, in case you were wondering
#'user/thing
user=> (thing 3)
3
user=> (thing 0) 
nil
user=> (map (fn [a] (+ a 2)) thing)
(3 4 5 6 7)
user=> (def triple (fn [a] (* a 3)))
#'user/triple
user=> (triple 4)
12
user=> (map triple thing)
(3 6 9 12 15)
user=> (map (fn [a] (let [t (- (triple a) 5)] (* 2 t))) thing)
(-4 2 8 14 20)

Now think about how you would go about explaining to a novice programmer that it has to be more complicated than that.

---

Footnotes

1 - [back] - And don't have a strong dislike for it, obviously.

2 - [back] - Install it through Leiningen, which is available in the Debian repos in stable and unstable.

3 - [back] - Yes, I'm fully aware that the Racket guys are trying to push this "We're totally not Scheme" thing. They're close enough from an external perspective. Just don't tell Jay McCarthy I said so.

4 - [back] - Which is certainly true, but mildly preferable to the alternative as long as you're used to that sort of thing.

5 - [back] - Except that Clojure is apparently missing Reader macros, which I always thought were kind of half-assedly implemented in Common Lisp. For what I consider the full-ass version, take a look at how Haskell does it.

6 - [back] - Plus how many ever *-equal functions you define for your own classes.

7 - [back] - The Common Lisp answer to the problems that call for try/catch in other languages.

8 - [back] - For the last, you can also define your own selectors using :reader or :accessor declarations.

9 - [back] - Which specific warning or error depends on implementation.

10 - [back] - Because it's not very much like map. It returns nil and works by side-effect. Meaning that if you expect a sequence from it, you'll need to construct it yourself.

Debian Testing, Pi and Git

That was a vacation, I guess.

It was suspiciously taxing, all in all. Time off from work hasn't been nearly as relaxing since we had a kid, but that's a digression. Over the past little while, I've managed to finally make use the 120G solid state drive I picked up half a year ago, install various distros, and put together about one third of a utility to ease a project or two I'm working on in my spare time.

New Drive

It's at once larger and smaller than the last one. On the one hand, thanks to its smaller physical profile, I can fit it into my laptop with no mods. On the other hand, df -h says 106G[1] instead of ~28G.

That's it, nothing else to see here.

Fresh Install

Since I was between drives anyhow, I took the opportunity to get the fresh version of Debian up and running. That was worth it, by the by, if for no other reason than they've apparently poured enough bucketfulls of time into the networking code that I can now reliably connect to my wifi access point even if I'm not within two meters of it. They also seemed to lick a problem I kept running into wherein the shutdown process would hang the machine[2].

There were a few changes in my install routine, which is still vaguely based on

## temporarily add
## deb http://packages.linuxmint.com debian import
## and `contrib non-free` to /apt/sources.lisp

apt-get install firmware-ralink firmware-realtek
apt-get install screen make emacs24 git gitk wicd-curses pmount htop gnupg unetbootin
apt-get install mplayer feh pacpl imagemagick x-window-system dmenu xmonad gimp inkscape firefox
apt-get install python-pip sbcl vrms

## remove the temporary repos

There are a couple of changes there from my usual. Firstly, Firefox has become my go-to browser. Its absorbed most of the goodness from Chromium, including the reduced toolbar footprint and fantastic JS console. It also has support for adblock, and a fairly good RSS feed reader, and it's no longer slow as molasses[3]. This raises the problem of the Debian packaging though; in the official repos, apt-get install firefox gets you a pretty ham-fisted re-brand with no plugin support called "Iceweasel". What I ended up doing, as you can see above, is temporarily adding the Linux Mint repo to install that[4]. Secondly, I'm installing emacs24 rather than just plain emacs. This is because the current default for emacs in Jessie is Emacs 23.something, and that doesn't have one of the main features I'm looking to finally adopt.

Emacs 24 supports package out of the box. In practice, this means adding

(require 'package)
(add-to-list 'package-archives '("melpa" . "http://melpa.milkbox.net/packages/") t)

to my .emacs instead of toting my old .emacs.d/ around. I can't actually remember every library I used to have around, so the list I settled on this time out ended up being

aes auto-complete autopair highlight-parentheses htmlize skewer-mode magit markdown-mode paredit redo+ smart-tab yasnippet

Which covers pretty much everything. Oh, one thing. I spent about half an hour figuring out what was going wrong with my .emacs config; libraries I was certain had been installed were coming back with not found errors when I tried to require them. It turns out that when you add a directory to the load path, you don't automatically add all its subdirectories. As you can see by the above list of packages, I use quite a few, each of which gets its own sub-directory in .emacs.d/elpa/, and wildcards don't work here either. So I was forced to add the following to convenience.el, just to save myself the tedium

(defun starts-with-dot-p (path)
  (= (aref path 0) ?.))

(defun list-subdirectories (path)
  (let ((all (mapcar 
              (lambda (name) (concat (file-name-as-directory path) name))
              (remove-if #'starts-with-dot-p (directory-files path)))))
    (remove-if-not #'file-directory-p all)))

(defun add-to-load-path (dirs)
  (mapc (lambda (p) (add-to-list 'load-path p)) dirs))

then called this near the top of that .emacs file:

(add-to-load-path (list-subdirectories "~/.emacs.d/elpa"))

This let me continue as normal. The only omission from that emacs package list is slime, which I've lately been installing from sbcl or what-have-you with (ql:quickload :quicklisp-slime-helper) rather than through Emacs itself. It works exactly as well as you'd expect, which is to say flawlessly.

Dicking Around With Pis

Doing that got me into an installing mood, so I also formatted a fresh couple of SD cards with the latest versions of ARM Arch and Raspbian respectively. I did this with the vague intention of getting deal to work with one or both, and it looks like that'll take a bit more work than just a straight-up ql:quickload. Differences before I get to that though.

The RPi arch is much closer to what I'm used to on my laptop. A brutally minimal installation of the few core utilities you need to get basic shit done, and nothing else. Specifically, it gives you pacman, perl, a working ssh server and a minimally intrusive wireless connection mechanism that could replace wicd-curses for me. Raspbian, by contrast, bundles a mandatory window environment along with a bunch of crap that's probably nice for most humans looking to use it as a desktop replacement, but that I'll never end up touching. Also, they bundle Scratch as well as Python 2 and 3. Finally, while they do provide an ssh server, it's off by default, and the first time you boot a Raspbian image, it forces raspi-config, which means that you must connect a Raspbian Pi to a monitor and keyboard at least once.

That minimalism ends up biting Arch a bit though; it doesn't come with the standard raspi-config utility, which lets you mess around with the hardware to some small extent, and easily resize the installation partition to fill the SD card[5]. The other thing that bites ARM Arch in the ass, as far as I'm concerned, is the fact that its package manager has very few of the things I want to install. Out of my usual menagerie, I found screen, make, clisp, python, emacs and nothing else. By contrast, I had to apt-get --purge a bunch of things over on Raspbian, but I was eventually able to get it working with an almost copy of my laptop environment.

Almost, because Lisp still has some problems.

Specifically, clisp segfaults on both ARM Arch and Raspbian when you try to load anything with quicklisp, while the ARM ccl failed to run at all on Arch[6]. Raspbian did run ccl appropriately, but errored out on me for two reasons. Firstly, there's something unsupported about the :ironclad MD5 digest, and secondly, the ARM architecture seems to treat bivalent streams differently than x86. Which means that even running its custom house server, :deal errored out.

I'll be trying to fix that over the next little while.

cl-git-fs

Finally, on a merely semi-related note, I'm working on a couple of projects on my own time that are eventually going to want to do some sort of file management. And I figured it would be nice not to have to bring git into it manually after the fact. To that end, I took a look at how gitit manages the trick of using git as a faux-database for its wiki pages. It's not that complicated, as it turns out. And here's the result of spending an hour or two porting that piece of functionality to Common Lisp.

The biggest problem I'm running into is that there isn't a standard shell-command or run-program defined in the various Lisps I want to support.

I'm tossing it up to my github, but calling it 0.01 because there's a fuckton of functionality and documentation missing. In particular, it currently only supports SBCL on Linux, and a few of the external API functions still return raw string results, rather than properly parsed CLOS instances. The documentation and parsing will be a priority no matter what, but I'll only see how it runs on other platforms and implementations as I need to deploy to them.

---

Footnotes

1 - [back] - Of course, the drive box says 128G, so Samsung and all drive manufacturers are lying shitbags, but I'm digressing again.

2 - [back] - And therefore keep drawing power until a forced shutdown.

3 - [back] - which it was last time I played around with it.

4 - [back] - No, since you ask, I've never just straight up tried Mint. It has something in common with most of the distros I get recommended, which is that it cribs heavily from Debian on everything that matters, and then tries to differentiate on the desktop environment almost entirely. Not that this is bad for end users I guess, but as you can see from the x-window-system and xmonad items in that installation list above, I do not use what you would think of as "a desktop environment". Don't let that stop you from trying it, of course, but I'm not going to.

5 - [back] - You can still do this externally via gparted when you image your SD card.

6 - [back] - running the included binary gave me a "wrong architecture" error, even though there's no way that's accurate.