cl-notebook Thoughts

No nuts and bolts this time. Here's a random collection of insights I've had while trying to put together the version 0.1 of a notebook style editor:

Being async Pays

My initial assumption was that evaluation would be an inline thing. That is, you send out a POST request, that code is evaluated synchronously, saved out to your fact base, and the HTTP response consists of that evaluated result which you can then display prettily on the front-end. Turns out that's a big fat no. Even discounting any issues with the Common Lisp return model, and the eventual attempt at making this a multi-user editor, the synchronous approach doesn't quite work. Which I found out the first time I accidentally wrote a loop without the required while. Something like

(loop for next = (pop stack) do (something-to-next))

The trouble should be obvious. That's the sort of computation that you want off in its own isolated thread. And you want to be able to kill that thread if it turns out to be as boneheaded a mistake as that was, which means that you have to notify the front-end of computations in progress, go off and perform it, then notify the front-end again when you're done. And that's not something you can do synchronously.

In the end, I end up spawning a single tracked thread to do the work of evaluation. I notify the front-ends when it begins work, and then again when it completes. At any point in between, a front-end can send a termination signal to kill that thread rather than waiting it out to completion. You can see the implementation here, here, and here. This will entirely coincidentally make it much easier to extend cl-notebook to a multi-user editor later.

Being surgical pays. Sometimes. Not in the way you'd think.

The first version of this editor basically spat out the complete current notebook state at the front-end on each action, and the front-end re-drew the entire thing on every change. You'd think that the sheer inefficiency of this approach would get to me, but you'd be wrong. It was actually fast enough that had performance been the only factor, I'd have left it at that. The problem with that approach is that you end up clobbering giant chunks of client-side state each time. In particular, re-drawing every cell (and hence, re-initializing each of their CodeMirror instances) meant that I was blowing away undo information for each editor window every time anything happened. And that's annoying as fuck. I'm not sure anyone's formalized the point into a commandment yet, but you really should act as though they have: Thou Shalt Not Mangle Thy Clients' Data. That applies no matter how temporary the data is. Even if your job is to take said data and transform it in some way, say for instance by evaluating it, you should do so with a copy instead of a destructive change. And even when you absolutely must be destructive, be as selectively destructive as you possibly can.

Thinking About Space Doesn't Pay. No, not even that much.

I almost left this one out entirely because it seemed so self-evident, but on reflection, it's something I've had to learn too. Space doesn't even begin to matter for systems like this. I'm talking both about memory and about disk space. Yes, there are some applications for which this is not the case, but it's true as a rule.

The fact-base project in particular has been sticking in my craw in this sense. I kept thinking things like 'Holy shit, I'm keeping history for every cell, on every edit forever. This is going to be huge! I'll have to figure out a way to condense these files, or start them off from a non-zero state so that I can throw out history at some point!'

Completely pointless.

At the beginning of the cl-notebook effort, I started a scratch file which I've been steadily editing, putting through various save/auto-save tests and just all round mauling with updates. This thing has been around for months at this point, and it's taken much harder beatings than the average notebook ever will. Wanna know what its current total size is?

inaimathi@self:~/.cl-notebook/books$ ls -lh base-zg1mm23z
-rw-r--r-- 1 inaimathi inaimathi 675K Apr 20 21:47 base-zg1mm23z
inaimathi@self:~/.cl-notebook/books$

That's smaller than many Word and PDF documents I've worked with, and those don't bother keeping my entire editing history around. So I figure I can get away with treating disk space as if it were infinite for my purposes. Absolute worst case scenario, I'll compress it. And since I'm dealing with plaintext files, that should be rather effective.

inaimathi@self:~/.cl-notebook/books$ pack -t tar.gz base-zg1mm23z
base-zg1mm23z
inaimathi@self:~/.cl-notebook/books$ ls -lh base-zg1mm*
-rw-r--r-- 1 inaimathi inaimathi 675K Apr 20 21:47 base-zg1mm23z
-rw-r--r-- 1 inaimathi inaimathi  23K Apr 20 21:47 base-zg1mm23z.tar.gz

Yeah. I think that'll be good enough.

Return Values are Complicated

I mean, I knew that already, but it turns out there are even more intricacies here. I initially assumed I'd be able to just keep a single return value per cell (by which I mean the return from a single Lisp function, which can be zero, one or more values). Then it hit me that a cell might have more than one expression in it. Then it hit me that return values aren't enough; you need to be able to handle *standard-output* emissions and warnings on a per-expression basis rather than on a per-cell basis, and that we'd want type annotations in some places since we'll be serializing various things to strings and it would otherwise get confusing. Then I hit me and sat down to write down something workable. Each cell now stores a result, which is zero or more values, each of which is actually a value and a type.

That lets the front end figure out what it needs to do on a per cell basis, which means that the server-side implementation of a cells' noise becomes very mechanically simple. It's basically just an extra fact we keep around as a label, which the front-end queries to decide how to transform the high-detail result.

cell-type is not the same thing as cell-language

Early on, I had this idea that I'd be semi-implicit about what's in a cell. At that point there were two cell-types; common-lisp and cl-who. The idea would be that this single cell-type would determine both display and evaluation properties of the contained code. Nope, as it turns out. And the thing that finally made this clear to me is thinking about how I'd treat test code. It's still Common Lisp, you see, so I'd still be evaluating it the same way as any other code cell, but I didn't want it showing up in certain exports.

The solution I ended up settling on is to be explicit about everything. Each cell now has a cell-type as well as a cell-language. The first one being one of markup (for prose blocks), code (for code blocks), and tests (for code blocks that I'd want to separate from actual runtime code).

Naming things is difficult

I think there's a joke about this somewhere. Something along the lines of

The only two difficult problems in programming are naming things, cache invalidation and off-by-one errors.

and man did that ever bite me this time. It's obviously bad to tie the file-system name of something to its display name, if for no reason other than it opens up various injection vectors that you'd rather not open up. It turns out it gets even more complicated when you're dealing with history trees of various documents, and you're trying to reduce headaches for your users. Here, think about this problem for a bit. Say you had a document type that you'd let your users name. We're obviously not naming the on-disk file after the display name of the document, so this is a matter of storing a record in the document that'll keep a user-entered name around for reference purposes. That gives you the additional benefit of being able to roll back renames, and the ability to see what a given document was called at some point in the past. Now, say you want to be able to branch said document. That is, instead of being a single history line, you want to be able to designate certain timelines as belonging to different branches than others. What you need now is four different levels of naming. Five, depending on how ham-handedly you've decided to store and relate those branches. At minimum you need

• The filename of the document, which is different from
• The display name of the same document (which might be different in different branches, and at different points in time), which is different from
• The display name of a particular branch of the document (which might need to be human readable, or user entered) which is different from
• The collective, still human-readable name for a set of branches belonging to one document.

If you've stored a branch collective as a file/folder on disk, you'll have to name that too. So, what would you do?

Confronted with this problem, I punted. Branching is basically going to be a copying operation. What you'll eventually get, once I put the branching system together and you try to use it, is a complete, self-contained fact base that happens to (possibly temporarily) have the same display-name as its origin (plus the word 'branch'), and a fact or two that point to a specific time point in that origin. From there, they'll diverge and be entirely separate entities. No, I'm not entirely sure this is the best approach, or even an acceptable approach, but it seems to be the only way to avoid asking the user to manage four different names in the space of one document. So I'll take it.

There's probably more where those came from, but they're all I could pull out of my head at short-ish notice. I'll try to let you know how the rest of the project goes, as it happens.

cl-notebook Introductory Thoughts

So it's about time I talked about this thing, and what the hell exactly I'm thinking. Because I've been working on it for a while, and while it's still kind of buggy, I've already found myself wanting some of the features it has when working with Emacs, or other text editors I've had to work with.

Notebooks

Actually, before I get to that, a little exposition. As far as I know, notebook-style editors already exist for Python, R, and Clojure. And a second one for Clojure. The general idea is to have a web-based interface, with code being divided into small, re-arrangable chunks called cells, each of which are associated with their evluation results. Some cells are code in whichever language the notebook supports, others are just prose, usually in markdown. The idea is that you get a dynamic environment that lets you selectively evaluate small chunklets of code, and intersperse relevant documentation in the form of prose and tests.

As someone who's been eyeing literate programming techniques for a while, this predictably appeals to me. So I've built my own, for my language of choice.

cl-notebook

You can find it at the other end of that github link I start out with. Last time I mentioned this project in passing, I noted that the ultimate goal was replacing Emacs as my Common Lisp IDE of choice, and that's no small task. Despite the existence of subpar, I don't have proper s-expression navigation yet, and I haven't wired up proper auto-completion or argument hinting yet, and there's a bunch of other stuff I still want to build, ranging from the necessary to the frivolous. On the whole, I think I'm on the right track, because certain things are somewhat easier here, and because there are some features that I find myself missing when I hop back into emacs.

Lets just get those out of the way right now, actually. Firstly, I get to program in my browser, which is surprisingly elegant once I hop into full-screen mode. It lets me tab over to search for relevant links to talk about, and since my browser can be set to start up with previously open tabs, I get to resume editing exactly where I was in a later session. Secondly, because of the back-end storage system I'm using, I get to have a running history of all the edits I've ever made, which is updated every time I evaluate a cell (I'm working on having it implicitly updated every so often between evaluations, but don't have that part checked in). Thirdly, I've got exporters wired up that let me put together a book, then export it as an HTML page, or as a .lisp file. And I'm planning to add two more, one to just extract tests and a second to just hand me an executable from the given book.

The first one is minor, and makes it all the easier to randomly check my email or github notifications, so pros and cons. The third could concievably be wired together in Emacs. The second one is huge. I don't know about you, but I've been programmed to hit save every few seconds in whatever editor I've got open just because crashes happen, and I don't want them to be too painful. I guess I could have wired up emacs to do that every so often, but it sounds fiddly as hell. You don't particularly want a standard editor saving every three seconds or so; you might be in the middle of an edit the currently keyed-in part of which doesn't make sense by itself, and most editors 'save' by overwriting your existing file. Which is exactly what you don't want when you've got an unfinished code edit. Hopefully, adding total-history retention to the equation softens the blow.

Core Concepts

Code is organized into books. Each book is the complete history of a bunch of cells. A cell can contain code, tests, or markup in a particular language (currently just Common Lisp, but given how many languages I blog about, it'll probably need at least highlighting support for a few more). The cells' language and type impacts the evaluation approach we take on the back end, as well as which exports it appears in, and in what form. Specifically, common-lisp/markup cells are evaluated as :cl-who forms, don't appear in .lisp exports and only contribute their results to an .html export. By contrast common-lisp/code is straight up evaluated (capturing warnings, errors and standard-output), contribute their contents to .lisp exports, and both their contents and results to .html exports.

In addition to a type, language and id, a cell has a contents, result, and a noise. The contents is what the user has typed in, the result is what that contents evaluates to and the noise dictates how the results are displayed. This is a normal cell:

(+ 1 2)
(+ 2 3)
(format t "Testing ")
(+ 3 4)
(format t "testing")
(format t ". One two.~%")
(format t "Testing.")
(+ 4 5)

Testing testing. One two.
Testing.
9 :: integer

This is the same cell with a noise of verbose:

(+ 1 2)
(+ 2 3)
(format t "Testing ")
(+ 3 4)
(format t "testing")
(format t ". One two.~%")
(format t "Testing.")
(+ 4 5)

3 :: integer
5 :: integer
Testing 
NIL :: null
7 :: integer
testing
NIL :: null
. One two.
NIL :: null
Testing.
NIL :: null
9 :: integer

And again with terse

(+ 1 2)
(+ 2 3)
(format t "Testing ")
(+ 3 4)
(format t "testing")
(format t ". One two.~%")
(format t "Testing.")
(+ 4 5)

9 :: integer

There's also a silent setting which lets you ignore the evaluation result entirely.

You can edit a cell (changing its contents), evaluate it (changing its result) delete it, change any of its mentioned properties, or change the order of cells in a notebook. Each of these is an event that gets initiated by a POST request and gets completed with an event-stream message to any listening front-ends (which means I'll relatively easily be able to make this a multi-user editor when I get to that point). Enough low level stuff, here's an example.

Example

This is a piece of code I actually wrote using cl-notebook.

---

A parameter is a thing that starts with #-. It might be nullary or unary. A parameter followed by a parameter or an empty list is interpreted as nullary. A parameter followed by a non-parameter is unary. Command line args are more complicated in the general case, but not in cl-notebook

(defun parse-args! (raw)
   (pop raw)
   (flet ((param? (str) (eql #\- (first-char str)))
          (->param (str) (intern (string-upcase (string-left-trim "-" str)) :keyword))
          (->arg (str) (or (parse-integer str :junk-allowed t) str)))
      (loop for next = (pop raw) while next
            if (and (param? next) (or (not raw) (param? (car raw))))
              collect (cons (->param next) t) into params
            else if (param? next)
              collect (cons (->param next) (->arg (pop raw))) into params
            else collect next into args
            finally (return (values params args)))))

(defun get-param (names params) 
  (loop for (a . b) in params 
     if (member a names) do (return b)))

GET-PARAM :: symbol

(parse-args! '("./prog-name" "-f" "-h" "something"))
(parse-args! '("./prog-name" "a" "b" "c" "d"))
(parse-args! '("./prog-name" "-p" "4040"))

(get-param '(:f :force) (parse-args! '("./prog-name" "-f" "-h" "something")))
(get-param '(:p :port) (parse-args! '("./prog-name" "-p" "4040")))
(get-param '(:p :port) (parse-args! '("./prog-name" "--port" "4040")))
(get-param '(:p :port) (parse-args! '("./prog-name" "--frob" "4040")))

((:F . T) (:H . "something")) :: cons
NIL :: null
NIL :: null
("a" "b" "c" "d") :: cons
((:P . 4040)) :: cons
NIL :: null
T :: boolean
4040 :: integer
4040 :: integer
NIL :: null

---

It's a small utility function for parsing command line arguments in :cl-notebook. You can see all the relevant features on display there; it starts with some documentation prose in a markup cell, has definitions in a code cell, and finally a bunch of example invocations of each thing in a tests cell. They're not really tests, because they don't encode my assumptions about the return values of those calls, but you could imagine them doing so. The point is, they won't be part of a .lisp export, but will show up in an html export like this one.

That's it for the introductory thoughts. I'll try to gather some insights into such editors into the next piece I put together. And I'll continue dogfooding until it gets good enough to call "delicious".

Housekeeping

Just some quick housekeeping while I draft up my next proper piece.

cl-cwd

A friend of mine mentioned that he had to hack up cl-git-fs to make it export cwd and with-cwd. Which was surprising when I heard it, but really shouldn't have been. This is the sort of problem Common Lispers seem to solve by copy/pasting a few implementation-specific snippets into projects that need the functionality. That's not good enough for me. So, here's cl-cwd; a library to get/manipulate/temporarily manipulate your current working directory in a cross-platform and cross-implementation way. Patches and bug reports welcome as always, and hopefully it's useful for you. I haven't gotten around to hacking the same components out of cl-git-fs, but I might eventually. Or, I might just use the os section of UIOP, which was only pointed out to me after I posted that cl-cwd repo. I'm not entirely sure what the right approach here is; are monolithic utility libraries preferable to very small, single-purpose packages? Not sure, but I kind of prefer the smaller, specific ones for my own purposes. Even though it never wastes enough resources to matter to me, it's deeply irritating on some level to include all of :alexandria and :anaphora for, essentially with-gensyms, aif, awhen and alambda.

fact-base updates and relevant minutia

I've gotten some work done on fact-base, which on reflection really should have been called cl-triple-store, with an eye on using it in production. We'll see if it actually happens, or if it just remains an idle fancy, but

1. you can now add indexes to your fact bases, which should make data retrieval faster[1]
2. writing deltas is now quite efficient[2]

Again, let me know if it's useful, or if it's sooooo-close-to-being-useful-if-it-only-had-this-one-feature. I might do something about it at this stage.

The only related piece of minutia is that I've found myself reading a lot about miniKanren, core.logic and logic programming in general. I think it might be a really good way to query structures like these triple-stores I've been building lately. Also, it turns out someone's already done most of the work of implementing that in CL for me, so I forked it and added/changed the few chunklets I needed to. Matthew Swank, if you're reading this and care, let me know and I'll send you patches. I assumed you wouldn't care, since the copyright line said 2008, but I might have been wrong.

Editors

Someone finally sat down and walked me through the installation for Light Table[3]. I haven't been paying attention to the propaganda, so I'm not entirely sure exactly how this is going to revolutionize editing[4], but it looks promising for a prototype. I was able to get to a productive-ish point with it within about an hour, and that's a high bar. I remember learning Emacs[5] over the course of weeks.

Another one I tried enough to appreciate is IPython Notebook. The idea of a browser-based editor has, shall we say, crossed my mind, but the idea of applying it to 1D coding hadn't occurred to me. I gotta say, I like the idea, and I'll be trying to do something about that right here. I've only got the easy part done so far; a front-end of code-mirror rigged up to a Lisp process that evaluates things and sends the result values back[6]. The hard part is going to involve a persistence layer, multiple cells, multiple notebooks and probably some smaller browsing/presentation/generation features to let me do whatever[7]. Spoiler warning: fact-base and its history/indexing operations will feature prominently. The ultimate goal is no less than replacing Emacs as my Common Lisp IDE of choice.

And that's sort of why I've been on another editor/tools kick lately.

I've been talking to some friends in the local programming community, and we seem to have collectively decided that we really, really want Light Table to succeed because it sucks not being able to recommend anything as a starting tool for Common Lisp development to programming newbies. Our options right now are what can charitably be referred to as a very intimidating operating system/rss aggregator/IRC client/moustache trimmer[8], a wannabe very intimidating blah blah blah, a very pretty, passably comfortable pair of handcuffs, a very performant, robust, ridiculously expensive pair of handcuffs, and vim. If you're recommending Scheme, there's also Racket[9], but that still doesn't count as a Common Lisp environment.

Some of those are things I'd recommend to people who are already programmers. One of them is something that I'd recommend to people looking to learn languages for the sake of learning them. But there isn't a tool anywhere on that list that compares to something like Idle or, more to the point, Light Table. I don't think it's quite at the point where I'd switch over to it myself, but I sincerely hope it gets there. And I'll be devoting some effort to pushing that along. You can too, since it's up and properly licensed and all...

---

Footnotes

1 - [back] - There's an article either in the works or already posted, depending on what order I decide to put these up.

2 - [back] - From a macro-optimization perspective; I'm sure I could pull some micro-optimizations on it too, but I'm not into that.

3 - [back] - Turns out that even the Debian Unstable repo doesn't have a recent enough version of Leiningen. I needed to latest using this.

4 - [back] - In fact, the only visible revolutionization of "randomly putting shit in your buffers that you can't edit instead of giving you a goddamn REPL", is profoundly annoying, though it's possible to disagree about these things. I guess I can imagine it looking really snazzy in presentations and demo videos.

5 - [back] - ...and before that jEdit, and before that Eclipse.

6 - [back] - And yes, it handles multiple values as well as *standard-output* emissions, in case you were wondering.

7 - [back] - At minimum the possibility to write :cl-who/:cl-css instead of markdown, a way of installing libraries that doesn't go through quicklisp, and maybe some git/drakma/shell support. I may end up having to add non-shitty static file serving to house for this...

8 - [back] - Pre-built Lisp IDE version optional.

9 - [back] - Which I do wholeheartedly recommend; it's very cool.

Debian Testing, Pi and Git

That was a vacation, I guess.

It was suspiciously taxing, all in all. Time off from work hasn't been nearly as relaxing since we had a kid, but that's a digression. Over the past little while, I've managed to finally make use the 120G solid state drive I picked up half a year ago, install various distros, and put together about one third of a utility to ease a project or two I'm working on in my spare time.

New Drive

It's at once larger and smaller than the last one. On the one hand, thanks to its smaller physical profile, I can fit it into my laptop with no mods. On the other hand, df -h says 106G[1] instead of ~28G.

That's it, nothing else to see here.

Fresh Install

Since I was between drives anyhow, I took the opportunity to get the fresh version of Debian up and running. That was worth it, by the by, if for no other reason than they've apparently poured enough bucketfulls of time into the networking code that I can now reliably connect to my wifi access point even if I'm not within two meters of it. They also seemed to lick a problem I kept running into wherein the shutdown process would hang the machine[2].

There were a few changes in my install routine, which is still vaguely based on

## temporarily add
## deb http://packages.linuxmint.com debian import
## and `contrib non-free` to /apt/sources.lisp

apt-get install firmware-ralink firmware-realtek
apt-get install screen make emacs24 git gitk wicd-curses pmount htop gnupg unetbootin
apt-get install mplayer feh pacpl imagemagick x-window-system dmenu xmonad gimp inkscape firefox
apt-get install python-pip sbcl vrms

## remove the temporary repos

There are a couple of changes there from my usual. Firstly, Firefox has become my go-to browser. Its absorbed most of the goodness from Chromium, including the reduced toolbar footprint and fantastic JS console. It also has support for adblock, and a fairly good RSS feed reader, and it's no longer slow as molasses[3]. This raises the problem of the Debian packaging though; in the official repos, apt-get install firefox gets you a pretty ham-fisted re-brand with no plugin support called "Iceweasel". What I ended up doing, as you can see above, is temporarily adding the Linux Mint repo to install that[4]. Secondly, I'm installing emacs24 rather than just plain emacs. This is because the current default for emacs in Jessie is Emacs 23.something, and that doesn't have one of the main features I'm looking to finally adopt.

Emacs 24 supports package out of the box. In practice, this means adding

(require 'package)
(add-to-list 'package-archives '("melpa" . "http://melpa.milkbox.net/packages/") t)

to my .emacs instead of toting my old .emacs.d/ around. I can't actually remember every library I used to have around, so the list I settled on this time out ended up being

aes auto-complete autopair highlight-parentheses htmlize skewer-mode magit markdown-mode paredit redo+ smart-tab yasnippet

Which covers pretty much everything. Oh, one thing. I spent about half an hour figuring out what was going wrong with my .emacs config; libraries I was certain had been installed were coming back with not found errors when I tried to require them. It turns out that when you add a directory to the load path, you don't automatically add all its subdirectories. As you can see by the above list of packages, I use quite a few, each of which gets its own sub-directory in .emacs.d/elpa/, and wildcards don't work here either. So I was forced to add the following to convenience.el, just to save myself the tedium

(defun starts-with-dot-p (path)
  (= (aref path 0) ?.))

(defun list-subdirectories (path)
  (let ((all (mapcar 
              (lambda (name) (concat (file-name-as-directory path) name))
              (remove-if #'starts-with-dot-p (directory-files path)))))
    (remove-if-not #'file-directory-p all)))

(defun add-to-load-path (dirs)
  (mapc (lambda (p) (add-to-list 'load-path p)) dirs))

then called this near the top of that .emacs file:

(add-to-load-path (list-subdirectories "~/.emacs.d/elpa"))

This let me continue as normal. The only omission from that emacs package list is slime, which I've lately been installing from sbcl or what-have-you with (ql:quickload :quicklisp-slime-helper) rather than through Emacs itself. It works exactly as well as you'd expect, which is to say flawlessly.

Dicking Around With Pis

Doing that got me into an installing mood, so I also formatted a fresh couple of SD cards with the latest versions of ARM Arch and Raspbian respectively. I did this with the vague intention of getting deal to work with one or both, and it looks like that'll take a bit more work than just a straight-up ql:quickload. Differences before I get to that though.

The RPi arch is much closer to what I'm used to on my laptop. A brutally minimal installation of the few core utilities you need to get basic shit done, and nothing else. Specifically, it gives you pacman, perl, a working ssh server and a minimally intrusive wireless connection mechanism that could replace wicd-curses for me. Raspbian, by contrast, bundles a mandatory window environment along with a bunch of crap that's probably nice for most humans looking to use it as a desktop replacement, but that I'll never end up touching. Also, they bundle Scratch as well as Python 2 and 3. Finally, while they do provide an ssh server, it's off by default, and the first time you boot a Raspbian image, it forces raspi-config, which means that you must connect a Raspbian Pi to a monitor and keyboard at least once.

That minimalism ends up biting Arch a bit though; it doesn't come with the standard raspi-config utility, which lets you mess around with the hardware to some small extent, and easily resize the installation partition to fill the SD card[5]. The other thing that bites ARM Arch in the ass, as far as I'm concerned, is the fact that its package manager has very few of the things I want to install. Out of my usual menagerie, I found screen, make, clisp, python, emacs and nothing else. By contrast, I had to apt-get --purge a bunch of things over on Raspbian, but I was eventually able to get it working with an almost copy of my laptop environment.

Almost, because Lisp still has some problems.

Specifically, clisp segfaults on both ARM Arch and Raspbian when you try to load anything with quicklisp, while the ARM ccl failed to run at all on Arch[6]. Raspbian did run ccl appropriately, but errored out on me for two reasons. Firstly, there's something unsupported about the :ironclad MD5 digest, and secondly, the ARM architecture seems to treat bivalent streams differently than x86. Which means that even running its custom house server, :deal errored out.

I'll be trying to fix that over the next little while.

cl-git-fs

Finally, on a merely semi-related note, I'm working on a couple of projects on my own time that are eventually going to want to do some sort of file management. And I figured it would be nice not to have to bring git into it manually after the fact. To that end, I took a look at how gitit manages the trick of using git as a faux-database for its wiki pages. It's not that complicated, as it turns out. And here's the result of spending an hour or two porting that piece of functionality to Common Lisp.

The biggest problem I'm running into is that there isn't a standard shell-command or run-program defined in the various Lisps I want to support.

I'm tossing it up to my github, but calling it 0.01 because there's a fuckton of functionality and documentation missing. In particular, it currently only supports SBCL on Linux, and a few of the external API functions still return raw string results, rather than properly parsed CLOS instances. The documentation and parsing will be a priority no matter what, but I'll only see how it runs on other platforms and implementations as I need to deploy to them.

---

Footnotes

1 - [back] - Of course, the drive box says 128G, so Samsung and all drive manufacturers are lying shitbags, but I'm digressing again.

2 - [back] - And therefore keep drawing power until a forced shutdown.

3 - [back] - which it was last time I played around with it.

4 - [back] - No, since you ask, I've never just straight up tried Mint. It has something in common with most of the distros I get recommended, which is that it cribs heavily from Debian on everything that matters, and then tries to differentiate on the desktop environment almost entirely. Not that this is bad for end users I guess, but as you can see from the x-window-system and xmonad items in that installation list above, I do not use what you would think of as "a desktop environment". Don't let that stop you from trying it, of course, but I'm not going to.

5 - [back] - You can still do this externally via gparted when you image your SD card.

6 - [back] - running the included binary gave me a "wrong architecture" error, even though there's no way that's accurate.

Combining SVG and HTML5 is still a Drag

Dragging SVG elements is harder than it might appear at first glance.

The underlying implementation of the SVG DOM and the HTML DOM is different in current browsers you see, so the standard HTML5 drag event doesn't apply to SVG nodes. Luckily, mousedown, mousemove and mouseup are supported, so you'd think it would be a straight-forward task to implement the fucker yourself. You probably imagine, as I did initially, something that takes a selector and a list of callbacks, and implements something similar to jQuery's .draggable() in ~30 lines of code by

• binding a callback to the targets' mousedown, mousemove, mouseup events
• storing the initial position of the element, and its delta from the mouse cursor
• preventing default on mousedown
• in addition to firing the callback, moving the target element by manipulating x and y coordinates using the current mouse position, initial delta and initial position

Maybe that's possible for the simple cases, but it's the edges that'll get you. And unless an implementation happens to dull all your edges, it's not really good enough.

Minor speed-bump. What if you need to drag multiple elements? If you were dealing with HTML elements, it wouldn't be such a big deal, but SVG elements have different properties to represent their coordinates. Rectangles and similar have an x and y that represents their top-left corner, circles and ellipses have cx and cy that represents their center, text elements have an x and y, but theirs represents the bottom-left coordinate, and I'm not even getting into the path elements. Bottom line, if you want to implement something that works, you're using transform settings. Also, you're not doing it naively through setAttribute, unless you're lucky enough to have a situation where you can guarantee that no other transformations will be applied to any draggable element[1]. The snippet that handles that particular piece looks like this in my codebase

(defpsmacro =translate (elem/s dx dy)
  `(try 
    (=>> ,elem transform base-val (get-item 0) (set-translate ,dx ,dy))
    (:catch (error)
      (=set-attribute ,elem :transform (+ "translate(" ,dx "," ,dy ")")))))

If you're one of the sad bastards who don't have macros at their disposal, I guess you're doing that or something fairly similar manually[2]. Like I said though, no big deal either way.

Medium sized speedbump. If you want to do this on an arbitrarily sized element, specifically a very small one, you'll discover that moving your cursor at even moderate speeds is enough to escape from the mousemove event and leave your draggable behind. One possible solution here is to also bind the mouseleave event and hope you never need to move fast enough to escape that too. Another approach is to have your chosen mousedown set up a global mousemove event, on body or html, use that to drag your element around, and have a global mouseup waiting to cut it off as soon as you're done[3]. A bit hacky, but doable.

Slightly larger speedbump. If you want to make these bindings switchable, you're in for a bit of a harder time. Not switchable as in "different objects should be able to do mildly different things", that's a given. I mean like "it should be possible to jump into a separate interaction mode where the same object does something mildly or wildly different under certain circumstances". If you want that, you need a level of indirection in your listener tree that you can swap out with other functions, and that level of indirection is going to be calling an externally specified function on each event trigger. Basically, you'll want to be working with hooks rather than listeners at this point[4]. I'll keep you posted on how this one goes in real life.

Large speedbump. Suppose you want to be able to use your dragging events, and a mouseup event on the same element. Better yet, suppose you wanted to implement drag/mousedown interactions, but let the user decide what layer to apply them on at any given time. Imagine a situation where you had the elements foo, overlapping bar, overlapping baz, and when a drag or mousedown hits, you want to let the user decide whether they want to be click/dragging foo and/or bar and/or baz. Near I can tell, there is no way of implementing this elegantly in terms of listeners on individual elements. What you need if you want this is a central listener that delegates particular events out to some intermediary functions, or eats them[5] as appropriate.

Keep in mind that the last two speedbumps I hit here probably won't be felt by most people going in the same direction. Still, I went into this figuring it'd take me a half hour at the outside to implement something workable. It ended up taking me the rest of the day, and will probably cost me another hour or two when I get back in on Monday.

Such is development sometimes, I suppose.

---

Footnotes

1 - [back] - If you are going to have other active transformations, using the setAttribute method would overwrite those, which is why it's a bad idea.

2 - [back] - if you are doing that, I should point out that the only reason I went the try/catch route here is that both =>> and =set-attribute take either an element or a set of elements as their first argument, and I wanted =translate to do the same. Since you probably won't have the same situation, you're likely better off with if/else.

3 - [back] - You wouldn't want to do this naively either, unless you knew there'd be no other mousemove events on that top-level element. If you did have that, you'd want to set up a hook that you could change out rather than messing with event listeners every time you dragged something.

4 - [back] - It just occurred to me that you might have no idea what I mean by "hook" in this context. Basically, something like this:

var mouseMoveHook = null;
document.querySelector("body").addEventListener("mousemove",
  function (event){
    //do other stuff
    if (mouseMoveHook) { mouseMoveHook(event) };
  });

If you have something that looks like that, you can change some of the behavior of your global mousemove event by assigning a new callback to the mouseMoveHook variable. I'm sure it's been used elsewhere, but I learned the term "hook" from Emacs, which provides standard event hooks in a bunch of different situations, and does it more or less this way, modulo some syntactic sugar and multiple hook support.

5 - [back] - In the case of the trailing mouseup event after a drag concludes.

Arbitrary Update 714

We're working on FBP in Common Lisp. You've heard that one before.

Step one was getting a basic system running. In fact, we've got two we're dealing with concurrently[1]. Both are kind of rickety because I'm still trying to work out what the essential parts of the approach are. One deals with OS threads and the other is built on top of cl-async. I'm liking the second one better for now, but am once again reserving judgment until I see how easily they explode.

Anyhow, that's price of admission, rather than the finish line. Our ultimate goal is to prove out visual programming in the large, without making anyone throw up about it. I have no idea how that's going to end up, except that I've definitely got a couple of interesting years coming up. In order to get to that ultimate goal, step two is finding or building an Acceptable™© Visual Editor.

And we can't find one, so "build" it is.

What we want, being that we're Lispers and polyglots to the last, can best be described as "Emacs for Diagrams". And that doesn't seem to exist. There's a number of general-purpose vector editors out there at various levels of readiness, but those are more like "MS Word for Diagrams". There's a "Sublime for Diagrams" floating around in the form of yEd, and one piece of fallout from the Russian space program is basically "Notepad++ for Diagrams". Microsofts' own flowcharting suite is something like "Eclipse: The Extra Shit Version for Diagrams. Also You Can't See Our Source Code, Which Is Probably For The Best Because It's Mostly Garbage You'd Laugh Out Of Your Company In Any Other Context". Finally, if we want to use Emacs for diagrams, the only thing even remotely workable seems to be artist-mode. It's nice, I guess. Certainly better than prostate cancer. But that's not quite what we're looking for.

The thing we are looking for needs to be

• A visual editor. This needs to be a tool to let humans construct diagrams and related diagrammatic artifacts for a variety of media.
• A diagram editor. We're not looking for a general purpose editor. In particular, there are very, very few things we'll be rotating, nothing we'll be coloring[2], and we only need one font at a fairy small variety of font sizes and weights. We'll also never be dealing with embedded bitmaps, gradients, brush-strokes, or shapes more complex than rounded rectangles/ellipses. You'd be surprised how much junk that lets you cut out.
• Fast. It shouldn't take longer to draw a diagram with this thing than it would take to describe it. This means a keyboard-oriented interface with easily re-bindable keys, reasonable performance, reasonable start-up, and no half-baked expert system getting in your way by trying to guess what you're trying to do.
• Scriptable. It should be possible to simply and easily construct sequences of operations to be summoned later. It should be easy to run automated queries and transformations on the programs' output and potentially edit the result in a visual manner. It should be fairly simple to select a particular subset of elements and manipulate them in some way.
• Flexible. We have no idea what the visual formalism we're finally going to adopt will look like. It might require fine placement of wires and connectors, it might involve different kinds of connections, and it will certainly involve multiple different connections between two nodes[3]. It also can't make any kind of semantic assumptions about what the content we're editing means, because that might change as we go through the discovery process of what kinds of processes and situations we want to formalize. This also means some sort of proper macro system, an end-user specifiable final/intermediate representation, and source code that we'll be able to tear apart if we need to.

The various editors I list above are split firmly into two camps. First, there are editors aimed at non-programmer humans for general vector construction. They're fairly flexible, but because they're for mouse-pushers, they don't provide a good intermediate representation for our purposes, don't particularly care about ease-of-use or keyboard-centricity, and don't make it particularly easy to script parts of your workflow. In the second camp, there are specialized flow editors aimed at programmers or technical managers, which do provide a good deal of flexibility on the keyboard binding front, and provide appropriate output, but also make all kinds of assumptions about what you're doing, why, and how you should want to go about it.

If you know of something awesome that I missed, let me know, but for the moment we're rolling our own. I've been working on it for about half a week solid now, and it's taking shape pretty quickly. I showed off the pre-alpha to some folks at the Erlang conference last week, and got reactions that didn't look like outright disgust. Once it takes some usable shape, I'm planning on posting some videos for you, and possibly showing it off a bit to the FBP group.

I was also going to talk a little bit about fact bases, and the inherent strength of simple models, but I think I'll leave that for another day. When I get permission to Free this editor, I'll have quite a bit more to say about that in any case.

---

Footnotes

1 - [back] - Har har, aren't I droll.

2 - [back] - Except for UI purposes, obviously. The various selection/manipulation layers are going to have distinct visual cues that rely on color, but I don't count these as part of the diagram proper, even though they may be visually associated.

3 - [back] - Which incidentally kills any use we might get out of svg-edit's or Inkscape's connection tools. Each of them assume a single connection between two shapes, anchored at each center-point. No idea if said connection can be directional or not, but we've already established that it won't do, so I'm not looking into it.

JS Frameworks

So I've spent the past week or so working through some examples with the various JavaScript MVC front-end frameworks. Let me save you the trouble: they're all shit. If you absolutely, positively can not live without a framework of some sort, use Backbone or Spine[1], because they seem to be as close to "minimal" as you can get, they won't get in your way too much, they help a little, and it's perfectly possible to run them alongside jQuery or similar if you feel like rolling certain pieces on your own.

While they do get in your way, the Javascript MVC movement is getting a couple of things profoundly right. Things I didn't really notice, or didn't think through all the way in the past, so I'm kind of shamefaced about having missed them, but they definitely seem like the right approach[2]. The reason I say

[The frameworks are] all shit -me

above is that none of them seem to be necessary to do the Right Thing©™[3], and none of them seem to help much with the big detriments of the approach.

Anecdote Time

Being that I haven't put a large project together with any of these techniques yet, this is the only example I'm willing to show, but it is illustrative. So I had a particular place where I needed to reorder pieces of information on the client side, then send them out to the server for persistence. Luckily, I was not the only one who had this problem and thought of using backbone to do it. Here's the solution recommended in that question:

Application = {};
Application.Collection = {};
Application.Model = {};
Application.View = {};

Application.Model.Item = Backbone.Model.extend();
Application.View.Item = Backbone.View.extend({
    tagName: 'li',
    className: 'item-view',
    events: {
        'drop' : 'drop'
    },
    drop: function(event, index) {
        this.$el.trigger('update-sort', [this.model, index]);
    },
    render: function() {
        $(this.el).html(this.model.get('name') + ' (' + this.model.get('id') + ')');
        return this;
    }
});
Application.Collection.Items = Backbone.Collection.extend({
    model: Application.Model.Item,
    comparator: function(model) {
        return model.get('ordinal');
    },
});
Application.View.Items = Backbone.View.extend({
    events: {
        'update-sort': 'updateSort'
    },
    render: function() {
        this.$el.children().remove();
        this.collection.each(this.appendModelView, this);
        return this;
    },
    appendModelView: function(model) {
        var el = new Application.View.Item({model: model}).render().el;
        this.$el.append(el);
    },
    updateSort: function(event, model, position) {
        this.collection.remove(model);
        
        this.collection.each(function (model, index) {
            var ordinal = index;
            if (index >= position)
                ordinal += 1;
            model.set('ordinal', ordinal);
        });

        model.set('ordinal', position);
        this.collection.add(model, {at: position});

        // to update ordinals on server:
        var ids = this.collection.pluck('id');
        $('#post-data').html('post ids to server: ' + ids.join(', '));

        this.render();
    }
});

var Instance = {};
Instance.collection = new Application.Collection.Items();
Instance.collection.add(new Application.Model.Item({id: 1, name: 'a', ordinal: 0}));
Instance.collection.add(new Application.Model.Item({id: 2, name: 'b', ordinal: 1}));
Instance.collection.add(new Application.Model.Item({id: 3, name: 'c', ordinal: 2}));

Instance.collectionView = new Application.View.Items({
    el: '#collection-view',
    collection: Instance.collection
});

Instance.collectionView.render();

$(document).ready(function() {
    $('#collection-view').sortable({
        stop: function(event, ui) {
            ui.item.trigger('drop', ui.item.index());
        }
    });
});

​

#collection-view {
   margin-bottom: 30px;
}

.item-view {
   border: 1px solid black;
   margin: 2px;
   padding: 10px;
   width: 30px;
}​

<ul id='collection-view'></ul>
<div id='post-data'></div>

And once you have all that in place, what you can do is drag the given elements around, and have returned a set of IDs in the order that they appear on the users screen! Isn't that amazing!? I gave it the benefit of the doubt, and tried to fit the code into my head for about half an hour before I realized something.

var util = {
    log : function (message) {
 $("#console").append(JSON.stringify(message)).append("<br />");    
    }
};

var templates = {
    rule : Handlebars.compile($("#tmp-list").html())
}

var rules = {
    render: function (rules) {
    $.each(rules, function (i, aRule) {
        $("#rules-list").append(templates.rule(aRule));
    })
    }
}

$(document).ready(function() {
    rules.render([{"id": 1, "name": "a"},
                  {"id": 2, "name": "b"},
                  {"id": 3, "name": "c"}]);
    $('#rules-list').sortable({
        stop: function(event, ui) {
        var ids = $("#rules-list li").map(function (i, elem) {
            return $(elem).find(".id").attr("title");
        }).get();
        util.log(ids);
        }
    });
});

#rules-list {
    margin-bottom: 30px;
}

#rules-list li{    
    border: 1px solid black;
    margin: 5px;
    padding: 7px;
    width: 50px;
}

<ul id="rules-list"></ul>
<script id="tmp-list" type="text/x-handlebars-template">
   <li><span class="id" title="{{id}}"></span>{{name}} -- {{id}}</li>
</script>
<div id="console"></div>

There. That's a solution weighing in at under half the SLOC, which gives precisely zero fucks about MVC frameworks and accomplishes the same task. Incidentally, I include underscore-min.js and backbone-min.js in that fiddle link because this was refactored from the above Java-style OOP soup, but I'm fairly certain that they're both unnecessary for this approach.

Note that I use, and wholly endorse Handlebars.js, or any of the similar standalone JS templating engines. I'll discuss why this is a good idea in a bit, when I clearly define the Right Thing©™, and what it implies.

The Right Thing...

...is separating out your front end into an entirely different application from your backend. Set them apart entirely, and have them communicate through JSON feeds and AJAX requests. It seems like an awkward thing to do principally because of how much harder it is to generate/template HTML inside of Javascript than it is outside, in server-side languages. Handlebars and similar libraries provide enough of a stopgap that the separation starts looking worth while.

Those of you who are already network programmers, or have dabbled with actors will intuitively understand why this is good. For the rest, let me try to explain what you gain and what you lose.

Bi-Directionally Agnostic Components

That's just a fancy way of saying that neither the front-end nor the back end really care what's on the other side of the channel, as long as it responds to the appropriate requests with well-formatted JSON[4]. That means that you could conceivably port your entire back-end without changing any client-side code if you really wanted to, or have certain requests get handled by specialized servers[5], or write multiple front-ends, or document your interfaces and let others write additional front-ends.

Hell, as long as it sent the right requests, and interpreted them correctly, there's no particular reason you couldn't ship a native desktop or native mobile front end that connected out to a production server this way. Decoupling project components to this extent also makes it much easier to make less radical and more controlled changes than the ones proposed above.

Simpler Components

Because each component can be made responsible for a particular concern, you get less code overlap. In retrospect, this has been a problem with most projects I've been on; if your server-side needs to be involved in templating, it's very tempting to "optimize" by having it emit ready-to-$.append() pieces. The problem is that these optimized pieces are harder to change later, and they sometimes require changes even when no other piece of server-side alters.

Doing the JSON communication thing completely tears this problem down. One end emits a series of expressions, the other consumes it. That means that your server has no involvement whatsoever in how the data is displayed to the user, and the client doesn't give a flying fuck about how it's stored on the back-end.

Security Concerns

One not-entirely-good part of the situation is that as soon as you decide to architect your application as a discrete set of network-communicating components, you have to solve one or two big problems that you could otherwise avoid. Specifically, you need to start dealing with throttling, authentication and network security right away, rather than leaving them for the point when you start scaling up. "Dealing with" doesn't necessarily mean "building", by the way, a legitimate choice is to make all of your handlers publicly accessible, but in that case you still need to make sure that no private information leaks out.

You also need to invest some thought into storage layout, since you won't necessarily be able to assume that the entire application is on the same machine.

Rich Client Side

This isn't necessarily a good thing. Yes there's a somewhat better interactive experience for the user, but they absolutely have to have JS enabled if they're frequenting your site. You can still bolt together a much simpler, pure HTML interface, but that will likely have to be an entirely separate piece if you want to keep any of the benefits of the decoupled approach. You still should take the approach, I think, but I wanted to note that there's a little bit more involved with supporting security-conscious users and older browsers.

HTML and Javascript

Barring a native front-end, you're stuck developing your client side in Javascript. The bad news is that it's Javascript. The good news is that a lot of people out there know it, and it means that your UI guys don't necessarily need to be up on their type theory or compiler concepts in order to be productive. And there's no reason for the back-end programmers not to use whatever language they find most productive[6].

You could Coffee Script or Parenscript or Clojurescript your way out of the worst of it, but any of those approaches necessarily couples you to some language or technology less common and less commonly known than Javascript. That particular tradeoff[7] is a conversation I plan to have with myself another day though, lets get back on topic.

What The Frameworks Do(n't)?

So back to They're All Shit. You'll notice that of the implications above, a framework can[8] conceivably mitigate one; the security concerns. The rest of them are either inherent advantages, or inherent disadvantages so fundamentally baked into the approach that no amount of JS-based syntactic sugar could make a difference.

The one extremely annoying thing each framework seems to do is try to layer additional object hierarchies on top of the DOM; an already existing object hierarchy that perfectly expresses the view end of an application. You're expected to maintain a view and a model tree in addition to that. I guess some people are already used to typing reams upon reams of code to perform basic tasks? And most of them work at Google? Ok, ok, maybe that's not entirely fair. It's certainly possible that the approach allows larger applications to be put together more easily, but I'm honestly not seeing it. Having taken in tutorials for Ember, Spine, Backbone, Angular, and Batman, the things they all have in common are:

• getting a simple task done is a lot more complicated with framework code than without it
• components built with frameworks are less composeable than components built without them
• everyone really, really, really wants you to declare a tree of classes before you do anything

It seems like all of those would add up to significantly increase complexity in a larger project, and I sort of had this goofy idea that complexity is a thing we're trying to reduce when we reach for library code.

In any case, I'm going to keep pushing straight up jQuery with Handlebars for the time being. With small, deliberate pinches of Underscore here and there. And I'll keep an eye out for pitfalls that might be pre-resolvable using some of the approaches I've seen this week. If something pops up and kicks my ass hard enough to change my mind, I'll write a follow-up, but until then I can't honestly recommend that you go with what seems to be the flow of web development on this one.

---

Footnotes

1 - [back] - Depending on how comfortable you are with Coffee Script, and how much you hate JS. Links to both in the sidebar.

2 - [back] - Again, I've only thrown the past week or so at this, so take that with a grain of salt.

3 - [back] - And, in fact, seem to make it much harder, more ponderous and more complicated to do the Right Thing©™.

4 - [back] - Or XML, or YAML, or whatever markup you end up actually using for communication. JSON seems like the right approach since it's extremely simple, and extremely easy to work with from within Javascript.

5 - [back] - For instance, you could have most of your app written in a powerful, expressive language without regard for performance, but have any real-time pieces handled by a server optimized for high concurrent throughput.

6 - [back] - As long as it supports easy serialization to and from whatever data format you guys have settled on.

7 - [back] -Number of practitioners vs. Quality of hiring pool.

8 - [back] - This doesn't mean they do, incidentally; from what I've seen, none of the current JS MVCs bother making it easier to do authentication, or secure requests.