Which Lisp Should I Learn?

I don't know why this keeps coming up lately, but it does. So, here we go:

My Recommendation

If you want to learn your first Lisp and already know something about the JVM[1], you should learn Clojure[2]. Otherwise, you should learn Scheme. Specifically, I recommend going the route that takes you through Racket[3], and possibly through SICP or HTDP.

If you absolutely, positively must, I guess go ahead and learn Common Lisp.

Why

Clojure the language, as opposed to the current, main, half-fused-with-JVM implementation, is cleaner and more consistent than Common Lisp, which should help you learn it more easily. I've gotten back talk about how there are lots more noobs learning Clojure, and as a result their libraries are in some disarray, and about the fact that the JVM is a sack of donkey balls you have to bite into every time you hit some sort of error[4], and about the general Clojure community pre-disposition to fashion trends. All of which may or may not be true, but I'm specifically talking about the language, not its ecosystem or stalwarts. Now granted, all of Racket, Clojure and Common Lisp

1. are built out of s-expressions
2. have defmacro

so depending on how much work you're willing to put in, you can do whatever the fuck you want in all of them[5]. However, in addition to knowing about prefix notation, and macros, and general Lisp program structure, here's an incomplete list of idiosyncrasies of Common Lisp that you have to commit to memory before you can be effective in it:

• Functions and variables are in different namespaces, and each has constructs that deal with them explicitly. You need to use defvar/defun and let/flet depending on whether you're using functions or variables. If you're passing symbol names around, symbols that denote variables can be sent around as 'foo whereas symbols that denote functions should be sent around as #'foo. If you're trying to pass a function around in the variable foo, you need to (funcall foo arg) or possibly (apply foo args), rather than just (foo arg).
• Most functions that deal with lists are functional, except the standard sort and the default mapcan, both of which destructively modify the list you pass them.
• You can define methods for your classes, but can't easily use certain default names that are bound to top-level functions. Such as length, or the arithmetic primitives. Which is why you frequently see methods like duration-add or matrix-mult.
• There are 7 commonly used equality operators, eq, eql, equal, equalp, string=, char= and =[6], each of which has mildly different, sometimes implementation-specific, behavior. Granted, because CL isn't a pure language, you need at least 2 of those, but 7 is still a bit much to have people memorize.
• There are three different local binding mechanisms that you must decide between depending on whether you want to be able to refer to earlier symbols in the same binding set, or whether you want symbols to be able to refer to themselves. You use let/flet if you don't care, let* for variables where you want later bindings to be able to refer to earlier ones, and labels for functions where you want bindings to be able to refer to themselves or earlier bindings.
• There are many, many implementations of Common Lisp. The popular ones at the moment are SBCL and CCL, but I've personally seen CLISP, ECL and Lispworks around too. More are available, and you might run into a lot of them in the wild. If you want to write portable code, you have to jump through some hoops. The implementation-specifics range from the finer points of equality operator behaviors, to the behavior of handler-case[7], to the types you can specialize on with defmethod, to the presence and behavior of threads, to the contexts in which you can pass streams around, to the names of various extension functions. For a small but representative example, take a look at what it takes to temporarily change your current directory in an implementation-independent way.
• Indexing into different constructs takes different forms. You need nth for lists, aref for arrays and vectors, gethash for hashes and slot-value for CLOS instances[8].
• You can't use the symbol t anywhere. No, not even local scopes. If you try, you'll get warnings or errors[9], because t is the top-level designated Truth symbol, even though anything other than NIL evaluates truthily in a boolean context.
• A hash isn't a list, and a CLOS instance isn't anything like either. One way you'd like them to be similar is when you're traversing them. It seems fairly reasonable to expect map-likes to operate on hashes by treating them as a sequence of key/value pairs, and instances by treating their bound slots as key/value pairs. This is not how things work. If you want to map over instances that way, you need to do something like this and this. If you want to map over hashes, you either use the hilariously mis-named maphash[10] or some idiosyncratic piece of the loop DSL that lets you iterate over hash keys and hash values.
• Common Lisp is case-insensitive. It takes whatever symbol input from you and upcases it internally. So foo-bar and foo-Bar both become FOO-BAR behind the scenes. This is usually not a huge problem, unless you try to interoperate cleanly with newer data standards like CSS or JSON. That leaves you fumbling with strings in situations where symbols and/or keywords really ought to do.

Like I said, this is a small sample. Just the stuff I thought of off the top of my head. I'm sure I could come up with more if I put a day or two into it. And I'm far from the most experienced Lisper out there, others would have more finer points for you, I'm sure. But that's half the problem with little issues like this; experienced Lispers completely forget about them. It's the newbs that have trouble cramming these things into their heads.

When I take a good look at that list, and then imagine the situations that led to each element, it's difficult to conclude that a wrong decision was made at any given point in time. Unfortunately, the sum of all of those potentially correct decisions is a giant system, the inherent rules of which look inconsistent if not outright hostile to human minds.

I don't know if Clojure solves all of them.

I've done very little work with it, for reasons entirely unrelated to the language. For all I know, when you get deep enough into it, you get to inconsistencies and/or restrictions which are worse than anything I've pointed out or could.

But do me a favor, if you're a CL user, either hop over to this web REPL, or install leiningen then hop into your local lein repl and type along here:

user=> (def thing [8 7 6 5 4 3 2 1])
#'user/thing
user=> (thing 0)
8
user=> (thing 3)
5
user=>(map (fn [n] (+ 3 n)) thing)
(11 10 9 8 7 6 5 4)
user=> (def thing {:a 1 :b 2 :c 3})
#'user/thing
user=> (thing :c)
3
user=> (thing :d 6)
6
user=> (thing :a 6)
1
user=> (map (fn [[k v]] (+ v 2)) thing)
(3 4 5)
user=> (def thing #{1 2 3 4 5}) ;; a set, in case you were wondering
#'user/thing
user=> (thing 3)
3
user=> (thing 0) 
nil
user=> (map (fn [a] (+ a 2)) thing)
(3 4 5 6 7)
user=> (def triple (fn [a] (* a 3)))
#'user/triple
user=> (triple 4)
12
user=> (map triple thing)
(3 6 9 12 15)
user=> (map (fn [a] (let [t (- (triple a) 5)] (* 2 t))) thing)
(-4 2 8 14 20)

Now think about how you would go about explaining to a novice programmer that it has to be more complicated than that.

---

Footnotes

1 - [back] - And don't have a strong dislike for it, obviously.

2 - [back] - Install it through Leiningen, which is available in the Debian repos in stable and unstable.

3 - [back] - Yes, I'm fully aware that the Racket guys are trying to push this "We're totally not Scheme" thing. They're close enough from an external perspective. Just don't tell Jay McCarthy I said so.

4 - [back] - Which is certainly true, but mildly preferable to the alternative as long as you're used to that sort of thing.

5 - [back] - Except that Clojure is apparently missing Reader macros, which I always thought were kind of half-assedly implemented in Common Lisp. For what I consider the full-ass version, take a look at how Haskell does it.

6 - [back] - Plus how many ever *-equal functions you define for your own classes.

7 - [back] - The Common Lisp answer to the problems that call for try/catch in other languages.

8 - [back] - For the last, you can also define your own selectors using :reader or :accessor declarations.

9 - [back] - Which specific warning or error depends on implementation.

10 - [back] - Because it's not very much like map. It returns nil and works by side-effect. Meaning that if you expect a sequence from it, you'll need to construct it yourself.

defsetf Examples

Have you ever defined a custom class with hash components? Something like, say, a session?

(defclass session ()
  ((started :reader started :initform (get-universal-time))
   (last-poked :accessor last-poked :initform (get-universal-time))
   (token :reader token :initarg :token)
   (session-values :reader session-values :initform (make-hash-table :test 'equal))))

If you have, you probably also figured it would be better to define some sugar for look-ups rather than doing (gethash foo (session-values bar)) every damn time. Something like

(defmethod lookup (key (session session))
  (gethash key (session-values session)))

And if you've gotten that far, you probably noticed that you can't just go ahead and say (setf (lookup :test foo) new-val), because if you tried, you'd get this

The function (COMMON-LISP:SETF COMMON-LISP-USER::LOOKUP) is undefined.
   [Condition of type UNDEFINED-FUNCTION]

Restarts:
 0: [RETRY] Retry SLIME REPL evaluation request.
 1: [*ABORT] Return to SLIME's top level.
 2: [ABORT] Abort thread (#<THREAD "repl-thread" RUNNING {10047D0113}>)

There are two ways of fixing this.

If You Don't Need CLOS Support

For instance, if your lookup is a function rather than a method, you can get away with doing something like this

(defsetf lookup (key session) (new-value)
  `(setf (gethash ,key (session-values ,session)) ,new-value))

or, annotated

(defsetf [name-of-lookup-function] (&rest [lookip-function-args]) (new-value)
  [macro-body])

Once you've defined that as appropriate, you can just

CL-USER> (lookup :test foo)
NIL
NIL
CL-USER> (setf (lookup :test foo) 'new-val)
NEW-VAL
CL-USER> (lookup :test foo)
NEW-VAL
T
CL-USER> 

This isn't satisfying if the lookup abstraction you've defined is a method though. Because then you get into this problem

CL-USER> (defmethod lookup (key (session session))
  (gethash key (session-values session)))

(defmethod lookup (key (hash hash-table))
  (gethash key hash))
STYLE-WARNING: Implicitly creating new generic function LOOKUP.
#<STANDARD-METHOD LOOKUP (T HASH-TABLE) {1005048E03}>
CL-USER> (defsetf lookup (key session) (new-value)
  `(setf (gethash ,key (session-values ,session)) ,new-value))
LOOKUP
CL-USER> (defparameter foo (make-instance 'session))
FOO
CL-USER> (defparameter bar (make-hash-table))
BAR
CL-USER> (lookup :test foo)
NIL
NIL
CL-USER> (lookup :test bar)
NIL
NIL
CL-USER> (setf (lookup :test foo) 'one)
ONE
CL-USER> (setf (lookup :test bar) 'two)

There is no applicable method for the generic function
  #<STANDARD-GENERIC-FUNCTION SESSION-VALUES (1)>
when called with arguments
  (#<HASH-TABLE :TEST EQL :COUNT 0 {10051DC633}>).
   [Condition of type SIMPLE-ERROR]

Restarts:
 0: [RETRY] Retry calling the generic function.
 1: [RETRY] Retry SLIME REPL evaluation request.
 2: [*ABORT] Return to SLIME's top level.
 3: [ABORT] Abort thread (#<THREAD "repl-thread" RUNNING {10047D0113}>)

Backtrace:
  0: ((SB-PCL::FAST-METHOD NO-APPLICABLE-METHOD (T)) #<unused argument> #<unused argument> #<STANDARD-GENERIC-FUNCTION SESSION-VALUES (1)> #<HASH-TABLE :TEST EQL :COUNT 0 {10051DC633}>)
  1: (SB-PCL::CALL-NO-APPLICABLE-METHOD #<STANDARD-GENERIC-FUNCTION SESSION-VALUES (1)> (#<HASH-TABLE :TEST EQL :COUNT 0 {10051DC633}>))
  2: (#:EVAL-THUNK)
  ...

If You Need CLOS Support

Then you'll need to define a setf generic. In the example we've been using, you could do the following:

(defgeneric (setf lookup) (new-value key session)
  (:documentation "Setter for lookup methods"))

(defmethod (setf lookup) (new-value key (session session))
  (setf (gethash key (session-values session)) new-value))

(defmethod (setf lookup) (new-value key (hash hash-table))
  (setf (gethash key hash) new-value))

Which would then let you polymorphically use setf exactly the way you'd expect.

CL-USER> (defparameter foo (make-instance 'session))
FOO
CL-USER> (defparameter bar (make-hash-table))
BAR
CL-USER> (list (lookup :test foo) (lookup :test bar))
(NIL NIL)
CL-USER> (setf (lookup :test foo) 'session (lookup :test bar) 'hash)
HASH
CL-USER> (list (lookup :test foo) (lookup :test bar))
(SESSION HASH)
CL-USER>

There. Hopefully the next person who searches for "defsetf examples" will find something more useful than I did.

Rebar Frustrations and LFE

I'm trying to get my head around Rebar at the moment, and failing pretty badly. It's honestly unclear whether I'm being an idiot here, but I've gone through all of four different tutorials on the subject, each purports to be a simple step-by step guide that'll get you up and running quickly, and each one hands you an error at around step four if you follow it as written.

The only piece of documentation I've been able to follow in its entirety without a crash dump is the basho Getting Started page. Except that it only gets me as far as starting and compiling a project, showcasing none of Rebar's dependency building, distribution creation or release handling. Which makes it marginally less useful than Make.

Following the upgrade_project option works fine on the dummy project provided as part of the rebar repo, but that project is shaped differently, and uses different config options than the ones rebar generates. Naturally, there is no documentation on what steps I have to take to get from what's generated to what works.

Ugh. Sorry, I had to vent for a bit there. Fuck this, I'm going back to Makefiles for the time being. I'll try rebar out again when my blood pressure lowers a bit.

Was that all?

Huh? Oh. No, I guess. I did also find this, which looks pretty badass. It's not exactly a Common Lisp or a Scheme. It's a purely functional, Lisp-2 with unhygenic macros that runs on top of the Erlang VM. It's about as officially supported as you can get, being made by Robert Virding, and it gets rid of quite a few things I don't like about Erlang.

Now, it's not a full Common Lisp, so don't expect quicklisp (sigh) or loop (though in theory, there doesn't seem to be a reason that you couldn't implement that one, if you wanted it badly enough), or CLOS, but it does hit the big ones. Namely full prefix notation, homoiconicity, and macros. And it retains the things about Erlang that I do find interesting, namely massive concurrency, the pure-functional approach, and standardized cross-process/cross-machine communication.

I'm going to keep playing with it, but I like it so far.

Indentation Ventilation

I need to vent, so just ignore this article.

Your formatting makes me want to puke.

For fuck's sake, when I ask questions about JavaScript or C, I don't bust out

lorem ipsum(dolor, sit) { 
    amet = consectetur(); 
    adipiscing (elit) 
        { Nulla_quis_purus = ac;
          arcu = consequat(consectetur(vitae(convallis,lacus)));}
        Aliquam { erat volutpat;}}

So as a point of basic courtesy, please learn to indent things in a way that doesn't make it look like you just loaded your parens into a shotgun before firing them at your editor. It's not as though it's difficult. Lisp is made of s-expressions, the only places where indentation gets ambiguous is in extended loop forms, and maybe one or two edge cases with lets. Just take the three minutes required to read the standard style, and stop pretending that you're showing us paren-using savages the proper way to waste vertical space.

Fuck!

I need some sleep.

Writing C in Lisp

I'm sick today, so I figured I'd write something so that I can at least retain the impression of productivity.

Lately, I've been working on a little pet project of mine which has to do with the codebase for Elite for Emacs (a port of Ian Bell and David Braben's original Elite game for my favorite editor). I liked the original, and while I didn't play it in 1984 (too busy being born that year), I did enjoy it quite a bit. The Emacs version is text driven, of course, but that's not all bad. I'll come back to why in a later post, but first I want to heap some scorn on this code. It's not that I hate it particularly. It works, for the most part, and it doesn't have to be fast (performance requirements are unsurprisingly low in a single-player game in Emacs). It's just that whenever I try to read any piece of elite-for-emacs, I come away with the impression that someone sat down and carefully thought of the worst possible way to do something, then implemented it. The result is an extremely verbose transliteration of a C program into Emacs Lisp. I'm going to dissect it here so that I can learn something, and so that hopefully, if there are hardcore C programmers out there learning Lisp, they won't pull things like this again.

This didn't start out as a re-write, just so you know.

I just wanted to fix a couple of bugs with weird control characters showing up in planet descriptions, change the behavior of a couple of commands for ease of use, and remove one or two components. It won't be a rewrite in the "throw it out and start over sense", but after peeking under the hood, it looks like I'll replace very close to all of the 4205 lines of code that make up this "port" of the original (either C or Assembly) codebase before I'm satisfied. There are some mistakes. Not in the sense that they don't produce working code, but in the sense that there are much simpler, easier and more accurate ways of doing the same thing in Elisp. Here are some before shots of common idioms I've found in no particular order:

(if foo
    (progn bar
           baz))

(defun foo (a)
  (let ()
    (progn
      (baz)
      (mumble))))

(if foo
    (progn (setq b bar))
    (progn (setq b baz)))

(defun foo ()
  (let ((a)
        (b)
        (c))
    (setq a bar)
    (setq b baz)
    (setq c mumble)
    (progn
      ...)))

(let ((a)
      (i 0)
      (b ()))
  (while (< i (length foo))
    (progn
      (setq a (car foo))
      (setq b (append b (list (mumble a))))
      (setq foo (cdr foo))
      (setq i (+1 i))))
  b))

If you want to see the complete, non-elided code, check out the project page. It's pretty much more of the same, with some odd byte/numeric indexing operations thrown in where they probably don't belong.

As far as I can tell, the snippets above should have been respectively

(when foo bar baz)

(defun foo (a)
  (baz)
  (mumble))

(setq b (if foo bar baz));; I'm leaving the setq in for demonstration purposes, but the actual task could have been done functionally

(defun foo ()
  (let ((a bar)
        (b baz)
        (c mumble))
      ...)) 

(mapcar #'mumble foo)

I'm not pointing this out to be mean. This code was written way back in 2003 (I'm using an earlier version that doesn't include combat/missions/GUI/windowed interface because my purposes demand simplicity, but the later code still makes use of the above idioms from what I saw). It's possible that some of this stuff was necessary at the time because of bugs in Emacs, or the peculiarities of Elisp. Not terribly likely, but possible. Anyway, here's why I don't like the above.

• The when macro exists. (when foo bar baz) does bar and baz if foo is true (unless is a similar construct that does exactly what you'd expect given when). Keep in mind that in Elisp, (), '() and nil are "false" and anything else is "true" for boolean purposes.
• The last value in a lisp block is returned implicitly. you can use this property to chain calls on objects instead of explicitly setfing an intermediate variable then copying. This applies to if too, which is why you can do (setq b (if foo bar baz)) instead of having to put setq in both branches of the conditional.
• You don't need to declare variables in lisp. If you want to establish local bindings (let ((a 1) (b 2) ...) [body]) is the way to do it. You can also use let* if you want the temporary variables to refer to each other, for example (let* ((a 1) (b (+ 3 a))) b) would return 4. You do need to keep the two straight in your head, because (let ((a 1) (b (+ 3 a))) b) would return an error (specifically, it would complain that the variable a is unbound). This is because let doesn't guarantee that its bindings will be done in the order they are presented. let* does this, but it's considered good style to use let where you can. If you need to define temporary functions, use flet.
• progn isn't necessary everywhere. Use it if you need to do multiple things in one branch of an if statement. Keep in mind that when, unless and cond have implicit progn for their blocks, so you don't need to type it out. New Lisp coders might think this is analogous to the missing curlies problem in C-like languages. I've been chewed out for doing things like

  if(foo) bar();
  else baz();

  in javascript code. The argument is always that if someone later adds mumble to the else block, but forgets to add curly braces, they'll get a fairly hard-to-find bug.

  if(foo) bar();
  else mumble();
       baz();
  

  In case you didn't catch it, that makes baz() unconditional, which is presumably not what you want. The correct way of doing it, I'm told, is

  if(foo){
      bar();
  } else {
      baz();
  }

  or

  if(foo)
  {
      bar();
  } 
  else 
  {
      baz();
  }

  depending on who's talking. In an imperative language with optional curlies/parentheses/brackets, this matters, so I'm not arguing that you should all stop using curly braces except where explicitly required. However, the fact that Lisp is fully parenthesized almost makes this a non-issue, a functional style mitigates it further, and in any case, adding progn all over the place isn't the right way to address it.
• It's very common in Lisp to want to iterate over a sequence, do something to each member of that sequence, and return the resulting list. The name given to this oddly specific idea is "mapping". The specific function you use is called different things (map, mapcar or similar) depending on which language you're in, and there are one or two subtleties (for instance, Elisp's mapcar only takes a unary function and a single sequence, Scheme's map can only take lists and not any sequence, etc.), but odds are that if you search the docs of a given functional language for "map", you'll find a function that does the above. When you're dealing with a sequence of things, it's a mistake to use while with an explicit, manual counter. I'd use mapcar or similar, and fall back to recursion (with tail calls where applicable) for more general purposes.

There are more things I could gripe about, and my sinister purpose in toying with this code is unrevealed, and I feel like I've only started off a much larger conversation about what it actually means to learn to program in a given language, but I think I need to go lie down now. This got enough stuff off my chest that I can continue to wade through the code for a while without bitching internally.

(defun my-thoughts (work time reflection) '(

I gotta tell you; when I started out with PLT Scheme, I wouldn't have thought I'd end up here. Here being an Emacs using CL-slinger.

It just sort of snuck up on me. It's really embarassing now, but the thing that tipped the scales most strongly in favor of PLT when I was starting out was the goddamn shiny IDE. It didn't enter into it how terse or how flexible the system was, all that mattered was that it be simple to use. This may be a primary driving force behind the general IDE craze that seems to be raging through development communities lately. It might just be my perspective, but it's feeling awful lonely over here in Emacs land. There are some popular adherents out there, but by and large, people I talk to these days hack primarily in MSVS or Eclipse. Hell, I went to the Toronto Lisp Users group last week, where you'd think there would be lots of support for Emacs, only to find that it was me by my lonesome. One guy hacked on Clojure in Eclipse, two of them used LispWorks and one didn't comment. The IDEs are winning in terms of number of users at least. Not sure whether it's a win on the productivity end, but hey.

So I guess it's not that embarassing.

Once I got hooked by the IDE, and the docs, and the package, it all just seemed so nice. Certainly better than my days hacking on Python/PHP (and even slightly better than my half-year of toying with Erlang). I never did figure out the profiler, but the macro stepper was really cool, and having errors highlighted with little arrows in DrScheme is so sweet that I actually started welcoming them for a while there. Slowly though, stuff changed. Without even realizing it, I was spending more and more time in Emacs. Whether in an editing mode, or the built-in GIT mode, or eshell, Emacs was starting to become my window manager. Before I noticed the change, I was using run-scheme rather than the IDE. I still had to pull it out for macro stepping, but it was tolerable.

Then I realized that I was really using three or four different libraries out of one category, and all of them were available for Common Lisp. Then I realized that CL also has a PDF-generating library. Aged and imperative, yes, but at least I wouldn't have had to roll my own that way. When the realization finally hit me that Emacs won in my mind, I sat down and thought about what really makes sense for me development-wise. Turns out that if you already know Emacs, and you already know a couple of lisps in addition to CL, SLIME is the best IDE you could hope for. Along with the built-in GIT support, swank, Lisp-Mode keybindings and Auto-Complete mode, SLIME's REPL/macroexpander/documentation-lookup/profiler push the environment over the edge.

So I hack on Common Lisp now, I guess. Man, I'd better get around to replacing that logo bar, it's getting pretty dated.)