Debian Follow-up, and StumpWM

Ok, so remember how I said I added some of the latest Ubuntu repos to my Debian sources.list just to hit one or two installs?

Don't do that.

For the love of God, don't. Or, at least, remove them from your sources file afterwards. I was happy enough with the stability of Debian this weekend, so I decided to run my install script (which includes things from audio-editing programs to inkscape, to emacs to several languages). Bad idea, to put it mildly. I had no idea what the hell I was doing at the time, so I just hit y ret a whole bunch of times, and in the process fucked up my sytem. I knew I was boned about the time I saw

removing gnome-network-manager
removing gnome-desktop
removing gnome-tray
removing x-server
[more bad stuff]

whizz by on the terminal window. When I next restarted, I got a friendly little prompt, and that's it. My data was still intact, but I didn't have a network connection for perhaps obvious reasons, so I mounted a usb-stick and got the shit I've been working on for the past day or so off, and prepared to reinstall (I don't know nearly enough about Linux innards to attempt surgery at this point).

I had a thought though; my current installation was Debian Lenny, and there was a "testing" version out called Squeeze ("testing" is in quotes, because by all accounts I've read, it's rock solid by this point). It took a bit of counter-intuitive navigation on the Debian site to get to the squeeze installer, but I guess that's reasonable; they want most people to install the "stable" release, not the "testing" or "unstable" ones. So there, I'm typing to you live from Debian Squeeze, and I have to tell you, it's good. The biggest gripes I had from the last post have been addressed; the new version of Gnome plays nice with two monitors out of the box, and the squeeze repos have more recent installs of the programs I use than Ubuntu. Specifically, I get out-of-the-box apt-get access to emacs23, sbcl 1.0.40, haskell-platform, pacpl and synfig.

So there. Debian beats Ubuntu from my perspective at this point.

At this point, since I was already ass-deep in installs anyway, putting in StumpWM seemed like a logical conclusion. So I did. And I'm in love. It's Emacs for window management. Just as a note, I've found that any software I could describe as "Emacs for [n]" is something I'd probably like. Sadly, between Emacs, Emacs for the web and Emacs for window management, I get the feeling we're about tapped out now. I like GIMP, but it's not exactly "Emacs for images". I've set up a minimal .stumpwmrc file like so

;; Psst, Emacs. This is a -*- lisp -*- file.
(in-package :stumpwm)
(message "Loading rc ...")

;;; Program definitions
(defcommand emacs () () (run-or-raise "emacs" '(:class "Emacs")))
(defcommand browser () () (run-or-raise "conkeror" '(:class "Conkeror")))

;;;Reloading
(defcommand reinit () () (run-commands "reload" "loadrc"))

;;;Keybindings
(define-key *root-map* (kbd "b") "browser")
(define-key *root-map* (kbd "C-q") "quit")
(define-key *root-map* (kbd "C-r") "reinit")

;;;Things that happen on StumpWM startup
(run-shell-command "/usr/bin/trayer --SetDockType false --transparent true --expand false")
(run-shell-command "nm-applet --sm-disable")

just to get everything up and running (trayer is needed to get the nm-applet working so I can has internets).

It's occurred to me that, now that my window manager is a Lisp machine, I could hard-wire "web-jumps" into my environment with Firefox. Not sure if it'd be worth giving up the keyboard shortcuts, but I would get HTML5 support, and all of my scripting would be done in Lisp at that point (rather than a JS/Lisp split). I'm really not up for that this weekend, but I'll keep playing with Stump. So far, it's good stuff.

Debian

I'm temporarily writing in Gedit (lacking the internet connection I'd need to apt-get intall emacs23 and bend it to my will. Already I find myself annoyed by the pretentious little toolbar, tab array and pretty icons at the top of the screen, and the useless little status bar at the bottom. On my widescreen monitor, I probably wouldn't have noticed, but this is a pretty severe infringement of a netbook's screen real-estate.

That's not what I wanted to talk about though.

I ended up installing Debian linux on three of my machines just to find out that it's about the same deal as Ubuntu these days, right down to the window manager. It installs fewer things by default, and you can't download a single-disc installer for it, and you can't REALLY install it without a network connection, but I still tried it. It even accepts apt-get commands, so I don't have to port my startup scripts. Though I did have to add a couple of repositories in order to get apt-get access to pacpl, emacs23 and a version of sbcl that doesn't crash like a champ when trying to install ironclad through quicklisp. This means that I just have the task of presenting you with diff Ubuntu Debian on three different computers, which is significantly easier than reviewing Debian.

The first thing that struck me is that Debian has a working amd64 implementation. Theoretically, Ubuntu does too, but here's the thing. I have two Intel machines (an old Pentium 2 and an Atom) and one AMD64 machine (a Phenom 2 x4). The intel startup disks I burned (for each 9.04, 9.10 and 10.04 Ubuntu) never have a problem. Not once. The install has gone flawlessly each time. The AMD machine is the precise opposite story. Not only did I end up making two copies of each startup disk (one at the standard speed, one at the slowest possible) just to make sure the CDs weren't at fault, but I also tried the same install from a bootable USB key. No dice. Ubuntu does not like AMD, apparently, because it took me no less than 17 attempts to get a single copy of Ubuntu 10.04 working on that computer. Once it worked, it wasn't quite smooth sailing either; it would crash out randomly, and I became convinced through hardware surgery that it was a software problem. My intel machines (including that 8 year old laptop I wrote about a little while back) have never had problems. So, one giant check for Debian.

The missing repos are a bit annoying. Debian by default has access to older versions of some programs I use frequently. This isn't a problem for the most part, until it is. The way I found out was trying to install ironclad through quicklisp, which kept complaining that it didn't have access to sb-byte-rotate (which is an SBCL component that shouldn't need to be installed separately). This was infuriating until I took a look at the output of sbcl --version, only to find out it was 1.0.14. I don't need to be bleeding edge, exactly, but that was a little ridiculous. Now this isn't a huge deal, because all it takes is adding a couple of lines to your .sources file and apt-get updateing, and you're good to go. About 10 minutes after finding out what the issue was, I was sitting on sbcl 1.0.29 (which includes sb-byte-rotate) and merrily quicklisping away. After I did that, I checked back through all the programs I installed and confirmed that Debian has significantly older versions of mplayer, Emacs and conkeror, and it lacks any access to pacpl (even an older version). These out-of-date programs were old enough that fairly significant shortcomings were apparent (some libraries didn't work on SBCL, Emacs lacked some modules I'd gotten used to, and Conkeror lacked some convenience features that you really miss once you're used to them). Again, just adding the ubuntu repos, then installing fixes the problem but it's still not zero-effort, so one small check for Ubuntu.

In the same vein of missing repos, almost to the point that it should be the same objection, Debian really doesn't want you using proprietary drivers. This is ok, I like the initiative of aggressively pushing for open software. Except that my 8 year old guinea pig has a Broadcom wireless card that I need the bcm43 drivers to use. And my netbook has some Intel branded card that I still haven't gotten working. Wireless is important, guys. We don't all have (or want to have) a web of CAT5 running the floor of our computer and living rooms. Ubuntu solves this by leaving the proprietary drivers out, but gives you a simple interface for turning them on if it detects you have some proprietary hardware in your machine. On Debian, you need to research the problem, enable the correct repos and install these drivers yourself. A check for Ubuntu.

Ubuntu has out-of-the-box support for multiple monitors. Debian does too, except that by default, it just mirrors your screen, which isn't what I want. I got it working after about twenty minutes of reading up on the issue. It required some xrandr trickery, and a one-line change to xorg.conf (which thankfully didn't error out, otherwise I would have stuck with one screen). I suppose I could have just upgraded Gnome, but I'm looking to switch over to stumpwm in the near future anyway, so I want a way of using multiple monitors with X11 that doesn't depend on a specific window manager. This is still a small check for Ubuntu (the functionality's there in both, but it's effortless in Ubuntu and flexible in Debian).

So there. The way it looks to break down is that Debian stays on my AMD machine at home, and I keep Ubuntu on the various Intel machines I own. It shouldn't be too big a pain in the ass, given their similarities, they're almost silently interoperable and I finally have a working home behemoth.

The next step is putting stumpwm and xmonad through their paces. I need a tiling window manager.

In other news, I've started studying up on Haskell (as evidenced by additions to the right sidebar). It started with a question on programmers.SE about Erlang code. It yielded no answers, but a german guy came by and suggested Haskell instead. I dismissed it as another instance of language biggotry, but after a conversation with him (and a healthy amount of research on the language), I've decided it's certainly worth learning. I'm not giving up on Erlang, mind you, Yaws is just too good an app to pass up, but Haskell has some of my mindshare now, and its package manager is full of useful stuff from PDF generators to web markup utilities to language interop facilities (including one for interfacing with Erlang processes).

It's going to be a long December.

Thoughts on Cooks [sic] Source

No programming discussion this time. Just some thoughts I had today.

So I caught wind of this during my daily Reddit trawl at lunch, and I didn't think much of it. Then I noticed Neil Gaiman and Wil Wheaton were chiming in via twitter. Then I saw a link to the magazine's FaceBook page which was in the process of defacement. I chuckled about it, then I went back to work. When I got home, it was still happening. The FB was up to 2200 "friends", the discussion group was filled with people now investigating Cooks Source articles from the archives under suspicion that Gaudio wasn't the first victim of this editor (she wasn't, by the way).

The flippant, asinine email (which I won't copy here; you can find it easily enough by now) probably fueled internet anger more than any kind of legal offense, but this didn't stop people from writing things like "thief", "plagiarist" and "copyright infringement" on the Cooks Source wall. Many of them also wrote "Twatwaffle", and various Chuck Norris/Monty Python references, but hey, this is the internet. Well, while online content certainly isn't 'public domain' as Ms Griggs claimed, the accusations don't seem fair.

First, she's not a thief. In the same way that someone who downloads a movie illegally isn't a thief. That is an offense, but stealing ain't the right one because the victim isn't actually being deprived of anything when someone makes a copy (many argue that they're deprived of potential profits, but even if that held water legally, it's a moot point in this case because Gaudio's blog is free).

Second, she's not a plagiarist. She might be an idiot with a tin ear for period-piece writing, a poor editor, a worse speller, and a self-righteous, uninformed ass, but she did credit Gaudio for the article (whereas plagiarism would have entailed putting someone else's name on the byline).

The third accusation of "copyright infringement", is prickly too. It's not that she didn't do it, according to the letter of current law, it's pretty clear she did. And to be fair, this is pretty much the one place where I feel copyright is approaching a good thing (someone reproducing others' works without permission or compensation and for the express purpose of making money by it). I'm still torn between the implications here, because the outrage tells me that even while ACTA is on many minds, and most folks I talk to complain about how restrictive current copyright is, a very vocal part of the internet still considers it a good thing. The really worrying part for me is that a CC-attribution-share-alike license (which all of my online work has been released under) would expressly allow what happened here.

I hate to be the killjoy, or to call down the internet thunder, but I wouldn't have a problem with that part of what she did. Endorsing an open culture doesn't just work when you argue against DRM on music and games, or when someone tries to shut down Pirate Bay, or when people hell-bent on censorship cry "DMCA!" without cause. It means that everyone gets access to your stuff too.

By all means, continue trolling. Assholes deserve to be slapped around just for being assholes, doubly so when they're also grossly incompetent and quick to resort to the "N years experience" argument. But I'm going to go on record as saying that I think reproducing work you find online isn't a bad thing. If it is, it shouldn't be a bad thing. The entire advantage that the internet has over traditional media is that digital copying lets information fly around the globe at almost the speed of light. If you invoke copyright, you slow that information down by artificial means. It's at once inefficient and depressing that people would suffer the lack of light-speed just to keep their ideas under control.

Plagiarism sucks, self-righteous assholes suck, but copying shouldn't.

OS Experiments and Project Update

As much as I like sitting at post number 42, it's time for an update.

I've been testing another distro of linux (well, not another distro, really. An Ubuntu derivative called Crunchbang), and I really thought it was going to unseat Ubuntu as my OS of choice. The main killer, it turns out, was xorg.conf.

First, the good stuff. It's simple, it has better keyboard support for launching programs than Ubuntu, and its performance is through the roof. It actually flies on the toy laptop I have lying around for this kind of experimentation (a Compaq Presario R3000 I picked up for $20 about two years ago. It has a 1.4ghz processor, a whopping 256 megs of ram and a 5400rpm, 30gb hard drive). Even with those specs, Crunchbang is usable. So I was all psyched up to install it on my netbook to get a bit more performance out of it, just because I do actually do some development on it when I'm out and about. Two things are making it unacceptable though.

One, and this is the main one, xorg.conf. To the Linux veterans this is probably a joke, but it's pretty difficult for the newbs. There was sparse information on configuring monitors this way back when it was mandatory, and you could never be sure that you were pasting the appropriate things because, apparently, different distros had different conventions about it. When I searched (and I mean everywhere; old linux sites, the various StackExchange sites, the Ubuntu docs and the Ubuntu forums), all I found were horribly outdated pointers, and one or two comments about how "xorg.conf isn't really used anymore, just use xrandr". Which is great, except that xrandr doesn't seem to auto-detect additional monitors in Crunchbang, so I still need to know how to configure them the old way. The other common instruction I found was to copy-paste existing parts of your current conf file. Which I'm sure was great advice at one point, but current Ubuntu/cruchbang xorgs look something like

Section "Device"
        Identifier      "Configured Video Device"
EndSection

Section "Monitor"
        Identifier      "Configured Monitor"
EndSection

Section "Screen"
        Identifier      "Default Screen"
        Monitor         "Configured Monitor"
        Device          "Configured Video Device"
EndSection

which doesn't seem to contain useful information. All it tells me is that all these options are now configured elsewhere, and I have no idea where that is. The other big problem with xorg.conf editing is that you have to restart the X server each time you want to test it out, and (at least on Crunchbang and Ubuntu) the error messages you get aren't exactly useful. They basically say "there was an error in your xorg.conf".

Thank you.

Which section and line would be useful.

Even if you drill down further into the error log, it typically (again, for me at least) just said that the error was a missing EndSection on the last line, even though there were no missing EndSections, and the last line consisted of nothing else. It took me something like two hours to track down enough info about the basics to get the screen mirrored on an external monitor (before that, the extra screen was just displaying various seizure-encouraging light patterns). Xrandr still doesn't see the extra monitor by the way, so I still have to muck about further with xorg.conf.

Second, and this is the tiny problem almost not worth mentioning, but it's not an issue in plain Ubuntu. Crunchbang doesn't have apt-get access to emacs23. If I want to apt-get install emacs, it gives me version 22.1 or so. I could spend a bit of time figuring out how to script a wget to the GNU ftp site, then run the appropriate commands to untar and install it properly (it's probably just make and make install, though I've never had to do it so I don't know), but my fucking around time-budget is officially used up for this week.

So there. That's why I'm keeping Ubuntu 10.x. For what it's worth, until I hit the iceberg that is xorg.conf, this was going to be a blog post about why I switched. Seriously, I had notes ready and everything. So if you don't need more than one monitor for whatever reason, or you're a xorg wizard and still on Ubuntu, give it a try.

Other than my OS experimentation, I've also been toying with the formlet system I shamelessly lifted out of PLT Racket (still at the GitHub page, but now there's documentation), and I also released that CSS module I wrote about a little while back. I honestly wasn't going to, but those 12 or so lines of code found their way into every single project I've worked on since. Code reuse is good, I hear. The project page is here, and it is likewise accompanied by a nice little Markdown-enabled documentation sheet.

The big additions since you last saw these projects are

Formlets now support recaptcha.

You like how I'm adding high level support like this before the thing can even generate checkboxes, huh? Well, it might be a bit dumb, but I feel no shame in admitting that the order of operations is self-centered. Which is to say, I add features as I need them, not as they make sense in theory. It just so happens that I had call for text inputs, textareas, passwords and recaptcha before one of my projects called for checkboxes, radio buttons or select boxes. I'm sure it'll change soon.

CSS compilation

This one's obviously from the css generator. Basically, a task called for a several-hundred-line stylesheet. While it wasn't performing too badly, I realized that eventually, there would be a call for CSS that you could cache (as opposed to the inline styles then generated by cl-css). So, I added compile-css, which takes a file path and some cl-css-able s-expressions and outputs them as flat CSS. Tadaah! If you're running Hunchentoot (or any other server really, cl-css is simple enough that I can confidently call it portable) from behind Apache or nginx, you can now have the front-facing server serve out a flat-file stylesheet and still get the benefit of declaring your CSS more succinctly in Lisp.

Documentation

Like I said earlier, there's some simple docs included with both projects which you can thank GitHub for. The main reason I wrote them was seriously because there's this little notice that tries to guilt you into putting in a README if you don't have one already. It supports markdown, textile, rdoc, and a bunch of other psuedo-markup languages meant for displaying plaintext while still supporting automatic conversion to HTML. Mine are written in markdown, mainly because I'm already familiar with it, and I managed to find a decent Emacs editing mode for it.

ASDF-ability

I'm probably late to this party, given that quicklisp is now in public beta stages, but I set up ASDF packages for both formlets and cl-css, and hooked them up to the CLiki. So unless it (the CLiki, I mean) goes down again, you can install them both by doing

(require 'asdf)
(require 'asdf-install)
(asdf-install:install 'cl-css)
(asdf-install:install 'formlets)

EDIT: Ok, so ASDF-ability doesn't seem to be playing nice at the moment. I swear I tested the things out by installing them that way myself. I'll see what's up later today. EDIT2: Still not sure what's going on here. It looks like ASDF doesn't like https? EDIT3: Ok, so ASDF seems to assume that all well-formed web addresses start with "http://", which means that google docs, github and other hosts that use https:// are a no go. The current solution I'm resorting to is just chucking cl-css and formlets tarballs up on my linode. CLiki has been updated, so the ASDF packages finally work. Now I neet to talk to Zach to get this stuff added to quicklisp...

cl-inception

That's just about what I spent the last four hours feeling like. I've been working semi-feverishly on a version of the formlet macros for Common Lisp. Automatic validation is surprisingly difficult when you break it down. If you want to do it well, I mean. The thing could easily have been half-assed in half the time, and three-quarters the code. But I meant to do it well, so it took a while. It's still not anywhere near perfect. If you tried to get it to generate a form called "form-values", I imagine it would snarl at you like some lovecraftian horror. I'll be plugging the holes over the next little while with judicious use of with-gensyms, but that'll only add one layer to a construct already six dreams deep.

It's fucking bizarre, I tell you. On days like this, I can sort of see why people stay away from LISPS in general. Not all minds can be made to twist in on themselves indefinitely; mine was barely capable of six levels, like I said. By which I mean, the formlet system I wrote is made up of a function and a macro (show-[name]-form and def-form). show-[name]-form is a function that calls the show-form macro to generate partial HTML and invoke the form-element macro, which expands into the actual low-level HTML boilerplate. def-form is a macro that expands into the appropriate show-[name]-form function, and defines a validator function, itself composed of no less than four nested macros.

It sounds like a complete goddamn birdsnest, and it kind of is, but every layer of complexity is warranted, as far as I know (if it isn't, please, PLEASE tell me that, and point me to simpler code that does what this does).

Here's the thing. If you're just looking at how to generate the HTML on forms (a-la the PLT Formlet system), it's ridiculously easy. I could have gotten away with a macro and a half instead of the layer cake that I ended up with, but the issue there is that that way wouldn't have saved me much time or code. The hard part on a form is not displaying it. That is the easy part. If I may say so, the trivial part. If your system does nothing else, then it is balanced on a precarious edge between the twin pits of "break-even" and "not worth using". The difficult part is validation. At the high level, it sounds simple (which is probably why I was foolhearty enough to attempt it);

1. Take in the form results and a list of predicates.
2. Run the predicates over the results.
3. If they all passed, do something, otherwise, send them back with a little note saying what they need to fix.

Tadaah!

But those three steps (if you wanna do them properly) have so many moving parts that it necessitates many, many macros. For starters, there are fundamentally two types of forms; the type where you need to validate each field (like a registration form, or other long list of inputs), and the type where you need to check the whole form in aggregate (like a login form, where you really only care whether you were just handed a valid name/password pair, and in fact you don't want to tell the user which of the two they got wrong).

That second group runs out of situations fairly quickly; you just need to display one error message for the form and send the user back, or let them through if they got it right. Done.

The first group is what caused most of the work. First off, each input needs its own predicate, and its own failure message. If you want to provide good signage, it's also not sufficient to and the list of predicates over the inputs; you want to iterate through the full list no matter how many mistakes you find, and then mark them all off in the error listing. When you get back, you need to display each error next to the appropriate input, and (for non-passwords), you want to keep any inputs the user sent you that validated ok. As if that weren't enough, those pesky users like labels too, so that they can see what they're filling out, and you need to support (at minimum) input type=text, input type=password, textarea, select and checkbox if you want to be useful (cl-formlets doesn't, by the by, this 0.0001 release only supports inputs, passwords and textareas because I don't need the others for the app I'm currently writing. Stay tuned though). radio can be nice, but you can get by with just select. The end result is that you need to track

1. field names
2. field types
3. user input for each field
4. a predicate for each field
5. an error message for each field

and good god, do the interactions make me want to headdesk. #2 specifically sounds easy, but gets mean fast when you think about it. I won't go into the details, but keep in mind that it's not enough to just keep track of a type property and switch it out; a select has a fundamentally different tree structure underneath than a textarea, which is again fundamentally different from an input. When I say different, I mean that they track user input in different ways, need different things changed out when they error, have different consequences when setting their value (and different methods of setting it, too) and behave differently on screen.

I'm not posting the code here, it'll be at github. If you can do better, please, please do so, and let me know.

In the meantime, I honestly feel like I should be spinning a top and watching it warily.

(defun my-thoughts (work time reflection) '(

I gotta tell you; when I started out with PLT Scheme, I wouldn't have thought I'd end up here. Here being an Emacs using CL-slinger.

It just sort of snuck up on me. It's really embarassing now, but the thing that tipped the scales most strongly in favor of PLT when I was starting out was the goddamn shiny IDE. It didn't enter into it how terse or how flexible the system was, all that mattered was that it be simple to use. This may be a primary driving force behind the general IDE craze that seems to be raging through development communities lately. It might just be my perspective, but it's feeling awful lonely over here in Emacs land. There are some popular adherents out there, but by and large, people I talk to these days hack primarily in MSVS or Eclipse. Hell, I went to the Toronto Lisp Users group last week, where you'd think there would be lots of support for Emacs, only to find that it was me by my lonesome. One guy hacked on Clojure in Eclipse, two of them used LispWorks and one didn't comment. The IDEs are winning in terms of number of users at least. Not sure whether it's a win on the productivity end, but hey.

So I guess it's not that embarassing.

Once I got hooked by the IDE, and the docs, and the package, it all just seemed so nice. Certainly better than my days hacking on Python/PHP (and even slightly better than my half-year of toying with Erlang). I never did figure out the profiler, but the macro stepper was really cool, and having errors highlighted with little arrows in DrScheme is so sweet that I actually started welcoming them for a while there. Slowly though, stuff changed. Without even realizing it, I was spending more and more time in Emacs. Whether in an editing mode, or the built-in GIT mode, or eshell, Emacs was starting to become my window manager. Before I noticed the change, I was using run-scheme rather than the IDE. I still had to pull it out for macro stepping, but it was tolerable.

Then I realized that I was really using three or four different libraries out of one category, and all of them were available for Common Lisp. Then I realized that CL also has a PDF-generating library. Aged and imperative, yes, but at least I wouldn't have had to roll my own that way. When the realization finally hit me that Emacs won in my mind, I sat down and thought about what really makes sense for me development-wise. Turns out that if you already know Emacs, and you already know a couple of lisps in addition to CL, SLIME is the best IDE you could hope for. Along with the built-in GIT support, swank, Lisp-Mode keybindings and Auto-Complete mode, SLIME's REPL/macroexpander/documentation-lookup/profiler push the environment over the edge.

So I hack on Common Lisp now, I guess. Man, I'd better get around to replacing that logo bar, it's getting pretty dated.)

Lisp and CSS

So the Reddit/Y Combinator spike seems to have died down, which means I can return to blissful obscurity. Not that arguing with Jay freaking McCarthy of PLT Racket and getting to thank Xah Lee for his Emacs tutorials wasn't the high-point of my day yesterday, but I sort of write these posts in order to get stuff out of my head rather than to have them read.

I got to the point of needing some CSS in a lisp app a little while ago, and while I was typing it up, I thought "Hang on, self, I'm sure there's a way to get this done in lisp instead of repeating yourself this much in CSS". Checking online, sure enough there's a library for it (css-lite, which is available through asdf).

The asdf version seems to have some bugs in it, sadly.

* (asdf-install 'css-lite)

[snip installation trace...]

* (require 'css-lite)

NIL

* (css-lite:css (("body") (:height "50px" :width "100px")))

"
body {
height, '50px', width, '100px':nil;
}"

That's not exactly what I meant.

I'm sure the git-hub version has this stuff resolved, but by this point I was already on the "How hard could this possibly be?" train of thought.

Inputs and outputs are strings by the css-lite convention, so it seems like it should be pretty simple to output. Well, it is.

(defun css (directives)
  (apply #'concatenate 
         (cons 'string 
               (mapcar #'(lambda (dir) (format nil "~(~a { ~{~a: ~a; ~}}~)~%" (car dir) (cdr dir)))
                       directives))))

* (defvar test `((body :margin 5px :padding 0px :font-family sans-serif :font-size medium :text-align center)
             (\#page-box :width 1100px)
             (".box-one, .box-two" :width 200px :float left :overflow hidden :margin "0px 5px 5px 0px" :padding 0px)))

TEST

* (css test)

"body { margin: 5px; padding: 0px; font-family: sans-serif; font-size: medium; text-align: center; }
#page-box { width: 1100px; }
.box-one, .box-two { width: 200px; float: left; overflow: hidden; margin: 0px 5px 5px 0px; padding: 0px; }
"

Tadaah!

It could be more efficient if I used reduce instead of having mapcar and concatenate as separate steps.

(defun css (directives)
  (flet ((format-directive (d) (format nil "~(~a { ~{~a: ~a; ~}}~)~%" (car d) (cdr d))))
    (reduce (lambda (a b)
              (let ((final-a (if (listp a) (format-directive a) a)))
                (concatenate 'string final-a  (format-directive b))))
            directives)))

* STYLE-WARNING: redefining CSS in DEFUN

CSS

* (css test)

"body { margin: 5px; padding: 0px; font-family: sans-serif; font-size: medium; text-align: center; }
#page-box { width: 1100px; }
.box-one, .box-two { width: 200px; float: left; overflow: hidden; margin: 0px 5px 5px 0px; padding: 0px; }
"

* (defvar box '(:margin "32px 10px 10px 5px" :padding 10px))

BOX

* (css `((body ,@box :font-family sans-serif :font-size medium :text-align center)))

"body { margin: 32px 10px 10px 5px; padding: 10px; font-family: sans-serif; font-size: medium; text-align: center; }
"

That should do it. So yeah, there's a quick and dirty non-validating CSS generator. It took about 10 minutes to write (and most of that was trying to figure out why it wasn't working, then realizing that I'm no longer using Scheme and that foldl therefore doesn't exist), which is probably less time than it would take to go online, download a fix for css-lite, install it and try it again. I would submit it to git-hub or something, but 6 lines of code seems somehow unworthy of its own module.

I feel this also validates my statements about the format function in the last post. In scheme, this css transformer would have to resort to another couple of function calls. It's that short in part because I was able to take advantage of CL's embedded formatting DSL.

Yegge Strikes Back from the Grave

So I've been fooling around with some new stuff.

Actually, before I tell you about that, quick update. dieCast is now in the early beta stages. It's actually capable of supporting games, but it's got a long way to go before it's something I'll be proud of. We're about three months away from a public beta from where I'm sitting. For the testing stage, I'm ending up using some creative-commons enabled sprites. I'll probably keep them as a subset of the final sprite lineup, if the license permits, but the intention is to get original artwork up.

Ok, now then.

I've been fooling around with some new stuff.

Or rather, some very very old stuff. Over the last couple of days, I've decided to pick up Common Lisp and Portable Allegro Serve again. I gave up on trying to install PAS on SBCL after about twenty minutes though, and promptly switched out to Hunchentoot, and all I really have to say is

Holy.

Shit.

I already have some projects underway with PLT Racket (including Diecast), but goddamit, I think I made the wrong decision. The quote from Yegge goes something like "Most newcomers independently come to the same conclusion; Scheme is the better language, but Common Lisp is the right choice for production work." Bottom line, I remember disagreeing a long time ago, but I've uh... independently come to the same conclusion.

PLT Racket seems to be as good as Scheme gets. It has built-in support for everything from hashes and regexps to x-path and http. It has file-system bindings, guaranteed tail-recursion and pretty much the best package system I've seen (from the downloaders' perspective, at least, Scribble is a bit of a bitch to get familiar with if you plan to actually document your own code).

So why am I having serious second thoughts?

Lots and lots of little things. Now that I've actually had some time to play with both contenders, mastered both IDEs, played with both macro debuggers, ran web servers on both and lived in each language for a decent length of time, I think I can finally compare them, and gain some sliver of insight from the comparison. And it's a damn close race. The biggest differences turn out not to be what everyone was pointing at. I have a link in the sidebar over there pointing to "Scheme vs. Common Lisp", which purports to tell you the differences between the two, and maybe three of those actually trip you up to any significant degree.

So here's the big stuff. PLT Racket vs Common Lisp from a young hacker's perspective.

1. Documentation

The PLT Docs are badass, and centralized. Second to none. They have Ajax search running over all functions in their implementation (and you need it with the amount of stuff it has), code examples all over the place, and a comprehensive set of tutorials perfect for beginners. Common Lisp probably has more overall information on it, but it's scattered across CLiki, Common Lisp Directory, Hyperspec, various indie package pages and Bill Clementson's archives. M-x slime-documentation-lookup helps, but it only searches the Hyperspec. That's plenty of info for the veteran, but (if I could imagine my point of view about two years ago) it wouldn't be sufficient for someone who's, say, looking for a complete listing of format-string options (As a public service, the way you find that is to look up format, then scroll about half-way down the page where you will be pointed to section 22.3 for more information on formatted output).

2. Package Repositories/Installation tools

EDIT:

I can't believe I managed to go so long without adding this note. As of the end of 2010, quicklisp also exists, and is awesome. That means that the gripes I had about asdfing things are moot, since you don't need to for the most part. Thanks to ql:system-apropos, it's also fairly easy to find CL packages, so I guess PLT Scheme (now Racket) no longer wins this one. I have no idea what they've been up to for the last year though, so they probably made a thousand and one improvements all over the place too.

Thu, 11 Oct, 2012

Common Lisp has asdf, which is awesome compared to the tools found in most other languages I've used, but PLT beats it pretty handily. It's basically the same story as documentation. There's technically more stuff out there for CL, but it's scattered, and since development is distributed, you'll get some duplication of effort. There are four or five different HTTP servers, for example, and at least three HTML-templating libraries. Granted, there's a clear "best" in each category, but you really need to do your reading in order to find that out. PLT has a smaller offering (the biggest gaping holes are in the document generation area; there is no such thing as a good PLT Racket PDF/PostScript generator), but it's neatly organized, indexed, and accessed by typing (require (planet [package-name])) in the declaration section of whichever file you need the new package for. No hunting, no missing GPG keys. These first two points are probably the ones I'll miss most from the PLT offering.

3. The Web Server

This is actually a place where more choice would do PLT Racket some good. They do have a pretty cool web server, but it's far from fast in practice. It also seems to crash more often than I'd like for a production app. Nothing like once per week, but it's happened a few times so far. The trouble is how it behaves. It's basically Tomcat, minus the copious installation headaches; you need to get all your code in order, make sure it'll run, then execute. And that's it. If you need to make changes (like, while developing web apps) you need to tweak the code, then restart the server, then re-navigate to the page you were just on because it auto-generates new urls each time. This is an exercise in frustration, and is one reason that I've still kept up on my PHP and Python skills this entire time. The languages may be slightly worse, but they're interpreted, so a change doesn't need to bring down the whole server. That's how Hunchentoot works. You load your files, then start the server. If you need to make a change, you evaluate the new code against the actual, still-running server. I wouldn't use this in the wild, but during the development stage, it is hot, buttered, bacon-wrapped power. That alone seems to be enough to pull CL into the lead as far as my use of it is concerned.

Now in, PLT's defense, they're aware of this. There was a concern about keeping LISP's inherently reflective nature, and they decided not to because it trips up so many people that they figured it wasn't worth the headaches. So the server forces you out while it runs, and the REPL bugs you to do a clean run every once in a while if your source has changed. I appreciate the sentiment, because it really was made to be a teaching tool, but I'm being mighty tempted by the dark side regardless. There's also a concerted effort from PLT to keep things byte-oriented. For example, there is no supported way to get a list of POST/GET parameters out of a request (other than "manually") in PLT scheme. "Manually" entails getting a list of binding objects out of the request and mapping over them to get a list of byte-strings out. There's also a few other little gotchas (like how awkward it is to actually create a link whose result is another scheme function, and how url-based dispatch is for whatever reason NOT the default).

4. The Format function

This may sound like a nitpick, but I'm not into the nitpicks yet. This is actually a difference. In PLT Racket, you're stuck with (format "~a" blah). It only accepts formatting directives, rather than CL's richer set of formatting, flow control and kitchen sink. It also always returns its result, and doesn't have the option of printing to standard-out (you have to use printf for that). I didn't think this would make as big a difference as it did, actually, because I've gotten used to the simpler Scheme format, but hot damn is it awesome to be able to do something like (format nil "~a ~{ ~a: ~@[ ~a ~]~}" (car blah) (cdr blah)) instead of resorting to several function calls for the same effect.

5. Plists

Basically same story as above. I forgot how useful these actually were for day-to-day purposes. I mean, I still bust out hashes for bigger stuff, but little tasks all over the place are made just a tiny bit easier with the use of p-lists instead of a-lists.

6. Function names.

Ok, now we're into picking nits. It's not a huge deal, but the scheme conventions are cleaner and more consistent. If you're dealing with a predicate, it ends with "?", if you're dealing with a side-effect function, it ends with "!". Common lisp has a grab-bag. Some predicates end with "p" (as in listp), but most are just the unmodified word (as in member). Also under this category is the lisp-1 vs lisp-2 thing; because there's separate namespaces for functions and variables in CL, there's two let types (let for variables and flet for functions) and two definition types (defun and defvar). Because functions get treated differently from other variables, some things are a bit trickier in CL; for example, while you can still do (apply (lambda () 42) '()) or (mapcar (lambda (num) (* 2 num)) '(1 2 3 4 5)), you actually can't do something like (setf foo (lambda () 42)) followed by (foo) (you would either need to call foo with (funcall foo) or define it as (setf (symbol-function 'foo) (lambda () 42))). The Scheme equivalent is (define foo (lambda () 42)), after which (foo) does exactly what you think it will.

7. Macros

PLT Racket has define-syntax-rule and define-syntax, as well as library support for define-macro, which is a copy of CL's non-hygenic defmacro. In practice, I found myself using define-macro most of the time, so it shouldn't be too big a problem to switch here. Admittedly, define-syntax made it extremely easy to define recursive macros, but lisp has a number of iteration options that make it close to a non-issue.

8. Iteration

This one's probably the tiniest deal there is. Common Lisp has a bunch of iteration functions/procedures, from the loop macro to dolist, to mapcar and friends. Scheme really only had map and tail recursion, and I sort of preferred that. The reason I list this as "tiny deal" is that my particular CL implementation (SBCL if you must know) does tail-call optimization anyway, so I could just keep up my wicked, functional ways if I wanted to.

9. The IDE

For beginners, PLT wins it. I remember having this conversation with myself earlier; a binary IDE portable across OS X, Linux and Windows, with nice buttons to do things like "Run" and "Macro Step". It's perfect when you're starting out because it's nothing like the near-vertical learning curve of Emacs, but it ultimately limits you. Since I started with PLT Racket, Emacs has become the main program I use. Seriously, something like 75% of all my computer time is spent here, and the rest is split between Klavaro and Conkeror. I'm contemplating getting a shirt that says something along the lines of "Emacs is my master now". Long story short, once you know LISP (or, to be more precise, three LISPs), Emacs is by far the better IDE.

Now that I've laid down all my gripes, the pattern emerges, and it's definitely what Yegge was talking about. Scheme is built to teach and learn (and possibly prove things formally). Even PLT Racket, whose developers are self-declared hackers who go above and beyond the R6RS implementation to provide a pretty decent production candidate, errs on the side of making things easier for beginners rather than easy for veterans, and it stresses academic application over production application. Common Lisp is the precise reverse. It exacts a heavy toll in experience and patience, and the reward is a measure of power beyond other options. It's also crafted (or perhaps evolved would be a better word) for production rather than theoretical purity. I can appreciate that.

So there. If you want the executive summary:

PLT Racket: Theoretical purity and conistancy before practical considerations. Centralized development, indexed for your convenience. Make it easy to learn, consider the newbies.

Common Lisp: Get shit done first, consistency and purity are acceptable collateral damage for terseness. Distributed development, find what you can. Make it powerful, the newbies better watch and learn first.

The choice is pretty simple. Common Lisp wins as soon as you know what you're doing. But while you're getting your bearings straight, go for PLT Racket. For what it's worth, I won't abandon it. I still plan to put out a decent PostScript generation library for PLaneT before I get working on CL hardcore, and I'll always keep it around as a second scripting language (along with Ruby), and I still have several Scheme projects to maintain, but the days of typing M-x run-scheme instead of M-x slime consistently are over for me.

The Upside of Apathy

I think it's finally time to record my thoughts on the new workplace (and in the process muse over some things that have been on my mind lately). And yeah, that bleak sounding title is the reason I'm about twice as satisfied here as I was back at I Love Rewards.

First things first, full disclosure, my first project here was in PHP. And they do their share of .NET programming (and I suspect that a lot of the driver work gets done in C or maybe even Assembly, but that's unverified). They interact with the Postscript standard a lot, but no one does it by hand. Because we serve the medical industry, there's also large parts of CCR and HL7 slung around on a regular basis. That's particularly bad if you understand what those acronyms mean. CCR is an XML-based standard circa 1998, and it exemplifies everything that eventually made sane minds give up that markup for today's light-weight formats. It's the only data standard I've seen that, and I promise I'm not making this up, you have to pay in order to get a copy of. It seems like that would defeat the purpose. HL7 is what you would expect out of a Unix shop from the early ninties. It's a pipe-delimited text stream whose definitions document runs to a hundred pages or so.

OS-wise, I'm in the minority with one other developer who thinks we ought to be using linux-based servers, and I'm actually the only one electing to go with Linux + Emacs for my desktop needs. The place is thick with MSVS and the like. Finally, I'm the only one who uses source control (and it's GIT, as if you had to ask by this point).

So why am I happy in the face of these conditions? It's because, for the most part, I've been hacking Scheme and Elisp through the past few months, and using whatever additional tools I wanted to.

Scheme? At a place that also potentially uses Assembly? Well, yeah. It turns out that no one outside of IT is religious in the least about what languages, tools or systems you choose to use. They really only care that the result is business-applicable in some way, cheap and fast. So as long as you hit those, they don't particularly care. I couldn't get away with this at many other places. Being a Linux-toting, Emacs-using Scheme hacker in the middle of a vast Windows ecology, I mean. There are remarkably few job openings in the field, especially in the bustling, skilled-programmer-metropolis of Toronto. If you want a capitol-J Job here, you need to know (and be willing to work with) Java or C#, the COBOLs of the modern world. Every once in a while someone wants a Python or PHP hacker for some contract work.

But apathy has an upside, like I said, and I'll take it.

It's really interesting to me that once again, passionate, caring (but woefully uninformed) MBAs seem to fuck everything up. If Ken was a hands-on leader, I'd probably have had to learn Java or C# too, or at the very least content myself with constant PHP. This is the Paul Graham effect in full swing; big companies use languages and tools that get sold to the management, not to the devs (and to be fair, if the devs got to pick, we'd all probably be working in C++ and it would be worse). Little companies use the juice that gives the the highest ratio of miles to millileters.

This is the crux of the problem though. The decisions here are such that most people have no idea what the right answer is. Most business people pick what's advertised; thay want to go with the flow. Most devs pick what's fastest in the machine sense; once they're done writing reams of code for their hello world, they want the machine to execute as fast as possible (Contractors seem to want what gives them as many billable hours as possible while not raising too many eyebrows, but that's far from unique to the IT business so I'm disregarding it). It's pretty obvious to the reasoned observer that neither are the correct answer. They might accidentally yield correct answers, but there's nothing about either thought process that makes the right answer more likely than (or even equally likely as) the wrong one. The specific answer depends on the specific situation, granted, but I put forth that in all cases, the correct answer is the most expressive language that will fit on the target hardware. You do want to optimize speed, but not from the machine's perspective. Code needs to be fast and easy to make and maintain, rather than run, which means that you want as little a gap between what you can express and what you want to express. The complaint that gets levelled at non-C languages is that they're slow, which is true until you consider that ~2.4 gH dual-cores and 6gb ram setups are now common on the home market. If you're working on the latest 8-bit IC from Atmel, ok, yes, use C or Forth and constrain yourself to either manual bit-twiddling or the stack. If you're on the desktop, or god help you, a fucking server cluster and still managing memory yourself, then someone (and it may have been you) has made a poor choice on your behalf. This is the sort of obsesive behaviour whose logical conclusion is hand-counting your cereal flakes every morning to make sure you're getting precisely 1024 of them. I think we can agree that the sane thing to do is fill the bowl, hoping the offsets average out in the long term, and getting on with your day.

It seems that democracy doesn't really work here either, because as I hinted above, I doubt that most programmers would pick the most expressive language. They'd either pick what they know just because they happen to know it already, or they'd pick a language that let them code as close to the metal as possible. So how do I know they're wrong, as opposed to me being wrong? Well, that's where the issue goes to shit. The reason I know they're wrong is that I've worked with the low-level languages (C and Forth), I've worked with what I'll call the mid-level languages (Python, Ruby, Java, C#, PHP), and I've worked with four or five LISPs (which I stereotypically place at the top of the progression). As per PG's Blub Paradox, I know that the non-lisps are missing some crucial features that I find myself using on a fairly regular basis (not even macros, interestingly. The prefix notation itself seems to pack a lot of the punch on its own). Also, I've clocked myself as "fast" with some of these languages and "slow" with others. The reason the issue goes to shit here is that this argument will only convince people who have already given several LISPs a fair try and have worked in some other languages from accross the continuum. Those people don't need convincing; if they've come that far, it's a good bet that they've stuck with LISP in some form, or that they're one of the few, vocal pessimists around, loudly proclaiming "all languages suck balls". The people that need convincing are the ones that are currently convinced that C# or Java alone represents the limits of "programming", or the ones that look upon learning new languages as procrastinating.

It's ironic to consider, but it seems like it might just be easier to sell Scheme to the managers and let the traditional pyramid bear the change out. Managers understand the argument "this way is faster", and don't particularly care about the rest as long as you can prove that. There's trouble this way too, though. You see, it's fairly easy to make the argument "this way makes me faster", as long as it's true and you can demonstrate this. But the argument "This will make you faster" is another matter entirely. For starters, in a shop of C programmers, it's patently false. It'll take months of genuine practice to get to a higher level of productivity if your team doesn't already know LISP. There's exactly one other way to go, and that's competing at the company level. In other words, start a bunch of Scheme/CL shops and watch them out-compete the ever-loving fuck out of the Algol descendants. Watch them driven before you, and blah blah blah. It seems like it would work, as long as we stop the AI winter thing from happening again, which looked like it was the result of promising too much, delivering too little, while focusing too much on the math and linguistics and not enough on the business end.

In other words, LISP for business logic instead of research. I think I can do it. I'm certainly in a position to. Even if not, I'll try my hardest and let you know how it goes.

(define (update-again)

Oh fer chrissake, ok.

I figure it's about time to update because I haven't lately. It turns out that you don't stop being busy just because you've stopped working for an employer that demands 20 overtime hour a week. You just get busy doing other stuff. I;m beginning to think that "busy" is either contagious or hereditary.

Mostly the stuff going on here is work on that clandestine project I mentioned a little while ago. Basically, I got to thinking that I really wanted to pick up D&D again. The really sad thing is that my group of friends is pretty geographically disparate, and we don't really feel like driving out for 6 hours to meet up for 8 hours of dice-rolling only to have to weather the drive back. That's not the type of ordeal you can undergo every two weeks.

My first thought was "ok, there has to be something that will let me and my friends play D&D online". I took a look around and found Virtual Tabletop (which has a pricing-system/website so bysantine that I rate their chances of putting together an intuitive in-game interface at very near zero), Open RPG (which has that typical open-source affliction of horrible UI, make-based installation and on top of that neglects OS X, which most of my friends use) and Web RPG (whose link doesn't go anywhere because the only evidence I could find of the project is a page talking about why, specifically, it sucks).

And that's the point at which I started hacking together my own little PLT Scheme app to let me play D&D with my friends. I've registered diecastgames.com (which currently goes to a godaddy parking page), and put together a development blog here in the meantime. It isn't at the point where I want anyone using it yet, but it should be up and running in beta form in the next little while. It'll be almost ugly as sin until I put the art together, at which point we're ready to go. Then I'll just have to convince one of my friends to DM.

Which shouldn't be hard, all things considered. If they get uppity about it, I can just point out that I built a friggin online D&D system from the ground up.)