Sunday, September 26, 2010

Yegge Strikes Back from the Grave

So I've been fooling around with some new stuff.

Actually, before I tell you about that, quick update. dieCast is now in the early beta stages. It's actually capable of supporting games, but it's got a long way to go before it's something I'll be proud of. We're about three months away from a public beta from where I'm sitting. For the testing stage, I'm ending up using some creative-commons enabled sprites. I'll probably keep them as a subset of the final sprite lineup, if the license permits, but the intention is to get original artwork up.

Ok, now then.

I've been fooling around with some new stuff.

Or rather, some very very old stuff. Over the last couple of days, I've decided to pick up Common Lisp and Portable Allegro Serve again. I gave up on trying to install PAS on SBCL after about twenty minutes though, and promptly switched out to Hunchentoot, and all I really have to say is

Holy.

Shit.

I already have some projects underway with PLT Racket (including Diecast), but goddamit, I think I made the wrong decision. The quote from Yegge goes something like "Most newcomers independently come to the same conclusion; Scheme is the better language, but Common Lisp is the right choice for production work." Bottom line, I remember disagreeing a long time ago, but I've uh... independently come to the same conclusion.

PLT Racket seems to be as good as Scheme gets. It has built-in support for everything from hashes and regexps to x-path and http. It has file-system bindings, guaranteed tail-recursion and pretty much the best package system I've seen (from the downloaders' perspective, at least, Scribble is a bit of a bitch to get familiar with if you plan to actually document your own code).

So why am I having serious second thoughts?

Lots and lots of little things. Now that I've actually had some time to play with both contenders, mastered both IDEs, played with both macro debuggers, ran web servers on both and lived in each language for a decent length of time, I think I can finally compare them, and gain some sliver of insight from the comparison. And it's a damn close race. The biggest differences turn out not to be what everyone was pointing at. I have a link in the sidebar over there pointing to "Scheme vs. Common Lisp", which purports to tell you the differences between the two, and maybe three of those actually trip you up to any significant degree.

So here's the big stuff. PLT Racket vs Common Lisp from a young hacker's perspective.

1. Documentation

The PLT Docs are badass, and centralized. Second to none. They have Ajax search running over all functions in their implementation (and you need it with the amount of stuff it has), code examples all over the place, and a comprehensive set of tutorials perfect for beginners. Common Lisp probably has more overall information on it, but it's scattered across CLiki, Common Lisp Directory, Hyperspec, various indie package pages and Bill Clementson's archives. M-x slime-documentation-lookup helps, but it only searches the Hyperspec. That's plenty of info for the veteran, but (if I could imagine my point of view about two years ago) it wouldn't be sufficient for someone who's, say, looking for a complete listing of format-string options (As a public service, the way you find that is to look up format, then scroll about half-way down the page where you will be pointed to section 22.3 for more information on formatted output).

2. Package Repositories/Installation tools

EDIT:

I can't believe I managed to go so long without adding this note. As of the end of 2010, quicklisp also exists, and is awesome. That means that the gripes I had about asdfing things are moot, since you don't need to for the most part. Thanks to ql:system-apropos, it's also fairly easy to find CL packages, so I guess PLT Scheme (now Racket) no longer wins this one. I have no idea what they've been up to for the last year though, so they probably made a thousand and one improvements all over the place too.

Thu, 11 Oct, 2012

Common Lisp has asdf, which is awesome compared to the tools found in most other languages I've used, but PLT beats it pretty handily. It's basically the same story as documentation. There's technically more stuff out there for CL, but it's scattered, and since development is distributed, you'll get some duplication of effort. There are four or five different HTTP servers, for example, and at least three HTML-templating libraries. Granted, there's a clear "best" in each category, but you really need to do your reading in order to find that out. PLT has a smaller offering (the biggest gaping holes are in the document generation area; there is no such thing as a good PLT Racket PDF/PostScript generator), but it's neatly organized, indexed, and accessed by typing (require (planet [package-name])) in the declaration section of whichever file you need the new package for. No hunting, no missing GPG keys. These first two points are probably the ones I'll miss most from the PLT offering.

3. The Web Server

This is actually a place where more choice would do PLT Racket some good. They do have a pretty cool web server, but it's far from fast in practice. It also seems to crash more often than I'd like for a production app. Nothing like once per week, but it's happened a few times so far. The trouble is how it behaves. It's basically Tomcat, minus the copious installation headaches; you need to get all your code in order, make sure it'll run, then execute. And that's it. If you need to make changes (like, while developing web apps) you need to tweak the code, then restart the server, then re-navigate to the page you were just on because it auto-generates new urls each time. This is an exercise in frustration, and is one reason that I've still kept up on my PHP and Python skills this entire time. The languages may be slightly worse, but they're interpreted, so a change doesn't need to bring down the whole server. That's how Hunchentoot works. You load your files, then start the server. If you need to make a change, you evaluate the new code against the actual, still-running server. I wouldn't use this in the wild, but during the development stage, it is hot, buttered, bacon-wrapped power. That alone seems to be enough to pull CL into the lead as far as my use of it is concerned.

Now in, PLT's defense, they're aware of this. There was a concern about keeping LISP's inherently reflective nature, and they decided not to because it trips up so many people that they figured it wasn't worth the headaches. So the server forces you out while it runs, and the REPL bugs you to do a clean run every once in a while if your source has changed. I appreciate the sentiment, because it really was made to be a teaching tool, but I'm being mighty tempted by the dark side regardless. There's also a concerted effort from PLT to keep things byte-oriented. For example, there is no supported way to get a list of POST/GET parameters out of a request (other than "manually") in PLT scheme. "Manually" entails getting a list of binding objects out of the request and mapping over them to get a list of byte-strings out. There's also a few other little gotchas (like how awkward it is to actually create a link whose result is another scheme function, and how url-based dispatch is for whatever reason NOT the default).

4. The Format function

This may sound like a nitpick, but I'm not into the nitpicks yet. This is actually a difference. In PLT Racket, you're stuck with (format "~a" blah). It only accepts formatting directives, rather than CL's richer set of formatting, flow control and kitchen sink. It also always returns its result, and doesn't have the option of printing to standard-out (you have to use printf for that). I didn't think this would make as big a difference as it did, actually, because I've gotten used to the simpler Scheme format, but hot damn is it awesome to be able to do something like (format nil "~a ~{ ~a: ~@[ ~a ~]~}" (car blah) (cdr blah)) instead of resorting to several function calls for the same effect.

5. Plists

Basically same story as above. I forgot how useful these actually were for day-to-day purposes. I mean, I still bust out hashes for bigger stuff, but little tasks all over the place are made just a tiny bit easier with the use of p-lists instead of a-lists.

6. Function names.

Ok, now we're into picking nits. It's not a huge deal, but the scheme conventions are cleaner and more consistent. If you're dealing with a predicate, it ends with "?", if you're dealing with a side-effect function, it ends with "!". Common lisp has a grab-bag. Some predicates end with "p" (as in listp), but most are just the unmodified word (as in member). Also under this category is the lisp-1 vs lisp-2 thing; because there's separate namespaces for functions and variables in CL, there's two let types (let for variables and flet for functions) and two definition types (defun and defvar). Because functions get treated differently from other variables, some things are a bit trickier in CL; for example, while you can still do (apply (lambda () 42) '()) or (mapcar (lambda (num) (* 2 num)) '(1 2 3 4 5)), you actually can't do something like (setf foo (lambda () 42)) followed by (foo) (you would either need to call foo with (funcall foo) or define it as (setf (symbol-function 'foo) (lambda () 42))). The Scheme equivalent is (define foo (lambda () 42)), after which (foo) does exactly what you think it will.

7. Macros

PLT Racket has define-syntax-rule and define-syntax, as well as library support for define-macro, which is a copy of CL's non-hygenic defmacro. In practice, I found myself using define-macro most of the time, so it shouldn't be too big a problem to switch here. Admittedly, define-syntax made it extremely easy to define recursive macros, but lisp has a number of iteration options that make it close to a non-issue.

8. Iteration

This one's probably the tiniest deal there is. Common Lisp has a bunch of iteration functions/procedures, from the loop macro to dolist, to mapcar and friends. Scheme really only had map and tail recursion, and I sort of preferred that. The reason I list this as "tiny deal" is that my particular CL implementation (SBCL if you must know) does tail-call optimization anyway, so I could just keep up my wicked, functional ways if I wanted to.

9. The IDE

For beginners, PLT wins it. I remember having this conversation with myself earlier; a binary IDE portable across OS X, Linux and Windows, with nice buttons to do things like "Run" and "Macro Step". It's perfect when you're starting out because it's nothing like the near-vertical learning curve of Emacs, but it ultimately limits you. Since I started with PLT Racket, Emacs has become the main program I use. Seriously, something like 75% of all my computer time is spent here, and the rest is split between Klavaro and Conkeror. I'm contemplating getting a shirt that says something along the lines of "Emacs is my master now". Long story short, once you know LISP (or, to be more precise, three LISPs), Emacs is by far the better IDE.

Now that I've laid down all my gripes, the pattern emerges, and it's definitely what Yegge was talking about. Scheme is built to teach and learn (and possibly prove things formally). Even PLT Racket, whose developers are self-declared hackers who go above and beyond the R6RS implementation to provide a pretty decent production candidate, errs on the side of making things easier for beginners rather than easy for veterans, and it stresses academic application over production application. Common Lisp is the precise reverse. It exacts a heavy toll in experience and patience, and the reward is a measure of power beyond other options. It's also crafted (or perhaps evolved would be a better word) for production rather than theoretical purity. I can appreciate that.

So there. If you want the executive summary:

PLT Racket: Theoretical purity and conistancy before practical considerations. Centralized development, indexed for your convenience. Make it easy to learn, consider the newbies.

Common Lisp: Get shit done first, consistency and purity are acceptable collateral damage for terseness. Distributed development, find what you can. Make it powerful, the newbies better watch and learn first.

The choice is pretty simple. Common Lisp wins as soon as you know what you're doing. But while you're getting your bearings straight, go for PLT Racket. For what it's worth, I won't abandon it. I still plan to put out a decent PostScript generation library for PLaneT before I get working on CL hardcore, and I'll always keep it around as a second scripting language (along with Ruby), and I still have several Scheme projects to maintain, but the days of typing M-x run-scheme instead of M-x slime consistently are over for me.

20 comments:

  1. 6 is a bit off. (apply (lambda (foo) bar) baz) is perfectly fine, as the first argument to APPLY evalutes to a function object. Assigning a lambda to a variable is fine, too: (setf foo (lambda () 42). The difference is that you can't then call (foo) to call that lambda and get 42, you have to use (funcall foo). But that doesn't work when you need to apply a function bound to a variable in Scheme, either.

    ReplyDelete
  2. Thanks for pointing that out. My intention was just to show that there isn't a perfect isomorphism between variables and functions in CL the way there is in Scheme.

    Fixed above.

    ReplyDelete
  3. Another note about point 6: you can do (setf (symbol-function 'foo) (lambda (num) (* num 2))), in which case you will be able to call FOO as a function afterwards.

    ReplyDelete
  4. I freakin' love plists while prototyping and exploring! So much clear info in debugging statements or when working in the REPL.

    I rarely replace them with classes once I'm done.

    ReplyDelete
  5. Minor mistake in point 4 : your last example of the CL:FORMAT function is missing the STREAM argument.

    ReplyDelete
  6. @Drew: You are correct, I've updated the post. While I was reading over, I also noticed a mis-called cl:map in number six. Really, I meant mapcar, since you need to pass a collection type to map in CL.

    @Jānis: Added a note to that effect. The point wasn't "you can't pack functions in varibles in CL", just that there's some overhead involved since it's not the default option and the two are treated as separate. By contrast, in Scheme, there is no difference. So anything you want to do in order to pass a variable around will also work on a function.

    ReplyDelete
  7. hi Inaimathi,
    Nice article.

    btw, i tried to subscribe your blog. Tried follow. Google reader says am already following but your blog doesn't show. Nor can i find a atom webfeed...

    ps thanks for links to my elisp tutorial.

    ReplyDelete
  8. oh nvm it works now. :) either i was mistaken or going thru blogge/reader manager made it appear.

    ReplyDelete
  9. @Xah Lee: Are you kidding? Thanks for the tutorials! They've been the most useful resource for me in learning Emacs (along with M-x apropos and M-x describe-key).

    ReplyDelete
  10. Hi Inaimathi,

    I'm the developer of the Racket Web Server. I have a few questions based on your post.

    "They do have a pretty cool web server, but it's far from fast in practice. It also seems to crash more often than I'd like for a production app. Nothing like once per week, but it's happened a few times so far."

    I'm interested in how it crashed, logs, stack traces, anything like that. The only thing like this I've experienced in deployments is memory related and the LRU manager alleviates that.

    [see next comment for more]

    ReplyDelete
  11. "Now in, PLT's defense, they're aware of this. There was a concern about keeping LISP's inherently reflective nature, and they decided not to because it trips up so many people that they figured it wasn't worth the headaches. So the server forces you out while it runs, and the REPL bugs you to do a clean run every once in a while if your source has changed. I appreciate the sentiment, because it really was made to be a teaching tool, but I'm being mighty tempted by the dark side regardless."

    This may be what others say. In my opinion, it is good to not allow this because it stands in the way of simple inlining optimizations and analysis.

    ReplyDelete
  12. "There's also a concerted effort from PLT to keep things byte-oriented."

    The HTTP protocol is byte-oriented. In Racket, strings are always UTF-8, but HTTP data may not be, so there is a concerted effort to not error when non-UTF-8 data is sent to the server.

    "For example, there is no way to get a list of POST/GET parameters out of a request (other than "manually") in PLT scheme. "Manually" entails treating the request as an object with byte keys, which you can't map over cleanly."

    I'm not sure what you're talking about. request-bindings gives you an alist of all the parameters with their value. request-bindings/raw gives you a list (which you can map over) of all the binding data structures that contain this same information but in a byte-safe/file-upload-safe way. Is there something I'm missing?

    ReplyDelete
  13. "There's also a few other little gotchas (like how awkward it is to actually create a link whose result is another scheme function,"

    (define (other-function req)
    "You clicked me!")
    (send/suspend/dispatch (lambda (make-url)
    `(html (body (a ([href ,(make-url other-function)]) "Click me")))))

    I personally don't find this awkward because almost every response function starts with "(send/suspend/dispatch (lambda (make-url) ...))" so the only thing you need to do is write "(make-url the-function)" which seems pretty streamlined to me.

    Can you give me some suggestion to make it better?

    "and how url-based dispatch is for whatever reason NOT the default)."

    I don't understand what you mean by "default". What is inconvenient that you wish were more convenient? As far as I can tell, there isn't really any default.

    Jay

    ReplyDelete
  14. Hello, Jay. I know who you are; this post aside I'm actually a fan of PLT.

    "I'm interested in how it crashed, logs, stack traces, anything like that. The only thing like this I've experienced in deployments is memory related and the LRU manager alleviates that."

    It probably was a problem with limited memory, but I don't have any kind of reporting set up on the machine t's running it. What I know that I ran the server and checked back some days later to find that it no longer was (the machine wasn't restarted; other processes I started at the same time were still active and hadn't been restarted)

    "This may be what others say. In my opinion, it is good to not allow this because it stands in the way of simple inlining optimizations and analysis."

    My mistake. I thought I read a story somewhere (possibly the PLT blog; I'll check) about how difficult it was to predict the interactions of newly evaluated code with existing macros. The story seemed to suggest that this difficulty was the reason to make reloading the entire module mandatory (as opposed to letting the user evaluate piecemeal changes over top).

    In any case, my opinion is that the extra iteration speed you get during development outweighs the downsides introduced by interactive evaluation.

    "The HTTP protocol is byte-oriented. In Racket, strings are always UTF-8, but HTTP data may not be, so there is a concerted effort to not error when non-UTF-8 data is sent to the server."

    True, but most ways I've seen of handling GET/POST input don't require the developer to think at that level. Hunchentoot lets you retrieve a list of bindings (or just lets the your response functions take them as arguments), PHP gives you the $_POST variable, and Python does something similar.

    PLT's web server is the clear outlier, since it expects you to do something like (bytes->string/utf-8 (binding:form-value (bindings-assq #"[field_name_here]" (request-bindings/raw req)))) to get a single binding out in string form.

    ReplyDelete
  15. [continued from above]

    "request-bindings/raw gives you a list (which you can map over) of all the binding data structures that contain this same information but in a byte-safe/file-upload-safe way."

    Now that you've pointed to the use of request-bindings/raw, it seems like you could map over the results by doing something like (map (lambda (b) (binding-form-value b)) (request-bindings/raw req)) which would get you a list of form outputs as byte-strings (My fault; I'll retract that objection from the blog post but keep the commentary for historical reasons). That means there's a workaround, but being that I'm quite likely trying to use any (non-file) bindings as vanilla strings somewhere, would it not make sense to provide that output in a simple manner? To me, the particularly bothersome part is that PLT once had such functions. In fact extract-bindings does precisely what I'm looking for in this situation, but has had a dire note attached in the documentation for quite a few versions stating that it exists for backwards-compatibility only.

    "Can you give me some suggestion to make it better?"
    "I don't understand what you mean by "default". What is inconvenient that you wish were more convenient? As far as I can tell, there isn't really any default."

    The default seems to be the send/suspend/dispatch + embed/url way of doing it (which involves wrapping your actual response function in a `local`, then dispatching to it, or passing a lambda to send/suspend/dispatch as you showed in your example above). The url-based dispatch is provided in a separate module entirely. A better way would be the way that Hunchentoot does it, not to put too fine a point on it. Which is to say

    (define-easy-handler (test-page :uri "/easy") ()
    (page-template (:p "This is easier")))

    When I want a link to result in that function, I just make sure the url it calls is "/easy". This doesn't involve a call to embed/url, or an explicit local definition and dispatch call (define-easy-handler likely expands into these, but I don't have to keep typing them myself).

    ReplyDelete
  16. PS from above

    That story was indeed from the PLT Blog, but not quite the same context. It was a post by Matthias Felleisen at http://blog.racket-lang.org/2009/03/drscheme-repl-isnt-lisp.html explaining why the DrScheme REPL expected you to reload an entire scheme file rather than making it easy to send in one command at a time. In my defense, the explanation does involve making this design decision specifically because Matthias saw so many people "stumble across the state of the repl". I assumed the same reasoning ran with the web server.

    I could be wrong, but I still respectfully disagree with the decision.

    ReplyDelete
  17. It sounds like you want to use the 'web-server/http/bindings' library and deal with the potential problems that other framework simply hide from you. We have no intention of removing this library.

    s/s/d is no more a default than dispatch-rules. They are both part of web-server/servlet (the library for writing servlets.)

    Your "easy handlers" are basically dispatch-rules, although it also gives you a URL creating function so you can write (test-page arg ...) and have that expand into "/easy/arg ..." so you don't need to remember what URL calls test-page.

    The main difference is that dispatch-rules is functional: you write all the URL patterns in one place, whereas define-easy-handler appears to imperatively modify a global dispatch table. It should be easy to make a new library like dispatch-rules that behaved that way, but we haven't done it. (We = me; and I haven't done it because I think it is bad taste.)

    However, the dispatch/untyped package does do something similar to what you want, so you may want to use that instead.

    Thanks for writing this post and I hope I can find ways to improve the Web server for users like you,

    Jay

    ReplyDelete
  18. @Jay: Thanks for taking the time to ask.

    I meant it when I said I was a fan. The world needs a learning-friendly Lisp that can extend to production purposes, and you guys are doing a lot of good on that front.

    Keep up the good work.

    ReplyDelete
  19. Nice review from someone who tried both platforms.

    As Yeggie stated, I have come to same conclusion too.

    > Granted, there's a clear "best" in each category, but you really need to do your reading in order to find that out.

    Would you please share your findings? To me it seems that Weitz's packages - HUNCHENTOOT, HTML-TEMPLATE, etc. - are to be preferred when developing Web apps. Maybe, Weblocks too.

    ReplyDelete
  20. @lele - Yeah, the rule of thumb seems to be "If Edi Weitz wrote a package for something you need to do, use his". As far as I'm concerned, Hunchentoot is at the head of the pack in terms of application servers (and if I had more static pages to host, I'd do so using Antiweb). I also use cl-who for HTML generation. I haven't tried any of the web frameworks for real yet, so they may make things easier.

    I can't really back up my claims with specific evidence, unfortunately. I didn't document my decision process. What I remember is pulling up http://www.cliki.net/Web and going through lots of the documentation/"getting started" tutorials before settling.

    ReplyDelete