Deal Journal - Interlude One -- A Treatise on the Design and Construction of Special-Purpose HTTP Servers using Common Lisp with a focus on Readability and Real-World Deployment

Just a heads up: you won't see any of the below code checked into the Deal repo quite yet. And production still uses the nginx PushStream module for its asynchronous needs, and will continue to do so either until I finish enough of the other tasks to pull back enough time to re-structure its server, or until the contest ends and I no longer have to worry about making fast forward progress. Once I get around to it though, yes the real, actual Deal server is going to use this approach. It'll still use nginx as a reverse proxy to make sure static files are being served as fast as possible, but it won't rely on external SSE machinery longer than it has to. Not that the machinery's bad, mind you. It just complicates deployment more than I'd like.

Why and When

Lets just get this out of the way. Yes, there already exist some pretty good general-purpose web servers written in Common Lisp. The reason you'd build your own is if you had a very particular purpose in mind. If you're out to host a vanilla web site, or a traditional stateless, HTML-emitting web application, you'd already have your bases covered. But imagine you were building an application that had a heavy focus on JSON-interaction and session-tracking, with built-in support for SSEs. Say for the sake of argument, something like a web-game. You probably could make one of the existing things work for you. If you tried hard enough, and were willing to hack in support for one or two things[1]. But as you can see by reading the define-handler mini-language I put together for Deal, using a general purpose server for such a specific task incurs complexity penalties that you could completely sidestep by building something minimal and specialized.

That's the point at which I'd start aiming for a goal like this. When using the existing, tested, reasonably-performing options is introducing conceptual and deployment complexity into my project. And, just in case you missed it, it is.

Gotchas

HTTP is old. A child of the mid-ninties. There are people alive and programming professionally today who were born after it was created. Lisp is a bit older. And I can only assume no one working on its standardization knew how big the web would actually get, so they made a couple choices that must have made sense to them, but that will annoy the fuck out of you if you need to generate valid HTTP responses from a CL application.

#\newline is a "platform independent line break", you see. It expands to #\linefeed, #\return or #\return #\linefeed depending on where and how you write it. And the format directive ~% outputs a #\newline. This is relevant because HTTP specifies terminators for header lines and request bodies as #\return #\linefeed, which means that if you were expecting an expression like (format stream "~a ~a~%~{~a~%~}~%~%~a~%~%" protocol response-code headers body) to output a valid response string, I have some news for you.

It's not particularly easy to do this sort of formatting with the format function, ironically. There aren't directives like ~% specified for crlf, or even for #\return and #\linefeed individually. About the best you can hope to do using the defaults is (format t "Hello world.~C~C" #\return #\linefeed), which is a shit sandwich if I ever saw one.

So. Step one, if you're going to be hacking on HTTP from lisp is to define a shortcut for that. It can either be something like

(defun crlf (&optional (stream *standard-output*))
  (write-char #\return stream)
  (write-char #\linefeed stream))

or it can be something like

(defconstant crlf (list #\return #\newline))

(defun cat (&rest things)
  (apply #'concatenate 'string things))

I'm not committed yet, and may just go with defining all of the above. The crlf function looks like it would be more useful for an approach where I tried to do as close to single-traversal output as possible, while the crlf constant/cat function pair looks like they'd be more useful for testing purposes. Like I said, they don't really get in each others' way, so "both" is not, in fact, an unreasonable choice.

That out of the way, lets spend just a moment discussing...

The Approach

As far as I can tell, there are two legitimate approaches to building an async server in Common Lisp.

• First, you could install libevent-core-2.0 and libevent-extra-2.0, and make use of the cl-async bindings. This way seems to save you having to buffer messages yourself, and it probably provides better performance at the expense of a very mildly complicated deployment.
• Second, you could go pure Lisp and build the whole thing out of usockets.

I'm taking that second approach in this article, but you can use the same theory to construct a cl-async-based equivalent without too much trouble.

Now Then

We're building a toy example. The simplest async server that can possibly be constructed while remaining worthy of the name. We're going to have three applicable handlers:

• one to serve up a basic front end, which will use JavaScript to connect to our subscription handler and get future messages from our server
• one to handle that subscription
• one to trigger a message send to all existing listeners

nothing fancy like user-specified messages, or multiple channels, both of which will be fairly easy changes once you understand what the basic server structure is. To start with, we need to generate responses, which means tacking a body message onto some situation-dependent HTTP headers, and handling the above crlf problems elegantly. So, here are the basics:

(ql:quickload (list :cl-ppcre :usocket :cl-who :parenscript :babel))
(defpackage :ts-usocket (:use :cl :cl-ppcre :usocket :cl-who :parenscript :babel))
(in-package :ts-usocket)

(defvar crlf (list #\return #\linefeed))
(defparameter day-names '("Mon" "Tue" "Wed" "Thu" "Fri" "Sat" "Sun"))
(defparameter month-names '("Jan" "Feb" "Mar" "Apr" "May" "Jun" "Jul" "Aug" "Sep" "Oct" "Nov" "Dec"))

(defun http-date ()
  (multiple-value-bind (second minute hour date month year day-of-week dst-p tz)
      (get-decoded-time)
    (format nil "~a, ~a ~a ~a ~a:~a:~a GMT~@d"
            (nth day-of-week day-names) date
            (nth month month-names) year hour minute second (- tz))))

(defun cat (&rest seqs)
  (apply #'concatenate 'string seqs))

(defun response (&key (response-code "200 OK") (content-type "text/html") keep-alive? body)
  (cat "HTTP/1.1 " response-code crlf
       "Content-Type: " content-type "; charset=utf-8" crlf
       "Cache-Control: no-cache, no-store, must-revalidate" crlf
       (when keep-alive?
         (cat "Connection: keep-alive" crlf
              "Date: " (http-date) crlf
              "Expires: Thu, 01 Jan 1970 00:00:01 GMT" crlf))
       (when body 
         (cat "Content-Length: " (write-to-string (length body)) crlf crlf 
              body crlf)) 
       crlf))

crlf and cat are pretty self-explanatory. The response function is going to take some basic options, populated with sane defaults, and use them to generate appropriate HTTP headers. The http-date function just formats the current date in a particular format. In a real implementation, I'd probably end up ditching this one. Its presence/absence doesn't seem to make much of a difference, and it seems like it might be resource intensive[2]. Next up, lets set up our three responses:

(defparameter *channel* nil)

(defparameter interface
  (response 
   :body 
   (with-html-output-to-string (str nil :prologue t)
     (:html
      (:head (:title "Test page"))
      (:body
       (:div :id "console")
       (:script
        :type "text/javascript"
        (str (ps (defvar src (new (-event-source "/sub")))
                 (defun p (msg)
                   (let ((elem (chain document (get-element-by-id "console"))))
                     (setf (@ elem inner-h-t-m-l)
                           (+ (@ elem inner-h-t-m-l) "<p>" msg "</p>"))))
                 (setf (@ src onerror)
                       (lambda (e) 
                         (p "ERROR OCCURRED...")
                         (p (chain -j-s-o-n (stringify e))))
                       (@ src onopen)
                       (lambda (e) (p "STREAM OPENED..."))
                       (@ src onmessage)
                       (lambda (e) 
                         (p "GOT MESSAGE!")
                         (p (+ "data: " (@ e data)))))))))))))

(defun subscribe! (sock)
  (let ((s (socket-stream sock)))
    (write-string (response :keep-alive? t :content-type "text/event-stream") s)
    (force-output s)
    (push sock *channel*)))

(defun publish! (msg)
  (loop for sock in *channel*
     do (handler-case
            (ignore-errors
              (let ((s (socket-stream sock)))
                (write-string (cat "data: " msg crlf crlf) s)
                (force-output s)))
          (error (e)
            (format t "~s" e)
            (setf *channel* (remove sock *channel*))))))

The interface is just a flat file. We'll be sending out exactly the same one to anyone that asks for it, and it would be fairly resource-intensive to generate that each time, so I just cached the result of a response call. What you see in the ps there is a call to create a new EventStream, and calls to attach onopen, onerror and onmessage listeners to it. When the client trips any of those events, it'll add some relevant information to the innerHTML of a div with id console[3]. The subscribe! function sends headers appropriate for starting an SSE stream, calls force-output then pushes the relevant socket into *channel*. Finally, publish! takes a message and sends it out to each member of *channel*. If the write errors, the relevant socket is removed from *channel*.

Notice that we're doing literally everything so far in a very naive way. The http-date mechanics could be made much more efficient, or removed entirely with little negative effect, we're calling concatenate 'string like it's going out of style, and we're calling remove on each and every socket removal. It's just conceptually easier this way. Once we get to a reasonably well-tested server, we'll go back and make sure that we traverse messages as little as possible, writing directly where we can, and make sure to collect running sockets for re-assignment rather than incurring a traversal per removal. Maybe. If the end result doesn't profile well enough.

Don't worry about it for now. Next up is the actual handler.

(defmethod handle-request (socket request-uri)
  (let ((s (socket-stream socket)))
    (cond ((string= "/sub" request-uri)
           (subscribe! socket))
          ((string= "/pub" request-uri)
           (publish! (format nil "Got a message! ~a" (gensym)))
           (write-string (response 
                          :content-type "text/plain"
                          :body "Published!") s)
           (socket-close socket))
          (t
           (write-string interface s)
           (socket-close socket)))))

That'll do it. If we get a request for /sub, we just pass the socket off to the subscribe! function, and pointedly don't close it. If we get one for pub, we publish! a unique message, and send a small, token response to the current requester, then close the socket. Note that if we're closing the connection right away, we don't need to worry about force-output calls, since that automatically gets done on cleanup. Lastly, if we get a request for any other resource, we send out the interface we defined earlier.

We're almost done. We need exactly one more component; a piece that'll monitor a particular port, buffer incoming HTTP requests and call the handler when it gets a completed one.

(defvar *socket-handle* nil)

(defun stop ()
  (when *socket-handle*
    (loop while (socket-close *socket-handle*))
    (setf *socket-handle* nil
          *channel* nil)))

(defclass buffer ()
  ((contents :accessor contents :initform nil)
   (started :reader started :initform (get-universal-time))
   (state :accessor state :initform :empty)))

(defmethod buffered-read! (stream (buffer buffer))
  (loop for char = (read-char-no-hang stream nil :eof)
     until (or (null char) (eql :eof char))
     do (push char (contents buffer))))

(defmethod starts-with? ((prefix list) (list list) &optional (test #'eql))
  (loop for (p . rest-p) on prefix for (l . rest-l) on list
     when (or (and rest-p (not rest-l)) (not (funcall test p l))) 
     do (return nil)
     finally (return t)))

(defun start (port &optional (log-stream *standard-output*))
  (stop)  
  (setf *socket-handle* (socket-listen "127.0.0.1" port :reuse-address t))
  (let ((conns (list *socket-handle*))
        (buffers (make-hash-table)))
    (loop (loop for ready in (wait-for-input conns :ready-only t)
                do (if (typep ready 'stream-server-usocket)
                       (push (socket-accept ready) conns)
                     (let ((buf (gethash ready buffers (make-instance 'buffer))))
                       (buffered-read! (socket-stream ready) buf)
                       (when (starts-with? (list #\newline #\return #\newline #\return)
                                           (contents buf))
                         (format log-stream "COMPLETE ~s~%"
                                 (coerce (reverse (contents buf)) 'string))
                         (setf conns (remove ready conns))
                         (remhash ready buffers)
                         (let ((parsed (parse buf)))
                           (format log-stream "PARSED: ~s~%" parsed)
                           (handle-request ready (parse buf))))))))))

Ok, I may have gone a little overboard in defining starts-with? myself, since there's a similar one already available as part of the library called alexandria, but you get the idea. stop stops our listener explicitly because usocket doesn't free up the socket it's listening on if it errors out, and it also clears out *channel* for the next go.

The buffer class and accompanying buffered-read! procedure are going to make the job of collecting possibly chunked requests easier, and starts-with? does exactly what it says on the tin. It walks a list and a prefix, comparing for equality element-wise by a user-specified predicate, and it keeps going until

• it finds a predicate failure (in which case, it returns nil)
• the list runs out before the prefix (in which case it returns nil again)
• the prefix runs out (in which case it returns t)

The last piece is the meat. The start function is going to take a port and a log-stream, and listen on that port while dumping logging data to log-stream. Lets take this one slow.

...
  (setf *socket-handle* (socket-listen "127.0.0.1" port :reuse-address t))
...

That sets up the listener on localhost listening on the specified port and ensures *socket-handle* is a reference to that listener.

...
  (let ((conns (list *socket-handle*))
        (buffers (make-hash-table)))
...

That initializes conns to be a list whose only element is the listener we just set up. It also sets up a blank hash-table called buffers.

...
    (loop (loop for ready in (wait-for-input conns :ready-only t)
...

We're setting up an infinite loop here. And on each iteration, we're going to wait for some of the sockets in conns to need attention, at which point we will iterate through all ready sockets to give it some attention.

...
                do (if (typep ready 'stream-server-usocket)
                       (push (socket-accept ready) conns)
...

If the ready socket is a stream-server, that means it has a new listener wanting to connect to our server. We push the newcomer onto conns; its request will be handled when it becomes ready.

...
                     (let ((buf (gethash ready buffers (make-instance 'buffer))))
                       (buffered-read! (socket-stream ready) buf)
                       (when (starts-with? (list #\newline #\return #\newline #\return)
                                           (contents buf))
                         (format log-stream "COMPLETE ~s~%"
                                 (coerce (reverse (contents buf)) 'string))
                         (setf conns (remove ready conns))
                         (remhash ready buffers)
                         (let ((parsed (parse buf)))
                           (format log-stream "PARSED: ~s~%" parsed)
                           (handle-request ready (parse buf))))))))))

If it's not a stream-server, that means it's a regular usocket, which means that it has some data ready for us to read. If we wanted to be extremely naive here, we'd just call read-line repeatedly, but just because some data is ready doesn't mean that it represents a complete HTTP request. If the client on the other end is particularly slow, or has a particularly large request to make, it might arrive in chunks. Lets go through that step-by-step.

...
                     (let ((buf (gethash ready buffers (make-instance 'buffer))))
                       (buffered-read! (socket-stream ready) buf)
                       (when (starts-with? (list #\newline #\return #\newline #\return)
                                           (contents buf))
...

We're either getting this particular connections' buffer (if one exists), or assigning it a fresh buffer. We're then calling buffered-read! which will result in the contents of that buffer getting filled with all data available for reading from the ready socket. If that data ends with a reversed crlf crlf, we should do something about it. As a Note to Self here, if it doesn't we should make sure that the buffered data doesn't exceed some pre-determined threshold and that this particular request hasn't been around for too long. buffer already has a started slot, we'll just need to check it and evict ones that get too old.

...
                         (format log-stream "COMPLETE ~s~%"
                                 (coerce (reverse (contents buf)) 'string))
                         (setf conns (remove ready conns))
                         (remhash ready buffers)
                         (let ((parsed (parse buf)))
                           (format log-stream "PARSED: ~s~%" parsed)
                           (handle-request ready (parse buf))))))))))

If we have a complete HTTP request, we emit some logging data, remove that socket from conns and its buffer from buffers, since we won't be reading from it again one way or the other, then call handle-request, passing along the socket and the result of calling parse on the buffer. Eventually, parse should provide a tree of relevant data such as POST/GET requests and all incoming headers, but for right now, we just care about one property of the incoming request.

(defmethod parse ((buf buffer))
  (let ((lines (split "\\r?\\n" (coerce (reverse (contents buf)) 'string))))
    (second (split " " (first lines)))))

That'll return the request uri of a well-formed, incoming HTTP request. handle will then run it through that logic we set up earlier to decide what is to be done.

And that's that. Assuming I've actually got the SSE points ironed out, and we'll see what the mailing lists say on that score, we have a very minimal asynchronous HTTP server built in just under 150 lines of un-obfuscated Common Lisp.

(ql:quickload (list :cl-ppcre :usocket :cl-who :parenscript :babel))
(defpackage :ts-usocket (:use :cl :cl-ppcre :usocket :cl-who :parenscript :babel))
(in-package :ts-usocket)

(defvar *socket-handle* nil)
(defparameter *channel* nil)
(defvar crlf (list #\return #\linefeed))

(defun stop ()
  (when *socket-handle*
    (loop while (socket-close *socket-handle*))
    (setf *socket-handle* nil
          *channel* nil)))

(defun start (port &optional (log-stream *standard-output*))
  (stop)  
  (setf *socket-handle* (socket-listen "127.0.0.1" port :reuse-address t))
  (let ((conns (list *socket-handle*))
        (buffers (make-hash-table)))
    (loop (loop for ready in (wait-for-input conns :ready-only t)
                do (if (typep ready 'stream-server-usocket)
                       (push (socket-accept ready) conns)
                     (let ((buf (gethash ready buffers (make-instance 'buffer))))
                       (buffered-read! (socket-stream ready) buf)
                       (when (starts-with? (list #\newline #\return #\newline #\return)
                                           (contents buf))
                         (format log-stream "COMPLETE ~s~%"
                                 (coerce (reverse (contents buf)) 'string))
                         (setf conns (remove ready conns))
                         (remhash ready buffers)
                         (let ((parsed (parse buf)))
                           (format log-stream "PARSED: ~s~%" parsed)
                           (handle-request ready (parse buf))))))))))

(defmethod handle-request (socket request-uri)
  (let ((s (socket-stream socket)))
    (cond ((string= "/sub" request-uri)
           (subscribe! socket))
          ((string= "/pub" request-uri)
           (publish! (format nil "Got a message! ~a" (gensym)))
           (write-string (response 
                          :content-type "text/plain"
                          :body "Published!") s)
           (socket-close socket))
          (t
           (write-string interface s)
           (socket-close socket)))))

(defun publish! (msg)
  (loop for sock in *channel*
     do (handler-case
            (ignore-errors
              (let ((s (socket-stream sock)))
                (write-string (cat "data: " msg crlf crlf) s)
                (force-output s)))
          (error (e)
            (format t "~s" e)
            (setf *channel* (remove sock *channel*))))))

(defun subscribe! (sock)
  (let ((s (socket-stream sock)))
    (write-string (response :keep-alive? t :content-type "text/event-stream") s)
    (force-output s)
    (push sock *channel*)))

(defclass buffer ()
  ((contents :accessor contents :initform nil)
   (started :reader started :initform (get-universal-time))
   (state :accessor state :initform :empty)))

(defmethod buffered-read! (stream (buffer buffer))
  (loop for char = (read-char-no-hang stream nil :eof)
     until (or (null char) (eql :eof char))
     do (push char (contents buffer))))

(defmethod parse ((buf buffer))
  (let ((lines (split "\\r?\\n" (coerce (reverse (contents buf)) 'string))))
    (second (split " " (first lines)))))

(defun response (&key (response-code "200 OK") (content-type "text/html") keep-alive? body)
  (cat "HTTP/1.1 " response-code crlf
       "Content-Type: " content-type "; charset=utf-8" crlf
       "Cache-Control: no-cache, no-store, must-revalidate" crlf
       (when keep-alive?
         (cat "Connection: keep-alive" crlf
              "Date: " (http-date) crlf
              "Expires: Thu, 01 Jan 1970 00:00:01 GMT" crlf))
       (when body 
         (cat "Content-Length: " (write-to-string (length body)) crlf crlf 
              body crlf)) 
       crlf))

(defun cat (&rest seqs)
  (apply #'concatenate 'string seqs))

(defparameter interface
  (response 
   :body 
   (with-html-output-to-string (str nil :prologue t)
     (:html
      (:head (:title "Test page"))
      (:body
       (:div :id "console")
       (:script
        :type "text/javascript"
        (str (ps (defvar src (new (-event-source "/sub")))
                 (defun p (msg)
                   (let ((elem (chain document (get-element-by-id "console"))))
                     (setf (@ elem inner-h-t-m-l)
                           (+ (@ elem inner-h-t-m-l) "<p>" msg "</p>"))))
                 (setf (@ src onerror)
                       (lambda (e) 
                         (p "ERROR OCCURRED...")
                         (p (chain -j-s-o-n (stringify e))))
                       (@ src onopen)
                       (lambda (e) (p "STREAM OPENED..."))
                       (@ src onmessage)
                       (lambda (e) 
                         (p "GOT MESSAGE!")
                         (p (+ "data: " (@ e data)))))))))))))

(defmethod starts-with? ((prefix list) (list list) &optional (test #'eql))
  (loop for (p . rest-p) on prefix for (l . rest-l) on list
     when (or (and rest-p (not rest-l)) (not (funcall test p l))) 
     do (return nil)
     finally (return t)))

(defparameter day-names '("Mon" "Tue" "Wed" "Thu" "Fri" "Sat" "Sun"))
(defparameter month-names '("Jan" "Feb" "Mar" "Apr" "May" "Jun" "Jul" "Aug" "Sep" "Oct" "Nov" "Dec"))

(defun http-date ()
  (multiple-value-bind (second minute hour date month year day-of-week dst-p tz)
      (get-decoded-time)
    (format nil "~a, ~a ~a ~a ~a:~a:~a GMT~@d"
            (nth day-of-week day-names) date
            (nth month month-names) year hour minute second (- tz))))

Loading that into your repl, then evaluating

(in-package :ts-usocket)
(start 3000)

should start that minimal server on your local port 3000. At which point you can go to http://localhost:3000 in one browser, and hit http://localhost:3000/pub twice in another to see output like

STREAM OPENED...

GOT MESSAGE!

data: Got a message! G1042

GOT MESSAGE!

data: Got a message! G1043

If you keep hammering F5 on localhost:3000/pub, you'll see new messages streaming into the listening window. Once I've gotten this tested, and worked out the kinks, I'll be able to use it to radically simplify the deployment process for the Deal project.

What you don't see above is an implementation of sessions, or very many performance optimizations, but there's a reason I titled this "Interlude Part 1".

---

Footnotes

1 - [back] - Of the three I linked, Araneida has been superseded by Hunchentoot, Hunchentoot works on a thread-per-request model so you'd need to add SSE/websockets/what-have-you support, and Wookie doesn't have sessions (or very good performance, apparently) out of the box.

2 - [back] - To be fair, I probably could put together a system that just generates one per second, and sends that pre-serialized version to each endpoint. That would, at least, save me the effort of having to generate it per-user, but it would complicate things.

3 - [back] - In case you're wondering, the main reason I didn't just console.log is that I happen to use a browser that doesn't have very good console facilities. Or leastwise, it didn't when I last installed it.

Angular Reflections

I've been working with Angular.js for the past little while, both on Web-Mote (still listed as a Perl project for some reason) and various work projects. Overall, the impression is very good. For the most part, the Angular approach saves a lot of lines and cycles over jQuery-style DOM-traversal and manipulation. It always saves a lot of lines over Handlebars-style HTML templating, which was honestly a bit of a surprise at first. Proper routing is slightly more annoying than its backbone.js counterpart, but forces you to break your app out into discrete, composeable pieces, which seems like it would help you scale up down the line.

There are a couple of small places where DOM-traversal seems to be the easier way forward[1], and there's one omission made by the Angular guys[2], but otherwise, I can heartily recommend the library, even after initial frustration.

The big perspective change you need to come to grips with is the shift from imperative/functional-ish programming to a model-centric, reactive approach. Using plain jQuery, you might define a dynamic list as a $.each/append call wrapped in a render method somewhere. You might define the template manually in your js code if it's simple enough, or do it using Handlebars or a similar HTML-templating if it's more involved. If you needed to collect the contents/order of the list later, you traverse the DOM and pull out the chunks you need.

It's not an unreasonable way of going about things, but Angular does it better; the same task is done by using the HTML-DSL to describe the relationship of a model (a literal JS list of objects) to the markup, and then populating that model. The framework reactively updates the DOM whenever model changes occur. Later, when you need the data back, you don't need to traverse anything. You just send the model out to wherever it needs to go.

Lets go through some before and after shots of web-mote for illustrative purposes. Specifically, lets take a look at the controls, since that's the simpler piece. Incidentally, I'm not claiming that this is the most elegant code either before or after. I just want to show you the structural and philosophical differences between approaches.

Before

First, the relevant HTML markup

...

<script id="tmp-control" type="text/x-handlebars-template">
  <li class="{{cmd}}">
    <button class="btn" onclick="mote.command('{{cmd}}');"
            {{#if held}}
            onmousedown="mote.hold('{{cmd}}');" onmouseup="mote.release();" onmouseout="mote.release();"
            {{/if}}>
      <i class="icon-{{cmd}}"></i>
    </button>
  </li>
</script>

<script id="tmp-control-block" type="text/x-handlebars-template">
  <ul>
    {{#each this}}
    {{#control-button this}}{{/control-button}}
    {{/each}}
  </ul>
</script>

...

<div id="controls"></div>

...

The target element gets its own id so that we can refer to it from our jQuery code. The script blocks are Handlebars template declarations. I've elided the rest of the HTML markup because it's all include/template/meta overhead, but you can see it in the appropriate Web-Mote commit if you are so inclined.

Here are the relevant JS declarations

Handlebars.registerHelper("control-button", function (ctrl) {
    return new Handlebars.SafeString(templates.control(ctrl));
});

var templates = {
    control: Handlebars.compile($("#tmp-control").html()),
    controlBlock : Handlebars.compile($("#tmp-control-block").html())
}

var mote = {
    pressed: false,
    hold: function (cmd) {
        mote.release();
        mote.pressed = setInterval(function(){
            mote.command(cmd);
        }, 200);
    },
    release: function () {
        clearInterval(mote.pressed);
        mote.pressed = false;
    },
    renderControls: function (controlLists) {
        $.each(controlLists,
               function (index, e) {
                   $("#controls").append(templates.controlBlock(e));
               })
            },
    command: function (cmd) {
        console.log(cmd);
        $.post("/command", {"command": cmd},
               function () {
                   if (cmd == "pause") {
                       var btn = templates.control({cmd: "play"});
                       $("#controls .pause").replaceWith(btn);
                   } else if (cmd == "play") {
                       var btn = templates.control({cmd: "pause"});
                       $("#controls .play").replaceWith(btn);
                   }
               })
    }
}

// older versions of safari don't like `position: fixed`.
// they also don't like when you set `position: fixed` in a stylesheet,
//   then override that with inline styles.
// what I'm saying is that older versions of safari are assholes
if ($.browser.safari) {
    $("#controls").css({ "position": 'absolute' });
    window.onscroll = function() {
        $("#controls").css({ 
            "top" : window.pageYOffset + 'px'
        });
    };
} else {
    $("#controls").css({ "position": 'fixed' });    
}

command is only relevant because it switches out the pause button for a play button when its pressed successfully. Observe that all of the rendering here is happening through DOM manipulations. We run .append over the result of calling the controlBlock template on each group of player controls, and each call to controlBlock itself applies the control template. When we need to do that button switch I mentioned, we do it by calling .replaceWith on the appropriate DOM selector. We probably could have avoided going to sub-templates for control buttons, but that would have saved us five lines at the outside; just the script tag boilerplate in the HTML markup, and that Handlebars helper definition.

Finally, here's the .ready() call

$(document).ready(function() {
    mote.renderControls(
        [[//{cmd: "step-backward"},
            {cmd: "backward", held: true},
            {cmd: "stop"},
            {cmd: "pause"},
            {cmd: "forward", held: true}
          //{cmd: "step-forward"}
        ],
         [{cmd: "volume-down", held: true}, 
          {cmd: "volume-off"}, 
          {cmd: "volume-up", held: true}]]);
});

That's that. Like I said, this isn't the most elegant code I've ever written. If I really put my mind to it, I might be able to shave off ten lines or so, and clarify my intent in a couple of places, but I think it would be pretty difficult to do much better without fundamentally changing the approach.

After

HTML markup first

<div id="controls" ng-controller="CommandCtrl" ng-style="style">
  <ul ng-repeat="controlsList in controlTree">
    <li ng-repeat="control in controlsList" class="{{control.cmd}}" ng-switch="control.held">
      <button class="btn" ng-switch-when="true" 
              ng-mousedown="command(control.cmd); hold(control.cmd)"
              ng-mouseup="release()" ng-mouseleave="release()">
        <i class="icon-{{control.cmd}}"></i>
      </button>
      <button class="btn" ng-switch-default ng-click="command(control.cmd)">
        <i class="icon-{{control.cmd}}"></i>
      </button>
    </li>
  </ul>
</div>

It should be fairly self-explanatory. That's not the clearest code you're likely to find, but it's illustrative. We've got a bunch of non-HTML directives strewn about; all the stuff starting with ng- is part of the Angular DSL. While we need to do the {{}} thing to evaluate code inside of standard HTML properties, any code inside of ng- properties is automatically run in the context of the controller CommandCtrl.

function CommandCtrl ($scope, $http) {
// older versions of safari don't like `position: fixed`.
// they also don't like when you set `position: fixed` in a stylesheet,
//   then override that with inline styles.
// what I'm saying is that older versions of safari are assholes
    if (util.browser().agent == 'safari') {
        window.onscroll = function() { 
            $scope.style = { position: "absolute", top : window.pageYOffset + 'px' };
        };
    } else {
        $scope.style = { position: "fixed" };
    }

    $scope.held = false;

    $scope.controlTree = [
        [ //{cmd: "step-backward"},
            {cmd: "backward", held: true},
            {cmd: "stop"},
            {cmd: "pause"},
            {cmd: "forward", held: true}
            //{cmd: "step-forward"}
        ],
        [{cmd: "volume-down", held: true}, 
         {cmd: "volume-off"}, 
         {cmd: "volume-up", held: true}]
    ]

    $scope.command = function (cmd) { 
        util.post($http, "/command", {"command": cmd})
            .success(function (data, status, headers, config) {
                $scope.data = data;
                if (cmd == "pause") $scope.controlTree[0][2] = {cmd: "play"}
                else if (cmd == "play") $scope.controlTree[0][2] = {cmd: "pause"}
            })
    }

    $scope.hold = function (cmd) {
        $scope.held = setInterval(function() { $scope.command(cmd) }, 200);
    }

    $scope.release = function (cmd) { 
        clearInterval($scope.held);
        $scope.held = false;
    }
}

That's all, by the way. You've seen all the code for the Angular version, and the two are functionally identical from the users' point of view.

Unlike in the jQuery solution, there's no DOM manipulation here. We've got a model called controlTree which contains the same specification of controls that the earlier version did, but this time, the actual construction of relevant templates is taken care of by the framework. We just specify the relationship between that model and the front-end in the form of the HTML code above, and Angular automatically updates. The clearest demonstration of that is these lines

if (cmd == "pause") $scope.controlTree[0][2] = {cmd: "play"}
else if (cmd == "play") $scope.controlTree[0][2] = {cmd: "pause"}

That's part of sending a command, and all it does is change the contents of our model. The view is updated as soon as this change is made. The equivalent from "Before" is

if (cmd == "pause") {
    var btn = templates.control({cmd: "play"});
    $("#controls .pause").replaceWith(btn);
} else if (cmd == "play") {
    var btn = templates.control({cmd: "pause"});
    $("#controls .play").replaceWith(btn);
}

Where we're back to templating ourselves. You can also see the same principles affecting that code hacking around older versions of Safari; we're just setting up some objects rather than doing DOM traversal ourselves.

if ($.browser.safari) {
  $("#controls").css({ "position": 'absolute' });
  window.onscroll = function() {
    $("#controls").css({ 
        "top" : window.pageYOffset + 'px'
    });
  };
} else {
  $("#controls").css({ "position": 'fixed' });    
}

vs

if (util.browser().agent == 'safari') {
  window.onscroll = function() { 
    $scope.style = { position: "absolute", top : window.pageYOffset + 'px' };
  };
} else {
  $scope.style = { position: "fixed" };
}

The effect is the same, but the particulars of updating and rendering are kept comfortably away from us.

As I said, the above example was picked to clearly illustrate the differences between approaches, not necessarily because it's the biggest gain in clarity I've gotten out of porting over[3]. I'm sure a headache or two will pop up down the line, but I submit that this is a fundamentally more humane way to craft responsive web front-ends than the alternatives.

And I'll be using it where I can from now on.

---

Footnotes

1 - [back] - (re-ordering complex elements is really the only one I've observed; stuff that's too complex to do like this, but where you still need to pass the current order of some set of UI elements back to the server for persistence. As I said already, angular-ui does it just fine for simple constructs, but for anything more complicated, the Angular solution is ~30-lines of sub-module, where the DOM-traversal solution is a mere 5)

2 - [back] - (the $http.post function doesn't do the jQuery thing of encoding an object as POST parameters. The default behavior is to dump the parameter object to a JSON string and pass that to the server as a post body. I could actually see that being the easier approach if you had perfect control of the server, since that would let you do some not-exactly-HTTP processing on the incoming structure. If you're using a pre-built one, though, you're probably stuck doing something manual and annoying like this

...
post: function ($http, url, data) {
  var encoded = _.map(data, function (val, k) { return encodeURI(k) + "=" + encodeURI(val); });
  $http.defaults.headers.post["Content-Type"] = "application/x-www-form-urlencoded";
  return $http.post(url, encoded.join("&"));
},
...

or (if you're concerned about going with the grain of the framework) this

myModule.config(function ($httpProvider) {
  $httpProvider.defaults.headers.post['Content-Type'] = 'application/x-www-form-urlencoded';
  $httpProvider.defaults.transformRequest = function(data){
    return _.map(data, function (val, k) { return encodeURI(k) + "=" + encodeURI(val); });
  }
});

Not too ugly once you throw in the usual pinch of underscore, but this is the sort of thing that really seems like it should be built in as a default behavior. Unless the Angular devs really think some large portion of their users are going to build their own servers to work the other way)

3 - [back] - (in fact, this is probably the least clarity I've gained by moving over to the reactive approach. As I said earlier, the line-count is usually halved without breaking a sweat)

WebMote the Right Way™©

A little while ago, I mentioned that while the new wave of JS frameworks I observed were shit overall, they encouraged the correct approach to web UI building. Well, since I'll have to do a pretty serious re-write of WebMote for use with the RasPi, I figured it would be a good time to apply the principle. This is going to be a two-parter; firstly because I really want to focus on the front-end today, secondly because I don't have the back-end done yet, thirdly because I'm planning to some pretty odd things or the server-side of this project, and fourthly because I'm trying not to bore the ever-loving fuck out of everyone reading this.

So.

First off, have a tree

web-mote
├── css
│   ├── custom-theme ## jQueryUI CSS and images
│   ├── icons ## the SILK icon set
│   ├── style.css
│   └── watermark
│       ├── audio.png
│       ├── folder.png
│       ├── image.png
│       └── video.png
├── js
│   ├── backbone-min.js
│   ├── handlebars-1.0.rc.1.js
│   ├── jquery.min.js
│   ├── jquery-ui-1.8.13.custom.min.js
│   ├── underscore-min.js
│   └── webmote.js
├── root-directory
└── webmote.html

Most of that is framework code, of course. The only files I'll be going through today are webmote.html, webmote.css and webmote.js. You'll note the presence of backbone and underscore; I use their Router class, but don't otherwise touch them for reasons.

<!DOCTYPE HTML>
<html lang="en-US">
  <head>
    <meta charset="UTF-8">
    <title>WebMote</title>
  </head>
  <body>

    <!-- --------- -->
    <!-- Templates -->
    <!-- --------- -->
    <script id="tmp-folder" type="text/x-handlebars-template">
      <li class="{{type}}">
        <button class="play icon play" onclick="mote.playDir('{{path}}')"></button>
        <button class="shuffle icon shuffle" onclick="mote.shuffleDir('{{path}}')"></button>
        <a class="dir-link" href="#navigate{{path}}">{{name}}</a>
      </li>
    </script>

    <script id="tmp-file" type="text/x-handlebars-template">
      <li class="{{type}}">
        <a class="file-link" href="javascript:void(0);" onclick="mote.play('{{path}}')">{{name}}</a>
      </li>
    </script>

    <script id="tmp-control" type="text/x-handlebars-template">
      <li>
        <button class="icon {{cmd}}{{#if css-class}} {{css-class}}{{/if}}" onclick="mote.command('{{cmd}}');">
        </button>
      </li>
    </script>

    <script id="tmp-control-block" type="text/x-handlebars-template">
      <ul>
        {{#each this}}
        {{#control-button this}}{{/control-button}}
        {{/each}}
      </ul>
    </script>
    
    <!-- ---- -->
    <!-- Body -->
    <!-- ---- -->
    <ul id="file-list"></ul>
    <div id="controls"></div>
    
    <!-- ------ -->
    <!-- Styles -->
    <!-- ------ -->
    <link rel="stylesheet" href="css/style.css" type="text/css" media="screen" />
    <link rel="stylesheet" href="css/custom-theme/jquery-ui-1.8.13.custom.css" type="text/css" media="screen" />
    
    <!-- ------- -->
    <!-- Scripts -->
    <!-- ------- -->
    <script type="text/javascript" src="js/jquery.min.js"></script>
    <script type="text/javascript" src="js/jquery-ui-1.8.13.custom.min.js"></script>
    <script type="text/javascript" src="js/handlebars-1.0.rc.1.js"></script>
    <script type="text/javascript" src="js/underscore-min.js"></script>
    <script type="text/javascript" src="js/backbone-min.js"></script>

    <script type="text/javascript" src="js/webmote.js"></script>

  </body>
</html>

That's actually the entire front-end markup. If you haven't seen HTML before, it might look daunting but it's extremely simple. You've got the head up top adding a little bit of metadata, all the text/javascript and text/css includes at the bottom, and only two actual elements in the body of the page: placeholders to pump a file list and control buttons into later. The interesting part is those four handlebars templates.

    <!-- --------- -->
    <!-- Templates -->
    <!-- --------- -->
    <script id="tmp-folder" type="text/x-handlebars-template">
      <li class="{{type}}">
        <button class="play icon play" onclick="mote.playDir('{{path}}')"></button>
        <button class="shuffle icon shuffle" onclick="mote.shuffleDir('{{path}}')"></button>
        <a class="dir-link" href="#navigate{{path}}">{{name}}</a>
      </li>
    </script>

    <script id="tmp-file" type="text/x-handlebars-template">
      <li class="{{type}}">
        <a class="file-link" href="javascript:void(0);" onclick="mote.play('{{path}}')">{{name}}</a>
      </li>
    </script>

    <script id="tmp-control" type="text/x-handlebars-template">
      <li>
        <button class="icon {{cmd}}{{#if css-class}} {{css-class}}{{/if}}" onclick="mote.command('{{cmd}}');">
        </button>
      </li>
    </script>

    <script id="tmp-control-block" type="text/x-handlebars-template">
      <ul>
        {{#each this}}
        {{#control-button this}}{{/control-button}}
        {{/each}}
      </ul>
    </script>

Firstly, notice that they're all in script type="text/x-handlebars-template" tags. I think this technically goes against some markup standard, so it may get screamed at by an XHTML validator somewhere, but it's not a huge deal. If you really feel that's something you want to fix, you can also put these templates in hidden divs instead of script tags; as you'll see later, it wouldn't make a difference in how we use them.

Also, note that I'm inlining the onclick events rather than dealing with them later. A while ago, I sort of got used to the idea of just putting ids and classes on pieces of the DOM tree, then using jQuery to apply events later with $("#foo").click(function () { thingsThatShouldHappen(); });. It looks cleaner at first glance because it separates the presentation and behavior of your controls, but there are two pretty big problems with the approach. First, it requires additional DOM traversals. Those may or may not get optimized away in modern JS engines, but I'm not entirely sure. Second, and more importantly, it makes it harder to change the layout at runtime later, and that's something you'll want to do fairly frequently if you're looking to make a responsive application.

That's actually why I ended up separating tmp-control out of tmp-control-block. I honestly wasn't even going to generate those programatically, but I ran into a situation where I wanted to replace one button with another under certain conditions. I'll point it out when I get to it; the short version is, had I stuck to $ing click events, I would have had to run that setup function each time the change happened[1]. Having those events inlined lets me do the simple thing of just re-rendering a template at the appointed place.

Back to the code at hand.

    <script id="tmp-folder" type="text/x-handlebars-template">
      <li class="{{type}}">
        <button class="play icon play" onclick="mote.playDir('{{path}}')"></button>
        <button class="shuffle icon shuffle" onclick="mote.shuffleDir('{{path}}')"></button>
        <a class="dir-link" href="#navigate{{path}}">{{name}}</a>
      </li>
    </script>

The interesting part of the templates are those snippets of code in the titular handlebars that look something like {{foo}} or {{#foo}}{{/foo}}. A template expects an object, or possibly a list as its input later, and these snippets act as lookups. So, for example, where it says {{path}}, that particular template will look up the value "path" in its argument and echo the result. Now that you know that, the first two templates are fairly self-explanatory.

The last two could use some close-up time though.

    <script id="tmp-control" type="text/x-handlebars-template">
      <li>
        <button class="icon {{cmd}}{{#if css-class}} {{css-class}}{{/if}}" onclick="mote.command('{{cmd}}');">
        </button>
      </li>
    </script>

The single control template demonstrates an #if block. When you see the construct {{#if foo}}{{bar}}{{/if}} in a template, what happening is argument.bar is echoed if argument.foo is not one of "", false, null, [] or undefined.

    <script id="tmp-control-block" type="text/x-handlebars-template">
      <ul>
        {{#each this}}
        {{#control-button this}}{{/control-button}}
        {{/each}}
      </ul>
    </script>

The control-block is responsible for outputting a group of controls. this in a handlebars block refers to the current argument, and the {{#each foo}}...{{/each}} construct iterates through a list. It doesn't seem like there's a good way of iterating over k/v pairs built in, but as that #control-button block demonstrates, it's possible to define your own helpers (spoiler, the control-button helper just renders an individual control).

More on that later though, let me take a very quick look at the CSS before we move on to the JS.

/** The Control Panel *******************************************/
#controls { text-align: center; width: 100%; position: fixed; bottom: 0px; padding: 5px 0px; background-color: rgba(255, 255, 255, 0.8); }
#controls ul { list-style-type: none; margin: 0px; margin-bottom: 3px; }
#controls li { display: inline; }

/** The Main File List ******************************************/
#file-list { list-style-type: none; width: 350px; margin: auto; }
#file-list li { margin-bottom: 10px; padding: 3px; clear: both; }
#file-list li button { margin-top: 0px; float: left; }
#file-list li a { min-height: 20px; padding: 4px; float: left; border: 1px solid #ddd; border-radius: 5px; background: right no-repeat; background-size: 50px; }
#file-list li a.dir-link { white-space: nowrap; width: 228px; }
#file-list li a.file-link { word-break: break-all; width: 320px; }

#file-list li.directory a { background-image: url(watermark/folder.png); }
#file-list li.mp4 a, #file-list li.mov a, #file-list li.ogv a { background-image: url(watermark/video.png); }
#file-list li.mp3 a, #file-list li.ogg a { background-image: url(watermark/audio.png); }

/** BASICS ******************************************************/
h1, h2, h3, h4, h5, h6 { margin: 0px; padding: 0px; }

body { margin-bottom: 130px; }

.pointer { cursor: pointer; }
.hidden { display: none; }
.clear { clear: both; }

/** BUTTONS, MOTHERFUCKER, DO YOU USE THEM?! ********************/
button { background: no-repeat 3px 3px; margin: 3px;  height: 28px; width: 40px; background-color: #ddd; border-radius: 8px; cursor: pointer; text-align: left; }
button.icon { text-indent: 15px; }
button.big { width: 90px; }

/***** specific button types ************************************/
button.volume-up { background-image: url(icons/sound.png); }
button.volume-down { background-image: url(icons/sound_low.png); }
button.mute { background-image: url(icons/sound_mute.png); }

button.rewind-big { background-image: url(icons/resultset_previous.png); }
button.rewind { background-image: url(icons/control_rewind_blue.png); }
button.stop { background-image: url(icons/control_stop_blue.png); }
button.pause { background-image: url(icons/control_pause_blue.png); }
button.play { background-image: url(icons/control_play_blue.png); }
button.ff { background-image: url(icons/control_fastforward_blue.png); }
button.ff-big { background-image: url(icons/resultset_next.png); }
button.shuffle { background-image: url(icons/arrow_switch.png); }

There isn't a lot to explain here. The button section sets up nice, clickable buttons with those icons I mentioned earlier, the basics provide some utility classes. The Control Panel and Main File List sections should give you a pretty good idea of what the final UI is going to look like, and what those watermark images are for.

Ok, now that we've taken that short, stylish break, onto the meat.

var util = {
    requestJSON : function (url, dat, type) {
        var res = null;
        $.ajax({
            url: url,
            data: dat,
            type: type,
            success: function (data) { res = $.parseJSON(data); },
            async: false
        });
        return res;
    },
    getJSON : function (url, dat) { return util.requestJSON(url, dat, "GET"); },
    postJSON : function (url, dat) { return util.requestJSON(url, dat, "POST"); }
};

var mote = {
    targetId: "#file-list",
    render: function (fileList) {
        if (fileList) {
            $(mote.targetId).empty();
            $.each(fileList,
                   function (index, e){
                       if (e.type == "directory") 
                           $(mote.targetId).append(templates.folder(e))
                       else 
                           $(mote.targetId).append(templates.file(e))
                   })
                }
    },
    renderButton: function (control) {
        
    },
    renderControls: function (controlLists) {
        $.each(controlLists,
               function (index, e) {
                   $("#controls").append(templates.controlBlock(e));
               })
            },
    play: function (file) {
        console.log(["cmd", "play", file]);
        $.post("/play",
               {"file" : file},
               function (data, textStatus) { 
                   console.log(["now playing", file, textStatus]);
               });
    },
    playDir: function (dir) {
        console.log(["cmd", "play", "directory", dir]);
        $.post("/play-directory", {"dir": dir});
    },
    shuffleDir: function (dir) {
        console.log(["SHUFFLE", dir]);
        $.post("/shuffle-directory", {"dir": dir});
    },
    command: function (cmd) {
        console.log(cmd);
        $.post("/command", {"command": cmd},
               function () {
                   if (cmd == "pause") {
                       var btn = templates.control({cmd: "play", "css-class": "big"});
                       $("#controls button.pause").replaceWith(btn);
                   } else if (cmd == "play") {
                       var btn = templates.control({cmd: "pause", "css-class": "big"});
                       $("#controls button.play").replaceWith(btn);
                   }
               })
    },
    navigate: function (dir) {
        console.log(["cmd", "display", dir]);
        mote.render(util.getJSON("/show-directory", {"dir": dir}));
    }
}

Handlebars.registerHelper("control-button", function (ctrl) {
    return new Handlebars.SafeString(templates.control(ctrl));
});

var templates = {
    folder : Handlebars.compile($("#tmp-folder").html()),
    file : Handlebars.compile($("#tmp-file").html()),
    control: Handlebars.compile($("#tmp-control").html()),
    controlBlock : Handlebars.compile($("#tmp-control-block").html())
}

var Routes = Backbone.Router.extend({ 
    routes: {
        "navigate*path": "nav"
    },
    nav: mote.navigate
});


$(document).ready(function() {
    mote.renderControls([[{cmd: "rewind-big"}, {cmd: "rewind"}, {cmd: "ff"}, {cmd: "ff-big"}],
                         [{cmd: "volume-down"}, {cmd: "mute"}, {cmd: "volume-up"}],
                         [{cmd: "stop", "css-class": "big"}, {cmd: "pause", "css-class": "big"}]]);
    mote.render(util.getJSON("/root-directory"));

    new Routes();
    Backbone.history.start();
});

I'm mostly using objects as namespaces here, and find myself wishing[2] that JavaScript let me express that more clearly. Ahem. Lets get the quick stuff out of the way first. util is the utility namespace, and contains three helper functions to let me pull JSON data from the server more easily than I could by default. I'm following the functional paradigm and having them return their results rather than depend on a callback to do work.

A bit further down,

Handlebars.registerHelper("control-button", function (ctrl) {
    return new Handlebars.SafeString(templates.control(ctrl));
});

var templates = {
    folder : Handlebars.compile($("#tmp-folder").html()),
    file : Handlebars.compile($("#tmp-file").html()),
    control: Handlebars.compile($("#tmp-control").html()),
    controlBlock : Handlebars.compile($("#tmp-control-block").html())
}

is the handlebars-related code. Handlebars.registerHelper is what makes the template helper function from earlier work properly. Note the new Handlebars.SafeString in that return line, by the way; handlebars templates escape their inputs by default, so passing a plain string won't quite do what you'd want it to in this situation. templates is the namespace in which we keep individual compiled templates. I mean ... templates. Notice that we're just identifying a DOM element by id, and running a dump of its .html() through the compiler. This is what I meant when I said that hidden divs would work just as well as scripts. Your templates can be stored in any DOM element[3], as long as you can reference it when your JS files load. Incidentally, that's why all the script includes in our HTML are at near the bottom of the structure, conveniently after our templates are defined.

Below that, and intruding slightly into the .ready() method is our in-page Router.

var Routes = Backbone.Router.extend({ 
    routes: {
        "navigate*path": "nav"
    },
    nav: mote.navigate
});


$(document).ready(function() {
    mote.renderControls([[{cmd: "rewind-big"}, {cmd: "rewind"}, {cmd: "ff"}, {cmd: "ff-big"}],
                         [{cmd: "volume-down"}, {cmd: "mute"}, {cmd: "volume-up"}],
                         [{cmd: "stop", "css-class": "big"}, {cmd: "pause", "css-class": "big"}]]);
    mote.render(util.getJSON("/root-directory"));

    new Routes();
    Backbone.history.start();
});

This is the entire reason I use backbone and its requirement underscore[4] in this file. The Routes object sets up routes to capture paths starting with #navigate, and calls mote.navigate if it finds one. We do this so that a user of this system will be able to save a link to a particular starting directory. That's also the reason we start the Router up after calling mote.render on the data coming out of /root-directory; that initial rendering would otherwise clobber the result our navigate call. The renderControls call displays all the assorted media buttons we'll need to acceptably control playback.

Lets take a detour before finishing up though; root-directory is for the moment just a text file with some test JSON data in it.

[{"path": "/home/inaimathi/videos/a-show", 
  "type": "directory", "name": "a-show"}, 
 {"path": "/home/inaimathi/videos/friendship-is-magic", 
  "type": "directory", "name": "friendship-is-magic"}, 
 {"path": "/home/inaimathi/videos/khan-academy", 
  "type": "directory", "name": "khan-academy"}, 
 {"path": "/home/inaimathi/videos/porn", 
  "type": "directory", "name": "porn"}, 
 {"path": "/home/inaimathi/videos/bizarre-porn", 
  "type": "directory", "name": "bizarre porn"}, 
 {"path": "/home/inaimathi/videos/horrible-porn", 
  "type": "directory", "name": "horrible porn"}, 
 {"path": "/home/inaimathi/videos/unforgivable-porn", 
  "type": "directory", "name": "unforgivable porn"}, 
 {"path": "/home/inaimathi/videos/Clojure-for-Lisp-Programmers-Part-1.mov", 
  "type": "mov", "name": "Clojure-for-Lisp-Programmers-Part-1.mov"}, 
 {"path": "/home/inaimathi/videos/Clojure-for-Lisp-Programmers-Part-2.mov", 
  "type": "mov", "name": "Clojure-for-Lisp-Programmers-Part-2.mov"}, 
 {"path": "/home/inaimathi/videos/Eben-Moglen--Why-Freedom-of-Thought-Requires-Free-Media-and-Why-Free-Media-Require-Free-Technology.mp4", 
  "type": "mp4", "name": "Eben-Moglen--Why-Freedom-of-Thought-Requires-Free-Media-and-Why-Free-Media-Require-Free-Technology.mp4"}, 
 {"path": "/home/inaimathi/videos/Epic-Wub-Time--Musicians-of-Ponyville.mp4", 
  "type": "mp4", "name": "Epic-Wub-Time--Musicians-of-Ponyville.mp4"}, 
 {"path": "/home/inaimathi/videos/Project-Glass--Live-Demo-At-Google-I-O.mp4", 
  "type": "mp4", "name": "Project-Glass--Live-Demo-At-Google-I-O.mp4"}, 
 {"path": "/home/inaimathi/videos/in-the-fall-of-gravity.mp4", 
  "type": "mp4", "name": "in-the-fall-of-gravity.mp4"}, 
 {"path": "/home/inaimathi/videos/beethoven-symphony-no-9.mp3", 
  "type": "mp3", "name": "beethoven-symphony-no-9.mp3"},
 {"path": "/home/inaimathi/videos/first-lsdj.ogg", 
  "type": "ogg", "name": "first-lsdj.ogg"}]

It's a list of files and directories that a server could reasonably put together and transmit out to the front end just by mapping over the output of listdir or similar. The front-end will expect data represented in roughly this format in order to display a list of files.

Now that we've got an idea of what the data looks like, the last piece of this system can fall into place.

var mote = {
    targetId: "#file-list",
    render: function (fileList) {
        if (fileList) {
            $(mote.targetId).empty();
            $.each(fileList,
                   function (index, e){
                       if (e.type == "directory") 
                           $(mote.targetId).append(templates.folder(e))
                       else 
                           $(mote.targetId).append(templates.file(e))
                   })
                }
    },
    renderControls: function (controlLists) {
        $.each(controlLists,
               function (index, e) {
                   $("#controls").append(templates.controlBlock(e));
               })
            },
    play: function (file) {
        console.log(["cmd", "play", file]);
        $.post("/play",
               {"file" : file},
               function (data, textStatus) { 
                   console.log(["now playing", file, textStatus]);
               });
    },
    playDir: function (dir) {
        console.log(["cmd", "play", "directory", dir]);
        $.post("/play-directory", {"dir": dir});
    },
    shuffleDir: function (dir) {
        console.log(["SHUFFLE", dir]);
        $.post("/shuffle-directory", {"dir": dir});
    },
    command: function (cmd) {
        console.log(cmd);
        $.post("/command", {"command": cmd},
               function () {
                   if (cmd == "pause") {
                       var btn = templates.control({cmd: "play", "css-class": "big"});
                       $("#controls button.pause").replaceWith(btn);
                   } else if (cmd == "play") {
                       var btn = templates.control({cmd: "pause", "css-class": "big"});
                       $("#controls button.play").replaceWith(btn);
                   }
               })
    },
    navigate: function (dir) {
        console.log(["cmd", "display", dir]);
        mote.render(util.getJSON("/show-directory", {"dir": dir}));
    }
}

The mote namespace contains all the relevant navigation commands that we'll need[5]. The render functions do exactly what you'd think. render clears the current file list, takes a list of file and directory objects, and runs them through the file or folder template as appropriate, appending the result to the file list. renderControls takes a tree of control objects and runs them through the controlBlock template, which in turn runs each through the control template and wraps the results in a ul.

The various play/shuffle functions pass file or directory names to whatever back-end process we'll be running this on top of. The command function is the thing that gets called by each control button. For the most part, it just passes that fact along to the back-end system and calls it a day, but there's additional logic for of pause and play signals. In that case, it also switches out the oause button for the play button, cycling between the two on every successful invocation. This is the difficulty I mentioned earlier, and it could still use a helper function or two[6]. Consider how you would write that code

• using $("#controls button").click() instead of inlined onclick events.
• without a separate control template.
• without using templates at all (handlebars or otherwise).

The last part is the navigate function. It uses util.getJSON to request a new set of folders/files and renders them.

And that's it.

This complete front-end, including the CSS weighs in at just over 200 lines of code. It doesn't do anything yet, but that's just because we haven't slapped it in front of an application server. Note that it will be completely, entirely separate from whatever back-end we end up using. It'll work as long as said back-end supports the appropriate requests, and returns the correct data from /show-directory and /root-directory. In fact, if you'd like to make a compatible back-end, all you need to do is provide the following:

• /root-directory
• /show-directory (we could probably get rid of the previous handler by letting this one take a zero-parameter request)
• /play
• /play-directory (if you wanted to provide a polymorphic play handler, you could do without this one)
• /shuffle-directory
• /command

That's slightly misleading because the command handler needs to respond to 10 different parameters, depending on what button the user presses, but it's still much simpler than re-writing the front end for each new back-end. This cuts both ways too; someone wanting to implement their own front-end to the corresponding server needs to implement as many of those handlers as they'll need, and they're good to go.

Keep in mind that this is half the total goal. Next time, we'll see exactly how simple the server-side can get when we completely remove user interface concerns from it. Anyway, thus endeth the lesson. I'm not updating the github repo until I get around to a complete RasPi-compatible media server, but the above code should be easy enough to play with.

---

Footnotes

1 - [back] - It would actually have been even more complicated than that, because it wouldn't have been enough to naively run $("button").click(...) again. That would put duplicate events on every button that wasn't replaced. I'd have gotten into the situation of either targeting just the new button separately, incurring code duplication, or writing the setup function in such a way that it unbound the click event for all buttons first, then re-bound them. That's ... inelegant.

2 - [back] - Not for the first time.

3 - [back] - Or even pre-compiled, if you feel like mucking about with node.js for a bit.

4 - [back] - To be fair, I guess, underscore.js also has a half-way decent looking templating system itself. It looks minimal, but flexible, and I'll be playing around with it a bit after this write up in an effort to potentially boot handlebars out of the include tree.

5 - [back] - console.log calls have been left in for testing purposes, in case you want to take this pure front-end for a spin, by the way. It actually works if you have a reasonably recent browser.

6 - [back] - Which I'll put in if it turns out that more than one button needs this kind of behavior.