Saturday, November 30, 2013

Combining SVG and HTML5 is still a Drag

Dragging SVG elements is harder than it might appear at first glance.

The underlying implementation of the SVG DOM and the HTML DOM is different in current browsers you see, so the standard HTML5 drag event doesn't apply to SVG nodes. Luckily, mousedown, mousemove and mouseup are supported, so you'd think it would be a straight-forward task to implement the fucker yourself. You probably imagine, as I did initially, something that takes a selector and a list of callbacks, and implements something similar to jQuery's .draggable() in ~30 lines of code by

  • binding a callback to the targets' mousedown, mousemove, mouseup events
  • storing the initial position of the element, and its delta from the mouse cursor
  • preventing default on mousedown
  • in addition to firing the callback, moving the target element by manipulating x and y coordinates using the current mouse position, initial delta and initial position

Maybe that's possible for the simple cases, but it's the edges that'll get you. And unless an implementation happens to dull all your edges, it's not really good enough.

Minor speed-bump. What if you need to drag multiple elements? If you were dealing with HTML elements, it wouldn't be such a big deal, but SVG elements have different properties to represent their coordinates. Rectangles and similar have an x and y that represents their top-left corner, circles and ellipses have cx and cy that represents their center, text elements have an x and y, but theirs represents the bottom-left coordinate, and I'm not even getting into the path elements. Bottom line, if you want to implement something that works, you're using transform settings. Also, you're not doing it naively through setAttribute, unless you're lucky enough to have a situation where you can guarantee that no other transformations will be applied to any draggable element[1]. The snippet that handles that particular piece looks like this in my codebase

(defpsmacro =translate (elem/s dx dy)
    (=>> ,elem transform base-val (get-item 0) (set-translate ,dx ,dy))
    (:catch (error)
      (=set-attribute ,elem :transform (+ "translate(" ,dx "," ,dy ")")))))

If you're one of the sad bastards who don't have macros at their disposal, I guess you're doing that or something fairly similar manually[2]. Like I said though, no big deal either way.

Medium sized speedbump. If you want to do this on an arbitrarily sized element, specifically a very small one, you'll discover that moving your cursor at even moderate speeds is enough to escape from the mousemove event and leave your draggable behind. One possible solution here is to also bind the mouseleave event and hope you never need to move fast enough to escape that too. Another approach is to have your chosen mousedown set up a global mousemove event, on body or html, use that to drag your element around, and have a global mouseup waiting to cut it off as soon as you're done[3]. A bit hacky, but doable.

Slightly larger speedbump. If you want to make these bindings switchable, you're in for a bit of a harder time. Not switchable as in "different objects should be able to do mildly different things", that's a given. I mean like "it should be possible to jump into a separate interaction mode where the same object does something mildly or wildly different under certain circumstances". If you want that, you need a level of indirection in your listener tree that you can swap out with other functions, and that level of indirection is going to be calling an externally specified function on each event trigger. Basically, you'll want to be working with hooks rather than listeners at this point[4]. I'll keep you posted on how this one goes in real life.

Large speedbump. Suppose you want to be able to use your dragging events, and a mouseup event on the same element. Better yet, suppose you wanted to implement drag/mousedown interactions, but let the user decide what layer to apply them on at any given time. Imagine a situation where you had the elements foo, overlapping bar, overlapping baz, and when a drag or mousedown hits, you want to let the user decide whether they want to be click/dragging foo and/or bar and/or baz. Near I can tell, there is no way of implementing this elegantly in terms of listeners on individual elements. What you need if you want this is a central listener that delegates particular events out to some intermediary functions, or eats them[5] as appropriate.

Keep in mind that the last two speedbumps I hit here probably won't be felt by most people going in the same direction. Still, I went into this figuring it'd take me a half hour at the outside to implement something workable. It ended up taking me the rest of the day, and will probably cost me another hour or two when I get back in on Monday.

Such is development sometimes, I suppose.


1 - [back] - If you are going to have other active transformations, using the setAttribute method would overwrite those, which is why it's a bad idea.

2 - [back] - if you are doing that, I should point out that the only reason I went the try/catch route here is that both =>> and =set-attribute take either an element or a set of elements as their first argument, and I wanted =translate to do the same. Since you probably won't have the same situation, you're likely better off with if/else.

3 - [back] - You wouldn't want to do this naively either, unless you knew there'd be no other mousemove events on that top-level element. If you did have that, you'd want to set up a hook that you could change out rather than messing with event listeners every time you dragged something.

4 - [back] - It just occurred to me that you might have no idea what I mean by "hook" in this context. Basically, something like this:

var mouseMoveHook = null;
  function (event){
    //do other stuff
    if (mouseMoveHook) { mouseMoveHook(event) };
If you have something that looks like that, you can change some of the behavior of your global mousemove event by assigning a new callback to the mouseMoveHook variable. I'm sure it's been used elsewhere, but I learned the term "hook" from Emacs, which provides standard event hooks in a bunch of different situations, and does it more or less this way, modulo some syntactic sugar and multiple hook support.

5 - [back] - In the case of the trailing mouseup event after a drag concludes.

Tuesday, November 26, 2013

Arbitrary Update 714

We're working on FBP in Common Lisp. You've heard that one before.

Step one was getting a basic system running. In fact, we've got two we're dealing with concurrently[1]. Both are kind of rickety because I'm still trying to work out what the essential parts of the approach are. One deals with OS threads and the other is built on top of cl-async. I'm liking the second one better for now, but am once again reserving judgment until I see how easily they explode.

Anyhow, that's price of admission, rather than the finish line. Our ultimate goal is to prove out visual programming in the large, without making anyone throw up about it. I have no idea how that's going to end up, except that I've definitely got a couple of interesting years coming up. In order to get to that ultimate goal, step two is finding or building an Acceptable™© Visual Editor.

And we can't find one, so "build" it is.

What we want, being that we're Lispers and polyglots to the last, can best be described as "Emacs for Diagrams". And that doesn't seem to exist. There's a number of general-purpose vector editors out there at various levels of readiness, but those are more like "MS Word for Diagrams". There's a "Sublime for Diagrams" floating around in the form of yEd, and one piece of fallout from the Russian space program is basically "Notepad++ for Diagrams". Microsofts' own flowcharting suite is something like "Eclipse: The Extra Shit Version for Diagrams. Also You Can't See Our Source Code, Which Is Probably For The Best Because It's Mostly Garbage You'd Laugh Out Of Your Company In Any Other Context". Finally, if we want to use Emacs for diagrams, the only thing even remotely workable seems to be artist-mode. It's nice, I guess. Certainly better than prostate cancer. But that's not quite what we're looking for.

The thing we are looking for needs to be

  • A visual editor. This needs to be a tool to let humans construct diagrams and related diagrammatic artifacts for a variety of media.
  • A diagram editor. We're not looking for a general purpose editor. In particular, there are very, very few things we'll be rotating, nothing we'll be coloring[2], and we only need one font at a fairy small variety of font sizes and weights. We'll also never be dealing with embedded bitmaps, gradients, brush-strokes, or shapes more complex than rounded rectangles/ellipses. You'd be surprised how much junk that lets you cut out.
  • Fast. It shouldn't take longer to draw a diagram with this thing than it would take to describe it. This means a keyboard-oriented interface with easily re-bindable keys, reasonable performance, reasonable start-up, and no half-baked expert system getting in your way by trying to guess what you're trying to do.
  • Scriptable. It should be possible to simply and easily construct sequences of operations to be summoned later. It should be easy to run automated queries and transformations on the programs' output and potentially edit the result in a visual manner. It should be fairly simple to select a particular subset of elements and manipulate them in some way.
  • Flexible. We have no idea what the visual formalism we're finally going to adopt will look like. It might require fine placement of wires and connectors, it might involve different kinds of connections, and it will certainly involve multiple different connections between two nodes[3]. It also can't make any kind of semantic assumptions about what the content we're editing means, because that might change as we go through the discovery process of what kinds of processes and situations we want to formalize. This also means some sort of proper macro system, an end-user specifiable final/intermediate representation, and source code that we'll be able to tear apart if we need to.

The various editors I list above are split firmly into two camps. First, there are editors aimed at non-programmer humans for general vector construction. They're fairly flexible, but because they're for mouse-pushers, they don't provide a good intermediate representation for our purposes, don't particularly care about ease-of-use or keyboard-centricity, and don't make it particularly easy to script parts of your workflow. In the second camp, there are specialized flow editors aimed at programmers or technical managers, which do provide a good deal of flexibility on the keyboard binding front, and provide appropriate output, but also make all kinds of assumptions about what you're doing, why, and how you should want to go about it.

If you know of something awesome that I missed, let me know, but for the moment we're rolling our own. I've been working on it for about half a week solid now, and it's taking shape pretty quickly. I showed off the pre-alpha to some folks at the Erlang conference last week, and got reactions that didn't look like outright disgust. Once it takes some usable shape, I'm planning on posting some videos for you, and possibly showing it off a bit to the FBP group.

I was also going to talk a little bit about fact bases, and the inherent strength of simple models, but I think I'll leave that for another day. When I get permission to Free this editor, I'll have quite a bit more to say about that in any case.


1 - [back] - Har har, aren't I droll.

2 - [back] - Except for UI purposes, obviously. The various selection/manipulation layers are going to have distinct visual cues that rely on color, but I don't count these as part of the diagram proper, even though they may be visually associated.

3 - [back] - Which incidentally kills any use we might get out of svg-edit's or Inkscape's connection tools. Each of them assume a single connection between two shapes, anchored at each center-point. No idea if said connection can be directional or not, but we've already established that it won't do, so I'm not looking into it.

Wednesday, November 13, 2013

Toronto FBP Group -- Inaugural Meeting Videos

I've managed to upload the first two videos from the first Toronto FBP Group Meeting to Vimeo.

First, here's Paul Tarvydas giving a brief introduction to FBP. If you're a programmer that's never really heard of Flow Based Programming before, it's a pretty good start.

Second, here's J. Paul Morrison talking about the history of FBP. He's the guy who first discovered the idea, and formalized it while he was working at IBM in the early 1970s, and he talks a bit about what led to the insight. Near the end, he also demos a visual editor he's currently working on.

Sadly, because of the Vimeo basic account limitations, the JP Morrison video has a pretty crappy resolution, and my quick demo talk about a noflo webserver won't be uploaded until next week.

If you really just want the audio and slides, you can get the relevant files right here. You can probably get about 60% of the gist from the audio, but all three of us used some relevant visuals in our presentations. In the case of Paul T. and Paul M., a powerpoint presentation, and in my case some noflo code. I'll try to put up .torrent links for the HD versions of the videos when I figure out the logistics, and the third video will go up next Tuesday.

There were no videos from the second meetup[1], and there won't be any from today[2]. We're basically trying to figure out how to pay for them; one idea I'm looking into is getting a monthly kickstarter going to let the market decide if there's enough interest to produce videos for a given month. I'm not sure how much advance notice we'll need, or how far we can actually plan out talks, but I'll keep you posted.


Torrents are now up along with the audio and presentation materials. If you do end up downloading these, please seed as long as you can.

Thu, 14 Nov, 2013

The third video is now up here. In it, I briefly introduce noflo. If you'd like a half-way decent drinking game, take a shot whenever I say "actually".


1 - [back] - We basically had a giant whiteboarding session.

2 - [back] - We've got Paul T. talking about visual compilation, and Jo'sh talking about the Drakon editor.

Sunday, November 10, 2013

From The Archives - Popularity

EDITOR'S NOTE: This is a piece I wrote about three years ago, then sat down to copy-edit and polish up today. It was very focused on a few particular, topical anecdotes that my then-co-workers were throwing about. Having read over it equipped with three extra years of hindsight, I'm not sure I'd still be quite as vehement about my opinions as I was here, but don't disagree on any large points. Sun, 10 Nov, 2013

I'm writing this post because I've seen one too many comments of the sort "Well, if PHP is so shitty, then why is it so popular?". Typically this is the main claim in a rebuttal to "PHP is a shit language"[1], and the end result seems to be that a lot of people just sit back and think "Oh yeah, I guess it is popular. It must not be shit, but rugged." Substitute "manly", or "quirky" or similar if you like. As in, yeah, it has flaws, but they're endearing flaws. Like mysql_escape_string being the deprecated precursor to a different function named mysql_real_escape_string (itself also deprecated). And static functions having an odd interpretation of "this". And the complete list of deprecated functions being longer than my average blog post thanks to the fact that they never actually get removed.

I don't care that we've known how to do better language design for at least thirty years, PHP is Rugged©™, goddamit! And it's still more popular than whatever properly designed tool it is that you use!

How do technologies get popular?

This probably won't turn into an article where I mention Common Lisp[2], by the way. The idea of language popularity is almost completely irrelevant to what I'm discussing here; this could be a discussion of any sort of tool and how you'd go about picking one for a particular purpose. The source of popularity is obviously difficult to determine because relatively few things get there, so lets back off a little bit.

What does it mean for a tool to be popular?

Does it have to solve some specific problem better than other tools? Does it need to be better/cheaper/faster than solving that particular problem by hand? Does it need to be more fun or easier to use? Does it have better marketing/sales/promotions than the competition? Is it the first tool to solve a problem sufficiently well?

No. To all of the above.

A tool is popular when enough people have chosen it to perform a given task. Any of the above points contribute to a tool getting chosen, but for each, you can find a large number of counterexamples. Both tools that lacked it and became popular, and tools that had it but went nowhere. So no single element of that list of points is going to make or break you.

Lets look at it from the other side instead though.

What does it take for someone to choose a given tool?

That's a simpler question, but it should get us the same answer. If "popularity" is "being chosen by enough people" then figuring out "how do most people choose" should tell us "what it means to be popular".

A big reason to choose a tool[3] is that it'll get you a job. Again, this has nothing to do with language choice. Lots of people claim "I learned [x] to get a job", and [x] can be "Java" or "C#" with the same probability as "MS Word", "Photoshop", "Wordpress", "typing", "cooking" or "how to drive a bus". So one reason people choose is to get a job. Before we drill down to the next level, any other reasons?

For fun. I know about as many people who paint/design professionally as those who just do it on the weekend to relax[4], and I know plenty of people who just plain like to program. For fun. Like, on nights and weekends. Granted, not everyone works this way, and not every tool has this effect on people to the same degree[5], but it's still one possible reason.

Fitness of purpose maybe? Well, not in practice, no. Fitness of purpose is how you pick a specific class of tool. Which is to say, that's how you know you want a rotary cutter as opposed to a reciprocating saw or a pen and not a sable brush. It still doesn't tell you that you want a Dremel as opposed to a DeWalt, or a Pilot instead of a Bic[6], or ahem, a Lua instead of a PHP. It's also not as high a bar as people might think. I try to be objective about it, but from observation, most people tend to treat "fitness of purpose" as "what tool do I currently know how to use that could sort of be put to this use?" rather than "what is the most effective tool for the problem I'm solving?"

Popularity is the only other big reason I can think of that tools get chosen, but I don't want to recur just yet, so we'll leave that one alone. Back up a bit.

Tools are popular if they'll get you a job.

When will a tool get you a job? Well, when enough employers start putting it on their job listings. Until that point, it's not worth learning it just for that. Tools before that point mostly get adopters that come by because it's fun for them or the tiny minority that have performed a sufficient comparison and found that tool to be the best fit for them out of the ones they compared. In other words "I choose tools that will get me jobs" translates to "I choose tools that employers choose".

So how do employers choose tools?

Well, here, I can actually share some small amount of real-world data. Anecdotal, so take it with a grain of salt, but enough to form a theory. If anyone wants to try being the experimentalist on this one, be my guest. If you did it well enough, I'm sure it'd be publishable.

There are a few major points that impact on what an organization does in terms of technical tool choice[7].

The biggest one is "We'll keep using what we're using". Which is to say, if the previous project turned out to be successful, there will be a big push to use the same tools on the next one. Interestingly, this happens even if the success of the last project had everything to do with the team pulling constant overtime, and nothing at all to do with tool choice. The tools can be actively detrimental to the goal and still reap a rep-boost if the project succeeds. This doesn't really answer anything. How does a company choose their tools on the first project?

A few different ways. The tool choice for Project Number 1™© is contingent on[8]

  1. What do our developers know how to use?
  2. What do our vendors use?
  3. What do our clients want us to use?
  4. What's the industry standard?

The first one at least has some expert input, but works oddly. You don't get choice bias towards the "best"[9] tool, but rather the most popular. If Bleeb is "better" than Blub, but only two people on a team of ten know Bleeb whereas everyone knows Blub, then the team uses Blub[10]. In other words, no help there; this criteria will get you the popular language without requiring any level of quality or rigor in its design principles.

The second one is just plain odd, and before sitting back and observing, I would have sworn that it would be a really weak reason to use a tool. Companies seem to not care though; if a given preferred vendor uses tool [x] for task [y], then the company tends to use tool [x], even if it's ridiculously awkward to actually use. The vendor is also a company, so they use this same process for picking their tools, so substitute that back once we're done.

The third one is obvious, I hope, but it also boils down to "popularity" because very few clients know the problem space. Typically, they listen to the first/best sales people that talk to them. They're a force though; if your target client wants it on MSSQL and .NET, then it'll either be that or it won't be.

The fourth one is the previous answer on a macro scale; "We'll keep using what we're using (as an industry)". In other words, if there were lots of successful companies using tool [x], we'll use it too[11].

Now, we need to go deeper[12].

How does the first company in an industry pick their tools?

Regardless of any other decision factors, the answer is almost by definition "before they really know what problems those tools will have to solve", and I've already discussed that one pretty thoroughly. There are no clients, so they can't pick that way, there are no other companies or vendors so there is no industry standard. They might look at what similar industries do. Would they use the best possible tool for the job though? Well, no. They're likely to go the "What do our developers know how to use?" route, which we've already discussed above.

The big reasons teams[13] pick a given technology include

  • how easy is it to hire people that know how to use this? (easier the better)
  • do we have existing code that we'll need to interface with? (if so, weigh whatever we used on that project favorably)
  • have we used any technologies in the past for similar purposes to what we're doing here? (if so, weigh those favorably)

and the big one

  • if I choose this, and anything blows up, will I still be able to make the case to non-technical humans that it was the right choice? (if not, weigh it very unfavorably)

1, 2 and 3 mean that the more popular the language is, the more chance it has of getting entrenched[14]. 4 means the same, but this time "popular" means overall, not just within the tech community. A non-tech has heard the name "PHP" before, enough times to associate it with "the web" and "Facebook" and "Wordpress", but probably hasn't looked into it closely enough to catch complaints from developers[15].

The end result is that, in a sufficiently large company, it's safer to use a popular tool that's poor in the technical sense than it is to use an excellent tool no one's heard of. And that's also been discussed thoroughly, and this time it wasn't even by me. The decision is made purely on the basis of popularity once again.


We just bottomed out our recursion. Just in case you haven't been keeping score, literally every single level at which a tool can be selected is likely to be filled by the most popular tool in some context, and this popularity never requires, therefore never implies, anything other than popularity. One more time: at no point in the process of selecting a toolkit do most choosers even try to see whether it's shit or not. So I don't care how popular your steaming pile of imperative, counter-intuitive security-exploits-waiting-to-happen is; it's still shit.

Don't let me stop you from eating it, but I remember what it tastes like so I won't be joining you. Or shaking your hand afterwards.


1 - [back] - Which I happen to agree with, actually. If you're looking for details on what languages(Spoiler warning) I'd recommend learning, you're better off reading this instead.

2 - [back] - Except for that.

3 - [back] - Generally among the people I know at least, no judgment here.

4 - [back] - That's what I get for going to design school.

5 - [back] - Point of fact, only one person I know drives a bus for fun. It's been his obsession to work for the TTC since grade 8. I haven't heard from him in a while, but I still remember his room being full of papercraft Orion 3s, and I'm pretty sure he spent every internship opportunity he had on some streetcar route or another.

6 - [back] - The pen fanciers among you are probably ready to tell me that these are the worst possible examples; they're just the most popular common brand pens, not the really good stuff, where quality can make a difference. Really, I should have used foo and bar. You can go now, if you ponder that point hard enough, you pretty much got the gist of the article.

7 - [back] - I'm reigning it in a bit to software tools because that's what I have experience with, but this still seems like it might be a general trend; again, experimentalists welcome.

8 - [back] - In varying order, in my experience, but always on these things.

9 - [back] - Which I'm still quoting. In a book, that's called "foreshadowing". In a game or movie, it's called "setting up the sequel".

10 - [back] - And you get a varying amount of childish name-calling and dismissiveness towards Bleeb.

11 - [back] - Again, disregarding whether the tool had any effect at all on success.

12 - [back] - And at this point, all bets are off, I'm just theorizing, because I haven't observed the decision making process in an industry-defining company. That would be an interesting research project though, let me know if you've got one lined up.

13 - [back] - Not individual developers, but groups of corporate developers complete with leaders, technical or not, who are ultimately responsible to non-tech people further up the hierarchy.

14 - [back] - And note that both points are completely unrelated to how "good" a language is, and entirely dependent on how popular it is.

15 - [back] - Or to determine whether there's a lot of overlap between "good developers" and "developers who complain vocally about PHP".

Briefly, Custom Server Redux

So you already know I'm building a custom HTTP server for Deal. A question has arisen in my mind: how customized do I want it?

My initial assumption was that I'd just make it the usual general purpose HTTP server, with a few pieces focused on my end goal, but now that I'm waist deep in the guts of this thing, it occurs that I could take it further if I wanted to. For my current purposes, a relatively small subset of HTTP would do just fine. Here are the points I've noticed:

  • Deal never runs into the situation where both a GET and a POST parameter have the same name. As a result, I can do the Hunchentoot-standard thing and mangle parameters so that get-params and post-params are actually kept in the same associative structure. That simplifies the structure of both request objects and of handler functions, at the cost of differentiating between parameter types.
  • Because there's never an ambiguity between GET and POST parameters, I don't even really need to differentiate between GET and POST requests. That'll simplify the class tree of incoming requests. Specifically, it'll cap its depth at 1.
  • There are only two handlers that allow session-less requests, the rest require that the requester have a session[1], so what I could do is just make each request start up a session if a valid session token isn't passed in. That'll occasionally burn a few milliseconds of extra work[2], but it'll remove the need to assert session presence at the application level.
  • Because Deal is already targeting recent browsers[3], there's no need to support any older version of HTTP.
  • Finally, because there's only one handler that's going to need to handle extended connections, and the rest will always return immediately, I can more or less ignore request headers. In particular, I never care about Connection[4], Accept[5] or Cache-Control[6].

Acting on each of these assumptions is going to narrow the usefulness of the server, but also significantly simplify it. The optimum is eluding me, though I suspect it might be "as simple as possible" given my end goal for this particular project. Interestingly, even if I decide to do the simple thing for each of the choices outlined above, the end result will still probably be a useful general-purpose game server.

The only really funny part is that, now that I've thought it through, implementing this system as a websocket server[7] seems like it would yield an even simpler architecture. I'll save that one for later though. Step one: simple SSE+session based engine, make sure it works, then rip out its guts again and see what the other way looks like.


1 - [back] - And most of those require that the session be associated with a particular, already-instantiated game table, but optimizing for that seems like it would be going too far.

2 - [back] - When someone who just wants to watch rather than play first checks out a table.

3 - [back] - We're using quite a few features of HTML5 to make the whole thing playable.

4 - [back] - That one that I mentioned will be assumed keep-alive, while the rest assume the reverse.

5 - [back] - You're expecting text/html on the main request, text/event-stream on the subscription and application/json on the rest. If you can't handle those, you won't be able to play in any case.

6 - [back] - Everything other than the static file handlers always re-generates content; the static handlers don't need to change unless the files themselves have actually changed.

7 - [back] - Which multiple people have already recommended to me, so I'm not taking credit for that particular insight.

Wednesday, November 6, 2013

FBP in Common Lisp

Guess what!


I can finally tell you what I'm doing at work. A friend has suggested that I just not mention that there are things I still can't talk about, so I won't. In any case, all the interesting stuff is fair game. Apparently, I'm allowed to talk about it in much greater detail than I can write about it because a persistent record is still frightening to some humans. Progress, I guess.

We've been doing Flow Based Programming in Common Lisp.

Damn it feels good to finally get that off my chest. I have no idea how Yegge stopped blogging. I'm guessing he hasn't, but rather just stopped publishing the results. I've been writing about one thing for the past little while, and the number of ideas I need to discuss with the rubber duck at some point is staggering. If my output rate were zero, I would probably have a pretty tenuous grip on my sanity.

This is heading off topic. Once more, with feeling

Flow Based Programming in Common Lisp

I'm not sure what I think about it yet. Lets just be clear about that up-front. You'll find plenty of FBP True Believers on the appropriate Google group[1], but I am not one of those. The fact that I'm willing to throw a couple years behind the idea implies curiosity, nothing more.

A matter quite apart from the underlying structure is our implementation of it, and I am certain that it's over-complicated. Granted, based on all the others I've seen, ours is the least over-complicated, but still. That's something I'll aim to fix, with a personal project as a last resort, if I can't convince anyone else about it. But I digress. Again.

Here's why I'm curious. This is what a basic web server looks like in flow-based terms:

And here's what it looks like once you add SSE capability to it:

and finally, here's what it looks like when we add sessions into the mix

The above is by far the most useful set of images I've got for understanding what's actually going on behind the scenes of a page-view. I've worked through the principles in multiple languages and spent quite a bit of time thinking about it, but until I sat down to draw it out, it didn't feel like I really understood what needed to be done. You probably don't know the same languages I do, but the above is still likely intelligible to you. So that's why I'm curious.

Flow Based Programming vs. Functional Programming

Before I go, I want to tackle this, because several people I've talked to have gotten tripped up in the comparison. Including me. I ended up deleting a few lines from this post that said

The underlying problem for my lack of "wow" reaction might actually be my usual languages. I'm used to thinking about streams moving between inter-connected, lazy processors. That's the main way I conceptualize Haskell. In fact, if you squint just a bit, it's the way you can conceptualize most functional programs, pure or not. -Inaimathi

The difference is that functional programming focuses on partial conceptual separation, whereas FBP takes the isolation concept a few steps further by enforcing complete conceptual separation as well as complete temporal separation. Here's the accompanying thought experiment, just to clarify what I mean by that.

Suppose you were writing in a functional language and wrote the following:

def foo():
   baz("something else")

def bar(arg):

def baz(arg):

Firstly, notice that, while the functions are conceptually separate units[2], foo still has to know about bar and baz. That prevents total isolation. Yes, you can re-define bar and baz in-flight if your language is dynamic enough, but you need to have them both defined and you need to have them named bar and baz before foo can actually run.

Secondly, note that what's happening there is most likely a bunch of synchronous work. That is, when foo calls bar and baz, both calls complete and then foo returns the return value of baz[3]. If you wrote the equivalent in a pure-functional language, the actual work may happen in a different order than you see it written out, subject to what the compiler can prove about the behavior of the functions involved, but bar and baz will still complete before foo does. If they didn't, you'd get some unexpected behavior in any callers of foo.

Now, lets take a look at the apparently equivalent, Lisp-flavoured, FBP-style program.

(define-part foo ()
  (out! :out "something")
  (out! :log "something else"))

(define-part bar ()

(define-part baz ()

(define-container box 
    (:foo (foo) :bar (bar) :baz (baz))
  ((:foo :out) -> (:bar :in))
  ((:foo :log) -> (:baz :in))))

First, notice that foo doesn't know anything about bar or baz, or anything about the existence of either. At some point during its execution, it outputs two messages to some implementation-dependent output structure, but critically foo itself is not responsible for delivering those outputs to their consumers. That allows for total part-agnosticism; you really can shuffle parts around with pin-equivalent parts and have the result work. In functional or actors-based systems, you can almost do the same; the exception is that since each sender/caller has to name targets in some way, you need to change small chunks of code in places where functions/actors interoperate.

Second, note that there's nothing in this system about the timing of bar and baz. This omission includes the fact that both, either or neither may run before foo completes. In this model of the world, if bar takes a while to run, neither baz nor foo, nor any of their callers are prevented from further operation. The second critical difference is, essentially, that asynchronous operation is the norm.


1 - [back] - Currently mostly Javascripters thanks to the hype surrounding noflo.

2 - [back] - Assuming the programmer hasn't done something "clever" with global variables, gotos or comefroms.

3 - [back] - which happens to be the return value of doOtherStuff.