Monday, February 14, 2011

Quicklisp, Linode, Hacking in the pejorative and other notes...

This week has been kind of a mixed bag for me; I've been thinking about a bunch of stuff, but not enough about each thing to justify a whole blog post. So here's the result of my mental mastication. It's not pretty, but perhaps it will be nourishing.

On serving data

I've been administering my own server for the last little while. First for DieCast (which is on hold for the moment 'cause the server was needed for something else) then to host some ASDF files (which are actually back up now; you should be able to asdf-install:install 'cl-css or 'formlets without serious problems), and now for a couple of websites I'm doing work on. The experience has taught me three things.

1. Common Lisp webapp deployment sucked balls before Quicklisp

My first nginx+Hunchentoot setup took hours. Some of this was for lack of familiarity with the apps because my second deployment took hours (fewer of them though). Which is an excellent improvement, but still not good in the absolute sense. The main problem was actually setting up Hunchentoot; it has many dependencies (many of which have several recursive dependencies of their own), each of which need to be downloaded and evaluated separately, each of which has at least one compiler warning, and one of which usually fails to download. The worst deployment after the first involved a key ASDF-package hosting site going down. That meant I had to go out and download + install + load all of Hunchentoots' dependencies recursively by hand in order to get them running. Sadly, lacking encyclopedic knowledge of Hunchentoot, this meant I had to try (asdf-install:install 'hunchentoot), wait for it to error out, get the piece it errored on, install it and try again. Once the server was up it was awesome, but getting it to that state was a pain in the ass the likes of which I'm having trouble analogizing properly. Quicklisp does it in 10 minutes, while simultaneously massaging my aching shoulders. I really hope zach doesn't start charging, because I get the feeling many Common Lispers would end up owing him their house (he welcomes donations, of course).

2. System setup sucked balls before Linode

I used to use Cirrus Hosting. And actually still do at work; we had them before I came in, and they're pretty good so I don't have a burning need to switch over, but we'll see what's possible once our subscription is up. Basically, I was used to a VPS being more or less just a regular server, except virtual. You have to spend a bit of time installing the distro, reboot, and install. It turns out that if you put thought into the process, a lot of that startup time can be done away with behind the scenes. Linode has put a lot of thought into the process. Going from one linux distro to another takes something like 5 minutes. I found this out bouncing between different linuxes (linuxen? linuces?); the process was initiated and that's typically a cue for sandwiches, but I didn't have enough time. Needless to say, it was a pleasant surprise the first time a deployment from bare metal to a running Common Lisp server took less than half an hour.

3. Break-in attempts are surprisingly common

If I'm to believe my auth.log, a concerted effort at hacking is made by some jackass roughly every two days. Needless to say, my iptables have been modified. It's different IPs, but always the same MO; they try some random common usernames, fail and go away. Apparently it's escaped their notice that I switched to RSA keys and disabled password/PAM authentication. To be fair, checking the logs, it seems that before the change to key-based auth, the situation regularly looked like

Feb 10 07:40:04 Invalid user abc from
Feb 10 07:40:07 Invalid user abc123 from
Feb 10 07:40:10 Invalid user benjamin from
Feb 10 07:40:12 Invalid user lstiburekz from
Feb 10 07:40:15 Invalid user kent from
Feb 10 07:40:18 Invalid user jabber from
Feb 10 07:40:20 Invalid user andres from
Feb 10 07:40:23 Invalid user dovecot from
Feb 10 07:40:26 Invalid user magda from
Feb 10 07:40:28 Invalid user alex from
Feb 10 07:40:31 Invalid user stefan from
Feb 10 07:40:34 Invalid user stefano from
Feb 10 07:40:36 Invalid user cristi from
Feb 10 07:40:39 Invalid user claudi from
Feb 10 07:40:42 Invalid user sarah from
Feb 10 07:40:44 Invalid user smokeping from
Feb 10 07:40:47 Invalid user fetchmail from
Feb 10 07:40:50 Invalid user backinfo from
Feb 10 07:40:53 Invalid user umberto from
Feb 10 07:40:55 Invalid user mauro from
Feb 10 07:40:58 Invalid user jana from
Feb 10 07:41:01 Invalid user adriano from
Feb 10 07:41:03 Invalid user xenie from
Feb 10 07:41:06 Invalid user lea from
Feb 10 07:41:09 Invalid user joule from
Feb 10 07:41:11 Invalid user Debian-exim from
Feb 10 07:41:14 Invalid user unbunutu from
Feb 10 07:41:17 Invalid user cacti from
Feb 10 07:41:19 Invalid user polycom from
Feb 10 07:41:23 Invalid user payala from
Feb 10 07:41:26 Invalid user nicola from
Feb 10 07:41:28 Invalid user melo from
Feb 10 07:41:31 Invalid user axfrdns from
Feb 10 07:41:34 Invalid user tinydns from
Feb 10 07:41:36 Invalid user dnslog from
Feb 10 07:41:39 Invalid user dnscache from
Feb 10 07:41:42 Invalid user qmails from
Feb 10 07:41:45 Invalid user qmailr from
Feb 10 07:41:47 Invalid user qmailq from
Feb 10 07:41:50 Invalid user qmailp from
Feb 10 07:41:53 Invalid user qmaill from
Feb 10 07:41:55 Invalid user qmaild from
Feb 10 07:41:58 Invalid user alias from
Feb 10 07:42:01 Invalid user vpopmail from
Feb 10 07:42:03 Invalid user ldap from
Feb 10 07:42:06 Invalid user gica from
Feb 10 07:42:09 Invalid user sympa from
Feb 10 07:42:11 Invalid user snort from
Feb 10 07:42:14 Invalid user hsqldb from
Feb 10 07:42:17 Invalid user member from
Feb 10 07:42:20 Invalid user chizai from
Feb 10 07:42:22 Invalid user yakuji from
Feb 10 07:42:25 Invalid user gijyutsu from
Feb 10 07:42:28 Invalid user kaihatsu from
Feb 10 07:42:30 Invalid user iwafune from
Feb 10 07:42:33 Invalid user oomiya from
Feb 10 07:42:36 Invalid user seizou from
Feb 10 07:42:38 Invalid user gyoumu from
Feb 10 07:42:41 Invalid user boueki from
Feb 10 07:42:44 Invalid user eigyou from
Feb 10 07:42:46 Invalid user soumu from
Feb 10 07:42:49 Invalid user hanaco_admin from
Feb 10 07:42:52 Invalid user hanaco from
Feb 10 07:42:54 Invalid user system from
Feb 10 07:42:57 Invalid user tenshin from
Feb 10 07:43:00 Invalid user avahi from
Feb 10 07:43:02 Invalid user beaglidx from
Feb 10 07:43:05 Invalid user wwwuser from
Feb 10 07:43:08 Invalid user savona from
Feb 10 07:43:10 Invalid user trthaber from
Feb 10 07:43:13 Invalid user proftpd from
Feb 10 07:43:16 Invalid user bind from
Feb 10 07:43:19 Invalid user wwwrun from
Feb 10 07:43:21 Invalid user ales from

whereas I now merely get the occasional

Feb 12 11:53:12 Invalid user oracle from
Feb 12 11:53:13 Invalid user test from
Feb 12 12:03:59 Invalid user apache from
Feb 12 20:16:59 Invalid user postgres from

So it helps, but the regularity of these attacks is still surprising to me. It seems a bit odd that a script would keep trying if it got the refused (publickey) error, so I'm forced to conclude that there are one or two spammers out there manually looking for servers they can break into. That's ... odd. And I can't shake this picture of a 12 year old in some spamming sweatshop somewhere failing to break into my server and missing his quota as a result.

On starting up

So remember back in the prehistoric ninties, when the likes of this strange creature walked the earth? When the Playstation first introduced the idea of CD-based games to the console market, a friend of mine flatly said he preferred his SNES. When questioned about it, his reasoning boiled down to one word.


For the youth who never experienced this; a Super Nintendo had no loading screens anywhere. You put the cartridge in, hit the power button, and it would go straight to the logo screen. While in-game, moving between areas was instantaneous. It seems like most people working in the consumer electronics industry today have either forgotten that instant usage is really good, or they never thought so to begin with. The latest generation of consoles has loading screens friggin everywhere. A different friend of mine purchased a TV recently that has a 30 second boot cycle, and comes with a network connection for the purpose of getting firmware updates. A fucking teevee. It's hilarious that between the TV boot time and the console boot time (and I won't even mention the install time on the console because it's really unfair), it actually takes longer to start a game of whatever in his living room than it does on my computer. Weird, because I thought the whole point of consoles was that they were special-purpose devices specifically designed to run games. Entertainment isn't the end of this trend though; my phone now also takes about a minute to start up (which is fair I guess, since it basically is a computer now, complete with a flavor of Linux and a web browser). Finally, my parents recently renovated their kitchen and procured for it a, I shit you not, dishwasher that needs to boot before it starts pulling in water.

At what point did this start happening? When the hell did the decision get made in the bowels of Sony corporate HQ that it was ok for my display to have a configuration cycle? If this is where the future of TVs is going, I may very well have already bought my last non-monitor display. But beyond entertainment, my greater concern is the trend of ephemeralization (as elaborated by Graham to mean "...the increasing tendency of physical machinery to be replaced by what we would now call software.") combined with the new human habit of sticking computers into things means that we are likely to soon have shoes, lip-balm and kitchen cutlery that come with their own fabulously designed and meticulously polished loading screens.

Somehow, I'm not enraptured by this prospect.

On data moving

It's come to my attention that the Canadian government has recently had a nontrivial (and ongoing) tussle with the CRTC and the major Canadian ISPs about whether or not they should be allowed to charge arms and legs for data overages. That last link was actually to the Open Media site, which is organizing a petition against the CRTCs move. If you're in Canada, you should probably sign it. My position is basically that I don't care, because the way I use the internet, 40GB is essentially unlimited. I'm not a netflix user (though I'm constantly told I should be) I don't torrent the games like the kids these days, and downloading Linux packages is a joke if you're running the minimal system I've got over here. The single largest component I install is haskell-platform, which takes something like 600Mb. With an M. Even with my fiancee being perhaps the worlds' biggest YouTube makeup video fiend, we've never actually approached the limit of our plan. My interest in this fight is purely in the interest of a theoretically unfettered future; one where data is as free as it could possibly be, and that world includes no limits on how much it's allowed to move per month (incidentally, that's also why I frown when I see things like this happening; freedom of information includes the right for said information to exist). So I'm against the CRTC here, but seemingly not for the same reason as anyone in a 100 km radius of me.

Friday, February 4, 2011

Heart Ruby

I really do have to update that header more frequently. It's been a good year and a half since I did anything other than some light make scripting in Python, JavaScript may as well be jQuery as far as my recent use of it is concerned, and I haven't done much more than some very lightweight playing in Erlang. Most of my time at work has been getting spent ass-deep in PHP which wasn't very pleasant even back when it was one of two languages I knew. The rest of it is tilted heavily towards the lisps (Common Lisp, Elisp, Scheme, in that order), and I'm still trying to get my head around Haskell through some tutorial and semi-involved practice. The practice will actually increase soon; a friend of mine wants some help with his website, and he's convinced that having it written in a lesser-known language will make it less likely to get hacked (he's had some security troubles lately). I tried to explain that this isn't actually how cryptography works, but he's having none of it. His instructions were "I don't care what you use, as long as it's not PHP". Score.

The last piece up there is Ruby, which I've had an odd relationship with. I tried out Rails a while back, but didn't like the amount of magic involved (and the various "convention vs. configuration"/security exploit stories I keep hearing about through friends aren't exactly tempting me back). I also tried out some Windows automation back when "Windows" was a thing I used for work rather than just for playing 10 year old video games. We also run Redmine at the office, so I've had to spend a very little bit of time making cosmetic UI changes. The point is, I've yet to write more than maybe 200 lines of Ruby in one sitting, but I still like it. It's clean somehow. Simple. In a way that Python never felt, even though the syntactic whitespace forces more visual consistency onto it. Despite my low line-count, ruby-full is still firmly wedged in the ## languages section of my installation shell-script, and the only reason my installation shell-script isn't itself written in Ruby is that the language isn't bundled with Debian.

I'm musing on this now, because I recently received a reminder of how beautiful it can be for simple scripting purposes. I had a problem with my XFCE4 setup. Actually, not a problem, just something that wasn't going quite as smoothly as it might have. I use multiple monitors on each of my multiple machines, you see. My desktop has two, my laptops share an external, and my work machine travels with me so it actually has two different monitors to interface with depending on where it is. The key is, no matter where I am, the situation is the same; I just want my monitors arranged left to right, each at the highest possible resolution. XFCE doesn't seem to have an option for that, so my initial approach was just to manually check xrandr output and type out the appropriate combination of --output, --mode and --right-of to get it working. It dawned on me the other day that this is pretty inefficient given how consistent the pattern is, and since I occasionally profess to know how to program, I should be able to do something about it. The problem is that step one of the process is parsing the output from a shell command, which surprisingly few languages care to do. Luckily, Ruby is one of them. My initial pass worked, but it was ugly (and I won't inflict it upon you here). After consulting codereview.SE, it was whittled down to


def xrandr_pairs (xrandr_output)
## Returns [[<display name>, <max-resolution>] ...]
  display = /^(\S+)/
  option = /^\s+(\S+)/

def xrandr_string (x_pairs)
## Takes [[<display name>, <max-resolution>] ...] and returns an xrandr command string
  cmd = "xrandr --output #{x_pairs[0][0]} --mode #{x_pairs[0][1]}"
  args = x_pairs.each_cons(2).map do |(previous_output, previous_mode), (output, mode)|
      "--output #{output} --mode #{mode} --right-of #{previous_output}"
  [cmd, *args].join(" ")

exec xrandr_string(xrandr_pairs(`xrandr`))

which is pretty beautiful, as far as I'm concerned.

It's elegant for shell-scripting for two reasons;

First, Ruby has a wide range of options for calling the shell. exec seems tailor-made for the purpose above (it replaces the current Ruby process with a call to the command you pass it), spawn is useful if you want to do things in parallel and ` delimits a special string type that gets executed synchronously as a shell command and returns that commands' output.

Second, any string can be a template. This includes special strings like regexes and backticks, which is why you can compose larger regular expressions from simpler pieces as in xrandr_pairs above. You can stitch #{ } into any string you like, and the contents can be any expression, not necessarily a string. A minor, but important part is that I didn't have to call a function in order to make a string a template (there's no call to printf or format), and the contents are inlined (I'm doing "Foo #{bar} #{baz}" as opposed to "Foo #{1} #{2}" bar baz) which makes the result that much more obvious. Neither would matter much proportionally in a big project, but when I'm working on a 16 line script, I'll knock out every bit of cognitive overhead I can.

That's why I still use it. I never liked Ruby for big stuff, or even medium stuff, but I instantly reach for it to do anything under 100 lines that needs to talk to the shell.

Wednesday, February 2, 2011

Old Machines

I'm not attached to things.

My grandfather was; whenever we'd do some carpentry or light construction, he'd insist that we save old screws and nails we found. It's always seemed weird to me because even in the old country, nails and screws are things you can get at the hardware store for $20 per 5 lbs, so there never seemed to be much of a point in saving old ones. Whenever I'd point this out, he'd reply "You never know when a nail will come in handy", and proceed to stash stray nails in variously sized glass jars. Ok, yes, this was across the atlantic, so what he actually said was "Nikad neznaš kad ćeš trebati čavlića", but you get the point. He had a musty attic full of clothes he wore decades ago, books he read once, games he played when he was a kid, travelling cases that had only been used once, and a thousand other treasures that I never got to see, but that he likely also could have discarded with no disadvantages. I'm sure there was some socio-political reason for this, but I'm digressing.

I'm not attached to things, and I can say this as someone who has observed humans attached to things. But I still feel a bit perturbed when someone throws a computer away. I'm sure my descendants will have the same reaction I had to the old nails. "Grandpa, you can buy computers for 2¢ per core, they come with ten free petabytes of memory and a lifetime supply of storage on the Googazon servers. Why are you keeping those old things?" I can already feel the urge to tell those smarmy cunts to get the hell off my lan...

For the moment though, I'm slightly less than insane for putting old machines to use. Last week I stumbled upon an HP Pavillion circa 1998 (well, I assume it was thereabouts since it still had Windows 98 installed). With a roaring 566 MHz processor, a truly awe-inspiring 64 MB of SDRAM, and a massive 15GB hard drive. I've been meaning to set up a backup server for my setup here anyway. I still had to spend some money on a couple of hard drives (about $50 each for 160GB IDE drives, I had one lying around, but the rest of my spares are all SATA. Could have saved some money by getting a couple of adapters instead, but I didn't think soon enough. I'll get some of these if the drives ever fail) and an ethernet card ($4.99).

The first thing I had to do was remove a few unwanted items.

As comitted as I am to reusing old machines, I've still got to admit that there's very little use today for a phone modem or floppy drive. They were fairly easy to remove; just a couple of mounting screws internally. What was slightly tougher was this plastic faceplate that covered the area next to the front-facing USB port; it was held in by a small, springy metallic assembly that I had to lever out with a swiss army knife (I wanted another hard drive to go there).

Next up, I ripped out the 15GB drive it came with, popped in one of my 160GB ones and threw in that ethernet card for good measure. Then I installed Ubuntu Server 10.10. It could have been Debian, but I wanted to try out the latest Ubuntu release, and there are some things I'd like to do with pacpl that don't seem to work on my debian machine. The tradeoff is that Emacs seems to misbehave out of the box on Ubuntu, but this isn't exactly going to be a development machine so that's ok. The only stuff that went on was SSH server, GIT and Ruby (my language of choice for quick and dirty scripting).

Once the system was installed, the CD drive could come out (not about to install MS Word or any such nonsense; any other software that goes on this machine will come in through the network). That turned out to be easier said than done though; it was secured by screws on both sides, so I had to completely disassemble the box to get at it.

The hard drive destined for the position was going to rattle in a slot that size, and while I don't plan to race this machine around the block or anything, it's probably better to be safe. A couple of drive brackets made sure it would stay in place. Shop around if you plan on buying some, incidentally, I just put that link up because it was the first I found; there were actually a couple of braces lying around from my last case so I didn't need to order any. It also seems like you could improvise a set if you didn't feel like buying them.

With everything hooked up, it was time to boot back into the machine.

That incongruent looking mesh plate covering the top drive is a spare from the same case that had the extra brackets. And yes, I named the machine "orphan". It seemed appropriate. Here's ls /dev, showing the new drives (still haven't formatted them, that'll be for next weekend).

And that's it. I dropped it into a little wheel assembly that's been going unused since I got that mammoth tower for my main machine. It gives it a somewhat R2-D2 feel (this may be the start of an art project).

I'll put together some scripts to copy out key directories from my other machines and that'll be that. I guess I could also use it as a full-out NAS (ok, I technically am, but you know what I mean) or streaming server, but I'm not sure how far those 566MHz and 64MB of RAM are going to stretch. In any case, even with the slightly higher price/GB I had to pay for IDE drives, converting this old machine was much cheaper than shelling out for a pre-built.

The Microsoft Arc keyboard came in quite handy with this project. It's fairly ergonomic, the arrow oddity isn't as annoying as it seems it should be, and the transmitter is easy enough to move around. It's definitely a step up from wrangling USB cables from my main machine about three feet to the side. My only complaint is that it friggin devours batteries, compelled like some primal beast, always growling for more. That's easy enough to solve, I guess, just remove the batteries when it's not in use, but that's a small annoyance on an otherwise perfect spare keyboard.

Tuesday, February 1, 2011

Confusion of Ideas

On two occasions I have been asked, 'Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out?' I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question. - Charles Babbage Babbages' response was sufficiently condescending in the finest grumpy-IT-guy tradition (or perhaps this was just a reflection in his journal and he merely rolled his eyes at the time, boiling with internal rage). Holding the implicit assumption that, at least while operating a computer, a technician must forsake their humanity and act with perfect precision. The quote usually gets trucked out to ridicule someone who doesn't understand how computers work. You're supposed to be thinking of the CS manager who downloads and runs every .exe attachment, then wonders aloud (perhaps in your direction) why the damn machine doesn't work. "How the hell", we are expected to reflect, "would a computer getting the wrong inputs still produce the correct output? It's ridiculous!" Well the notion wasn't ridiculous. It was just ahead of its time by a few hundred years.