Saturday, January 28, 2012

The Lisp Shell Followup

So, I may have to backtrack on what I was saying earlier. Specifically, I called clisp a toy shell, and I called the machine I'm currently typing this on a toy machine. I did this because, having just installed it and spent a grand total of five minutes poking around, I assumed

  • it wouldn't run some programs properly
  • my scripts would now be useless
  • cd wouldn't work
  • I'd lose tab completion on files
  • there would be no gains to offset all the losses
  • it would be a pain in the ass to use a regular shell when I hit the limits of clisp

It turns out that most of those don't apply. I did actually lose tab-completion when working with files, but that's it. Pretty much every program that I want to run typically[1] works just as well from clisp as it does in bash, scripts run exactly the same as under a standard shell when you use run-shell-command, cd is actually a function defined in clisps' cl-user, and when I need to run a regular bash for whatever reason eshell can pickup the slack.

There's also a few non-obvious things I gain to offset losing filename tab completion.

First off, I get to define helper functions at my command line. One situation I've already found this useful in is copying files off my previous computer. It's a fairly specific situation, because I didn't want to sync a complete directory, but rather surgically copy over some 12 or 13 irregularly named files. That would have taken 12 or 13 separate scp calls. In regular shell, I'd have to do something like write a script for it. Having an actual language available let me pull out my first trick

> (defun cp-file (file-name) 
    (run-shell-command (format nil "scp inaimathi@other-machine:.emacs.d/~a .emacs.d/")))


> (cp-file "example.el")

This isn't specific to clisp, obviously. I assume that any language shell you use could pull the same trick. Still, having the ability to define helpers on the fly is something I occasionally wish I had[2].

Another thing that I imagine would work in any language shell, is an easier way of defining shell scripts. I wrote a little set of ui utilities a while ago, one of which is pack, a translator for various archive formats so that I can write pack foo rather than tar -xyzomgwtfbbq foo.tar.gz foo


require 'optparse'
require 'pp'
require 'fileutils'

archive_types = {
  "tar" => ["tar", "-cvf"],
  "tar.gz" => ["tar", "-zcvf"],
  "tgz" => ["tar", "-zcvf"],
  "tar.bz2" => ["tar", "-jcvf"],
  "zip" => ["zip"]

########## parsing inputs
options = { :type => "tar", :excluded => [".git", ".gitignore", "*~"] }
optparse = do|opts|
  opts.on("-e", "--exclude a,b,c", Array,
          "Specify things to ignore. Defaults to [#{options[:excluded].join ", "}]") do |e|
    options[:excluded] = e
  opts.on("-t", "--type FILE-TYPE",
          "Specify archive type to make. Defaults to '#{options[:type]}'. Supported types: #{archive_types.keys.join ", "}") do |t|
    options[:type] = t

ARGV.each do |target|
  if not archive_types[options[:type]]
    puts "Supported types are #{archive_types.keys.join ", "}"
  elsif options[:type] == "zip"
    exclude = options[:excluded].map{|d| ["-x", d]}.flatten
    exclude = options[:excluded].map{|d| ["--exclude", d]}.flatten
  fname = target.sub(/\/$/, "")
  args = archive_types[options[:type]] +
    [fname + "." + options[:type], fname] +

So that was necessary in bash, and because shell scripts can't easily share data, the companion script, unpack, had to define almost the exact same set of file-extension-to-command/option mappings[3]. If I'm using clisp, I could instead write

(defun pack (file-name &key (type tar) (exclude '(".git" ".gitignore" "*~"))) 
  (pack-file (make-instance type :file-name file-name :excluded exclude)))

(defmethod pack-file ((f tar.gz))
  (run-shell-command (format nil "tar -zcvf ~@[~{--exclude ~a~^~}~]~a" 
                             (excluded f) (file-name f))))

and be done with it[4]. This is a similar, but more extreme version of the previous point. Instead of writing shell-scripts, I can now write functions, macros or methods. These are smaller conceptual units and deal with inputs more easily, letting me focus on expressing what I want the script to do. In fact looking at language shells this way makes it obvious that things like optparse are just hacks to get around the way that scripts accept arguments.

The last cool thing is to do with the package management. I could be wrong about this, but I don't think the Lisp notion of in-package exists elsewhere. So I can define a package like

(defpackage :apt-get (:use :cl))

(defun install (&rest packages)
  (su-cmd "apt-get install ~{~(~a~)~^ ~}" packages))

(defun update () 
  (su-cmd "apt-get update"))

(defun search (search-string) 
  (cmd "apt-cache search '~a'" search-string))

where the cmds are defined as something like

(defmacro cmd (command &rest args)
    (if args `(format nil ,command ,@args) `command)))

(defmacro su-cmd (command &rest args)
    (format nil "su -c \"~a\""
            (if args `(format nil ,command ,@args) `command))))

The issue I'd have with defining these in, for example a Python shell, is that I'd then have a choice. I could either import the file and put up with typing out the name of the module at every invocation, or I could import install, update, search from and then hope that I don't have to define conflicting functions[5]. In a Lisp shell, I can define it and load it and then do (in-package :apt-get) when I need to do a series of commands relating to installing new modules.

Now all of these, clisp-exclusive or not, are small syntactic fixes that work around basic shell annoyances. To the point that you're probably asking yourself what the big deal is. It's basically the same reason that macros are awesome; they get rid of inconsistencies at the most basic levels of your code, and the increased simplicity you get that way has noticeable impacts further up the abstraction ladder. The sorts of things that look like minor annoyances can add up to some pretty hairy code, and cutting it off at the root often saves you more trouble than you'd think.

I'll admit that tab completion on file names is a pretty big thing to lose[6], but the things I outline above are mighty tempting productivity boosts to my shell. To the point that I'm fairly seriously debating switching over on my main machine. Between Emacs, StumpWM/Xmonad and Conkeror, it's not really as if someone else can productively use my laptop anyway. Adding an esoteric shell really doesn't seem like it would be a big negative at this point.


1 - [back] - Including fairly complex CLI stuff like wicd-curses, mplayer and rsync --progress

2 - [back] - And now, I do

3 - [back] - Except for compression rather than expansion

4 - [back] - Defining methods for each archive type, and the appropriate class, obviously

5 - [back] - Or import another module that defines new ones with the same names

6 - [back] - And I'm going to put a bit of research into not losing it

Wednesday, January 25, 2012

WebMote and "Open" software


I've been working on a tiny little utility to marginally improve my life in an almost insignificant way.

You've already heard the casual references to my "media PC", and you may have come to the conclusion that it's just a standard NAS setup, but no. It's actually just a regular computer hooked up to my TV through a VGA and audio wire[1]. About a week ago, the wireless keyboard I was using started chugging and finally gave out. Well, being that I mostly listen to music and watch ridiculous low-res videos in mplayer, why not just get a remote working? As it happens, I also had an old iPod touch lying around literally collecting dust since I stopped carrying a mobile music device with me. So I put the two together and hacked up a little remote control server for the computer. It basically just starts mplayer with -slave -idle, runs Hunchentoot out front and then passes along commands when I click on the various video links. I'm using it to watch some downloaded videos from the science network as we speak. The only gap in the interface is that I can't control the actual TV the same way yet (so I still need to get up to change channels or up the output volume)

The code is up at my github, as usual. I just noticed that there's no license file, though.

Just a second.

Ok. Compiling and inserting that took almost as long as writing the actual code. There you have it, in any case. I don't seriously recommend you use this program until I've ironed out one or two things, but feel free to if you like. I definitely enjoy being able to control my media center from whatever HTML client I happen to have at hand.

Next step: figuring out how to control the TV through wifi (though early research is not encouraging).

fingerquotes open

This was just sort of depressing.

I dunno, maybe it's not that big a deal to most people, but I'm depressed.

A good third to a half of my last weekend was spent researching ways of getting Open Genera up and running, only to find out that "Open" doesn't quite mean what I thought it did in this context. Granted, the system was built back in the 80s, so I guess the word may not have had the same connotation, but I still got confused.

It's bizarre, because I honestly don't get the point of a closed-source Lisp system. The whole point is that the entire machine is there, open to pokes and prods at its various sources and definitions. Saying that it's not being released openly or freely is just ... I dunno, off. The message is so fundamentally incongruous with the medium that it seemed to come at me entirely out of left field[2]. I guess that'll teach me to read the license first next time.

I still have my ersatz lisp machine, I guess. And I could do a bit of poking and hacking on Movitz if I really wanted to. That'll have to hold me.


1 - [back] - Incidentally, the mediaphiles among you should refrain from telling me that I should be using HDMI instead. Enough of my friends tell me that already, and I could give a shit. I mostly listen to music and watch almost ridiculously low-res videos; there's a separate DVD player for the occasional high-def media I watch.

2 - [back] - Enough to delete the relevant article from my archives (sorry, to the people who linked to it already) and never speak of it again[3].

3 - [back] - Except for here, obviously.

Friday, January 20, 2012

How Close can you get to a Lisp Machine?

Aside from the obvious, I mean.

Here's what I've been playing with for the past little while; using clisp instead of bash.

Those instructions still work surprisingly well, given that they were published all of 11 years ago. Here's what I did to replicate them

apt-get install clisp x-window-system
echo "/usr/bin/clisp" >> /etc/shells

I then installed quicklisp and ran (ql:add-to-init-file), then manually added the following to .clisprc:

(ql:quickload (list :cl-fad :cl-ppcre :trivial-shell))
(defun startx () (execute "/usr/bin/X11/xinit"))
and the following to my .xinitrc clisp -x "(progn (ql:quickload (list :clx :cl-ppcre :stumpwm)) (funcall (intern \"STUMPWM\" :stumpwm))"

After poking around for a little while and making sure everything worked approximately correctly, I ran chsh and set my shell to /usr/bin/clisp.

Performance-wise, it's surprisingly snappy given

a - what it's running on and

b - that there are at least 3 instances of clisp at work at any given time. It's a toy, but quite a quick and fun toy, actually.

Now, granted, the title is supposed to be taken with a grain of salt[1], but this still feels like it's approaching the target. What I've got running is a fully open system[2] that implements most of its components in Lisp (the shell is Clisp, the WM is Stump and the editor is Emacs). I suppose I could also throw in Closure[3] and Climacs as well, but I'm done playing for today.


1 - [back] - since I've never used an actual LISP Machine or even the Open Genera System. Incidentally, these links are here to remind me to look into it when I have a spare moment, so I'm not sure how much longer I'll be able to say "never used 'em".

2 - [back] - Except that it uses b43-fwcutter for the wireless card.

3 - [back] - As an aside, that meme-space is getting pretty crowded. To the point that I have to disambiguate in conversation. There's Clojure (the language), Clozure (the Common Lisp implementation) and Closure (the common-lisp based browser/html-parser)

Monday, January 2, 2012

Passing Notes

So here's how you can pass messages to each other without your parents[1] reading them.

Step 1: Get Some Friends and Computers together

I can't help you with this. You need some computers[2] and you need some friends to talk to, otherwise why would you be passing messages?

Step 2: Get GnuPG

This should be pretty simple. If you're on Debian/Ubuntu type

apt-get install gnupg

If you're on Windows, you'll need to install Cygwin with the gnupg package or gpg4win, if you're on OS X, you'll need to get GPGtools.

Step 3: Make some keys

Each of you should create a set of keys. Do that by typing

gpg --gen-key

GnuPG will then give you a menu that looks like this:

gpg (GnuPG) 1.4.10; Copyright (C) 2008 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Please select what kind of key you want:
   (1) RSA and RSA (default)
   (2) DSA and Elgamal
   (3) DSA (sign only)
   (4) RSA (sign only)

Accept the default by hitting enter[3]. It will then ask you

RSA keys may be between 1024 and 4096 bits long.
What keysize do you want? (2048)

Don't accept the default here, take the longest possible by typing 4096 and hitting enter. Next, gpg will ask when your key should expire. You can either specify days, weeks, months, years or accept the default (never expire). It's a balancing act; the more often you get new keys the more secure they will be against attacks. On the other hand, each time you change your key, you need to send the new one to each of your friends and they have to remember to use the new one for passing you notes. For now, just take the default (gpg will ask for confirmation, so type y and hit enter). Next, fill in your name, email and comment and confirm them by typing o and then enter.

Next, enter a pass phrase. It can have spaces and it can be fairly long. Pick something easy to remember; a favorite quote or maybe a few lines from a song you like.

This next part may seem a bit weird, but gpg actually needs you to do some other stuff on the computer. Anything you like, but actually do things. Compose an email, check your favorite news aggregator, do some drawing, hit your head on the keyboard, kick the mouse around for a while, whatever you need to do. It may occasionally ask you to keep at it with messages like

Not enough random bytes available.  Please do some other work to give
the OS a chance to collect more entropy! (Need 148 more bytes)

Eventually, it'll give you back control of your terminal, and at that point your key has been created. The hard part is done.

Step 4: Share Your Keys

Ok, get those friends that I told you to have ready.

Each of you should run the command

gpg --armor --output your-name.txt --export your-name

(your-name should be the name you typed in as part of your information in Step 3). That will create your-name.txt; a text file that should look something like

Version: GnuPG v1.4.10 (GNU/Linux)


It won't look exactly the same, yours will be different, but it will be about that wide and surrounded by the --- PUBLIC KEY BLOCK --- tags. That's your public key. Give it to your friends and get theirs. It doesn't matter how; either bust out the USB sticks or hop on a network and use scp or just paste that text to your Deviant-Linked-Book-Space page or personal site. Each of you should get the key text files and run

gpg --import your-friends-name.txt

for every key you have. Once that's done, check that you have all your friends' keys imported by typing gpg --list-keys and hitting enter.

Step 5: Send A Message

Type up your message in a text file (we'll call it message.txt, but it can be any type of file) and then run

gpg --output message.gpg --encrypt --armor --recipient your-friend --recipient your-other-friend --recipient and-so-on message.txt

Here's what a text file containing "something something dark side" looks like after it has been encrypted.

Version: GnuPG v1.4.10 (GNU/Linux)


You can now send this to everyone. Email, Facebook, reddit, your blog, a random comment section, anything goes. Only people with the keys you specified as --recipients will be able to decrypt it.

Step 6: Read A Message

If you get a block like the above from one of your friends, save it to a text file called encrypted.gpg[4] and run

gpg --output decrypted.txt --decrypt encrypted.gpg

And decrypted.txt will now contain the note your friend passed you in plain text that you can read. Of course, anyone can read it now, so if it discusses anything you really want to keep secret, you should run shred on it once you've read it.

Boss Fight

Use the comment section of Joe Armstrong's Blog to send a message to a friend that only the two of you can read. If you're feeling adventurous, send it to all of the friends you got together in Step 1

Bonus Stage

Sending your friends messages that your parents can't read is nice, but if they catch you, they can still ground you until you decrypt it for them[5]. Ideally, you'd send your friends messages that your parents or teachers wouldn't even know are messages. To do that, you need a second program called steghide. On Debian GNU/Linux, just type apt-get install steghide as root. Cygwin supports this package too, but I have no idea how to get it on OS X so you Mac users are on your own here.

Get an image, like this one

encrypt your message, and then run

steghide embed -ef message.gpg -cf not-sure-if-secret.jpg

You will be asked for a pass phrase, leave it blank for now, but you should really agree to one with your friends and use it to protect these. You can now send your friends that image via email or imgur without raising suspicions (unless your parents are reading this blog). When your friends get it, they can run

steghide extract -sf not-sure-if-secret.jpg

(and enter the pass phrase if you set one) to get your encrypted message and then decrypt that to read what you sent them. You can use steghide to hide files in images or music that you can then send without raising suspicions.

Secret Boss Fight

Send your mailing list one of those stupid Fw: Fw: Fw: Fw: joke mails, but embed a secret message to one of your friends in the first picture. Ideally, that friend should be one of the people you send the email, otherwise you're needlessly spamming.

Secret Boss Fight -- Stage 2

Find a forum/reddit thread somewhere and carry on a steganographic, encrypted conversation about hipster ninjas with two or three of your friends.

Secret Boss Fight -- Final Stage

Sneak onto your friends' computer while they're in the washroom and change their desktop background to a steganographic message. Chuckle about it constantly. When asked "What's so funny", laugh maniacally and run out of the room with your stuff.


1 - [back] - Or teachers, or political enemies, or competitors, what-have-you

2 - [back] - If you don't have access to your own computer, you can use a usb stick to run Ubuntu live, or (better yet) TAILS and merely plug it into whatever computer you do actually have access to.

3 - [back] - If all of your friends can be trusted to keep secrets, you might opt to go with a symmetric cypher instead. The process of sending and receiving messages is more or less the same, but there's only one key that the entire group shares rather than there being two keys per person (a public and a private). The advantage is that there's less to keep track of. The downside is that if anyone finds out your key, all your notes can be cracked rather than just those sent to the person who let their key get compromised. I won't go through this method, but if you follow that link to the GnuPG manual, you should be able to figure it out.

4 - [back] - It doesn't actually need to be called that, just an example.

5 - [back] - They probably won't just ask for your key because they'd have no idea what to do with it.