Lisp, Agility, Readability

I recently had the opportunity to witness a curious dialogue on the mailing list for a local user group I participate in. The original poster, a public devotee and evangelist for Agile development practices, posted an announcement to the group for an upcoming Lisp workshop. A responder immediately threw down the gauntlet – How can you possibly be advocating Lisp as a development language, when Agile is so much about code readability? Doesn't Lisp encourage some of the most unreadable code on the planet? Doesn't Lisp stand for “Lots of Irritating Superfluous Parentheses?” Doesn't Lisp ... ?

I thought about this challenge for a minute, and had an immediate sense of both the rightness and wrongness of it. After all, haven't we all beaten our heads against the wall for days trying to debug some Rails code, only to realize that the answer was buried deep in some “magic” meta-programming hook? Or spent the afternoon hunting down the documentation for some obscure method that's throwing an annoying runtime exception, only to discover that our framework kindly generated it for us at compile time based on some esoteric configuration parameter? Lisp programs seem to represent the pinnacle of this kind of self-generating, self-manipulating metacode, and they can be damned hard to wrap your head around.

At the same time, isn't it hard to beat that sublime feeling of joy you experience when you write that minimal amount of declarative code, and everything Just Works? When you can alter some piece of metacode at the global level to effect an entire architectural change, without touching a single line of code throughout your entire application? Surely there is something deeply right about these kinds of abstractions that allow us to communicate at the level of meaning, as opposed to mechanics – to separate the why of a piece of software from the how.

It struck me then that we operate under two very different senses of the word “readability,” often slipping and sliding back and forth between the two without realizing it. To caricature them down to their essence, allow me to call them mechanical readability, and semantic or intentional readability. Mechanical readability is achieved when a piece of code clearly communicates what it does, whereas semantic or intentional readability is achieved when a piece of code clearly communicates what it means. And there's the rub – clearly these two concerns are constraining us constantly, tugging at each other from opposite directions, nearly always forcing us to sacrifice one for the other in any given instance. What's the answer? Can they be harmonized into some greater concern that preserves them both?

I don't think they can be harmonized exactly, but I do think there is a bit of sound earned wisdom that applies here – a jewel in the crown of the software developer's craft. Without trying to be too scientific or dogmatic about it, it might go something like this:

• When addressing lower layers of abstraction, strive for mechanical readability.
• When addressing higher layers of abstraction, strive for semantic readability.
• Within a given layer of abstraction, strive to decouple the mechanical and intentional concerns as much as possible.

So where does that leave us as far as Lisp's alleged agility? Well, in truth, Lisp actually does address mechanical concerns far better than you might think. But its ability to express the semantic level of software is where it clearly shines. The temptation is to become obsessed with the semantic approach, to the point of obfuscation of the plain, simple mechanical tasks your software needs to perform. This is possibly the greatest temptation programmers face when given powerful meta-programming facilities. There's nothing quite so frustrating as having to work within forced abstractions.

However, don't assume a more “straightforward” mechanical style will clarify your code either. The reason your DSL doesn't flow well enough, or map onto your domain well enough, might not be that it's too abstract. It might just be the wrong abstraction. The semantic realm is a very conceptual, philosophical sort of beast. You won't write better abstractions by learning more about technological wizardry, but by practicing the dark art of simple, clear thinking.

Ultimately, it's the right ideas that set really great software apart.