Will's Willow Wisp

Programming is a tricky. Hardware capability has increased, so have the size and complexity of our programs. A quick find -E . -regex '.*(c|cc|java|lisp|cs)' | xargs wc in my current project directories shows I have a just over a million lines of code to wrangle. The late Edsger Dijkstra once said, “If we wish to count lines of code, we should not regard them as lines produced but as lines spent.” A lot of spending has been going on. How can we spend less, save more, and put what we do have to better use?

I can’t say I have an answer to those questions, but I do know the challenges I face time and time again. I may have mentioned this previously, but I’m in the business of building interfaces. Some are graphical, some aren’t. Even with the ones that are, I don’t spend much time where the graphics are: I’m busy on the back side of the screen wiring at the source level. When I want to test things, that’s the only time I see any dancing polygons.

Loopy Debugging

To do this, test that is, I close the back panel (except for some break points sticking out), start the engine, rev it up, put it into gear, and drive around until I hit one of the break points. Then I get out to figure out just what’s happing, push the machine a few steps forward: everything looks okay. So, I get back in the cab, step on the gas. Snap, crackle, segv—looks like I continued too soon. I restart the machine and run it to this place again. If I’m lucky, I get a convenient stack trace showing exactly where things went to pieces. It won’t show me what could have caused them to go wrong: that’s for me to infer from forty levels of stack trace and the values of a few hundred variables. I get out, open it up, tinker, add a printf or two, and run the engine again. This is known as the edit-compile-debug loop. In practice, the loop is usually more elaborate: edit-compile-linkerFailure-editMake-compile-linkerFailure-editMakeAgain-compile-debug-realizeTestCodeHasBug-edit-google-compile-curse loop.

It’s doesn’t always go that way. On rare occasions, I have the privilege of writing in Lisp. Here the edit-compile-debug loop is tightened into the read-eval-print loop. Type a line, it’s read, evaluated, and the result is printed. Debug one function at a time. Keep a transcript and call it a unit test. While dot whoring the other day (searching the dot for opinions, insights, and other dribble), I found a this remark, “I’ve been reading up on Lisp and have suddenly become disappointed in the entire programming world – right here there’s a language with a featureset that it has taken other languages decades to catch up to.” And I bet he doesn’t even know how to go about implementing a read-eval-print loop: (loop (print (eval (read)))). “Lisp, not as clumsy or random as Java. An elegant language for a more civilized age.”

Having a REPL doesn’t solve all problems. In fact, it only solves one problem: how to test code in function sized bites. For larger components, components with concurrency and complex io, REP doesn’t cut it. More flexible tools are required. Don’t let the button nose and mouse ears fool you, SmallTalk does some nice things that we’re now seeing in other tools. Suppose we’re Squeaking along one day when all of a sudden, Popup!, a ominous stack trace appears. Usually, we just dismiss these beasts, wondering why they bother popping up in the first place. Sometimes we’re bold enough to click on some sort of inappropriately labeled “details” button. Squeak doesn’t have a “details” button; instead, “debug” lets you jump straight into the debugger. From here, you can sniff round until your heart’s content. Nothing special about that. You can make any changes you want, then click “Restart” to unwind the stack and begin again. This already goes a little farther than some other fix and continue systems. However, just having fix and continue isn’t going to do you any good unless you know what to fix. The quickest way to find out is to run a few tests. And this is exactly what the Squeak debugger lets you do. It gives you a REPL at every level of the stack trace. Just type an expression, hit command+return, and it will be evaluated in context. Pretty slick.

Debugging is all well and good for mucking through muddy details. But how does one get a bird’s eye view of what’s going on? How do you do it in a general, comprehensive way that lets you highlight important details or even find out what the important details are?

Debugging Debugging

Asking all these questions begs the question, “Why do we debug in the first place?” I debug to make sure my code does what I think it should. I also debug to figure out how other people’s code work. But isn’t reading the source enough to know what’s going on? Apparently not, and why not? For starters, implementation details can distract from what’s really going on. One promise of object oriented programming is being able to hide implementations behind interfaces. Keep the immediate code clean and delegate the details to someone else. Taken too far, endless chains of delegation make object oriented programming into a structured method for writing spaghetti code—all talk and no action.

Another thing that makes source hard to understand is context dependancy. What happens when you call a function can depend on any number of things. Two calls at different times could produce different results. Some make out functional programming as the art of limiting effects. Limited effects makes for very clean, concise code. Eliminating effects makes one wonder how to use functional programming for time sensitive reactive systems.

A further challenge is the shear size of large code repositories. There’s no good way of seeing the whole system at once. Structured programming organizes control flow within a procedure. Object oriented programming organizes state and procedures into a convenient container. But higher level organization requires something else. Inversion of Control is a method for managing dependencies between components, and Aspects let you implement features that cross object boundaries. “One (simplified but useful) way of thinking about the techniques [of] Aspect Oriented Programming is that they put the power of a good debugger into your programming language. ... You need only see a couple examples to realize how powerful and useful this can be. You can then treat some of the rest of the ideas of Aspect Oriented Programming as trying to tame these powerful constructs into something that won’t frequently shoot you in the foot (much like Structured Programming tamed the wild goto statement).”

The Will is the Power

This little trek into debugging shows some of the things that are vital in building a development environment. To make programs you need to be able to get inside and work on the program. The source provides one crude interface to how the program works. When you edit the source, your tell the program what it should do in advance. Often, you aren’t going to tell it the right thing. To discover what’s gone wrong, you might use a debugger, a crude tool to observing the state and control flow of the program. Debuggers become less crude when you can use them in conjunction with editors. As you the edit-compile-debug loop is tightened into a read-eval-print loop the distinctions between program, debugger, and editor become hazy. If we tighten further, the distinction disappears. Running, editing, and debugging become a single, integrated activity. It’s like opening up a clock, seeing how it works, and adjusting it without needing to stop it. A system where you can side step the normal interface, and look at it from a different perspective provides a meta-interface. For meta to be workable, I have a few ideas about what the interface needs to be like:

Disabling Meta. It should be easy to step outside of the normal flow of a program. However, this capability is not liability free. The meta-view gives you access to some things that are better left untouched. The ability to arbitrarily modify your system means the ability to arbitrarily ruin things. Having full meta-access is like running your system as root. It’s not secure for trusted entities, let alone potential abuse. Any meta enabled system needs a good security model: one that gives you the capabilities you want without the responsibilities you don’t.

Back to the Future: Meta-time. Meta includes having power over the control flow of your system. You need to be able to slow processes down, speed them up, even run them backwards. It’s a kind of uber-undo/redo that need to come for free as part of the system.

Responsive Modelessness. Meta-control means you can pause or escape from any operation. “I want to not have to ever see an hourglass, spinning beachball, or any blue bar of waiting.” A computation may take time, but it shouldn’t take your time.

Transparency. Not all meta-interfaces are created equal. A good meta-interface is like a polished spyglass. Both the big picture and the minute details need to be represented.

Those are a few points. I have a whole cart of them. Programming is a rich and multi-varied experience. So much so, that language is a necessary tool for being able to express yourself. Meta-programming, likewise, requires a language up to the task. There’s quite a few languages in the world and some are fairly well suited for the task. Making a good clean meta-interface takes a good clean language to write it in. It should be easy to read, easy to write, and easy to run. It needs to be small, friendly, and powerful. If you have the will, it should have the power.

What’s in a Name

Will’s the name of the language with which the meta-interface, Willow, will be built. I call the meta-interface Willow because it will live and grow in a manner not entirely dissimilar to the way a tree does. Will and Willow need to be easily embeddable in any system that I happen to work on. For this task, I’ll use Wisp a stripped version of Will on which the rest of the system can be bootstrapped. That’s the vision.

From now on, my entries will focus on special topics relating to whatever facet of the system I’ve been working on that week.