1.1 loop Problems, Loopless Solutions

loop suffers from several inherent problems that make of it a totally inappropriate construct that should never even have existed. Any attempt at redesigning loop to fix the flaws at the core of its design would destroy its very essence, therefore the whole concept must be scrapped. (This bold stance is justified below.)

Unfortunately, Common Lisp doesn't provide appropriate alternatives to loop for many of its common usage scenarios. Loopless provides utilities that fill the small but significant looping-related gaps in the language, eliminating any perceived need for loop.

Loopless provides highly intuitive and conservative extensions to Common Lisp, such as mapcar-like functions for processing of alists and plists (among other structures). Taken individually, each such utility may not look like much, but together they remove most of the guesswork out of converting (mentally or textually) a loop to another looping construct by providing an unambiguously best alternative in most cases (or deferring to Common Lisp if it already has a suitable solution).

In the rest of this section, I will expose some of the problems which afflict loop to its very core (I'm not the first to do this) and then explain why they matter and how Loopless eliminates all of them.

– [Sloppy start] –

(Some of the rest of this section, until “Sloppy end”, is not as clear as it could be, I rushed it out to meet my arbitrary self-imposed deadline of December 1st for version 1.0. My apologies.)

LOOP's full generality is seldom needed and using simpler, more focussed constructs when appropriate results in less conceptual complexity, in my opinion. I believe that using LOOP less results in appreciating the rest of the language more.

Because of its monolithic nature and its tons of features and resulting high conceptual complexity, it's really hard to learn loop and all its intricacies. You have to learn a bunch of loop-specific syntax and semantics before you can even get started using it. All this knowledge doesn't leverage widespread approaches found in the rest of Common Lisp, in fact it needlessly competes with them in many cases. In contrast, because of its similarity with the rest of what the standard already provides, Loopless builds on and reinforces standard Common Lisp approaches. Loopless is also very small compared to loop.

The simplicity of Loopless is something that beginners and masters alike can appreciate.

I complain a lot about how loop is “too powerful” for its own sake, however, even some of the more complex scenarios which loop seems especially well-suited for can be implemented conveniently without it.

loop is oftentimes totally overkill for the situation at hand. You often end up reusing the same idioms over and over again.

loop is monolithic. This really is the central issue and a source or enabler of most of its problems. There is no other construct in Common Lisp that has even a passible resemblance to it (except maybe format?), thus it often feels, to me at least, out of place.

One of the greatest strengths of Common Lisp is the ability to create domain-specific languages, and loop is often cited as an example of that capability. While I'm a huge fan of DSLs (it's my favorite secret weapon in programming), I think loop is the perfect example of what can go wrong with this approach if one is not careful.

To make matters worse, due to its “all-encompassing” nature, loop often provides solutions redundant with other simpler, more focussed constructs in the rest of the language. Take the example of mapcar. You might start with code like this (let's assume the list is dynamically constructed somewhere else):

     (mapcar (lambda (element)
               (list element element))
             '(a b c))
     ⇒ ((A A) (B B) (C C))

So far, so good. But now, let's say you want to add a number at the start of the list, based on the position of the element in the list. You could just stick with mapcar and write some manual binding and stepping, like this:

     (mapcar (let ((i 0))
     	  (lambda (element)
     	    (prog1 (list i element element)
     	      (incf i))))
             '(a b c))
     ⇒ ((0 A A) (1 B B) (2 C C))

That isn't too bad, but if you try pushing this approach to more complex scenarios, you'll find that it's really inconvenient, stylistically questionable and error-prone in the general case.

You might decide that you really like loop's “for-as-arithmetic” stepping after all, so whenever you find that you need arithmetic stepping, you rewrite whatever looping construct you were using into a loop.

However, it's really annoying to rewrite a whole piece of code into another style (loop) just to gain one feature (“for-as-arithmetic”). To avoid that problem, you might eventually just use loop all the time, and then you end up with code similar to this everywhere:

     (loop for element in '(a b c)
           collect (list element element))

What's the point of using a ridiculously “powerful” construct if you then use it for simplistic scenarios where another operator was built into Common Lisp just for this purpose??

Loopless provides for*, which lets you use “for-as-arithmetic” and “for as equals” in conjunction with any looping construct whatsoever. The above example with the manual incf would simply be rewritten like this (don't worry about the “mysterious” step*, for* is the only unconservative Loopless utility, and then it's not very complicated either):

     (mapcar (for* ((i from 0))
               (lambda (element)
                 (step*)
                 (list i element element)))
             '(a b c))
     ⇒ ((0 A A) (1 B B) (2 C C))

That's really a recurring pattern with loop. It's so “all-encompassing” that many of its potential uses are redundant with other operators (or combination of operators) that Common Lisp provides. You have two “worlds” with few interactions between them: loop, and the rest of Common Lisp.

One of loop's problems is that it's not extensible. Some alternatives to loop which share its core design, such as iterate, do provide for extensibility. While all else being equal, I'd rather have extensibility available rather than not, the issue here is that the only thing that makes one wish loop was extensible is that it's monolithic in the first place! Loopless proves that there is no need for such a construct.

But, clearly, it's better to have a set of small, simple, focussed utilities that you can mix and match with the rest of the language, as exemplified by almost every Common Lisp operator save for loop and perhaps a few others (do?) than an extensible walled-garden of a monolithic construct.

Similar operations should look similar.

loop doesn't have enough parentheses. This causes no end of formatting problems, among other things. For many years, loop just wouldn't format properly in Slime, and when it did, it would invariably soon get broken by an update. Relatively recently, most of the problems seem to finally have been fixed, but the simple fact that it took so much time and effort to get it right is a testament to the peculiarity and shall I say, inappropriateness of loop's syntax when compared with the rest of Common Lisp. I was still stumbling upon some mysterious loop formatting problems from time to time when I stopped using it.

– [Sloppy end] –

By the way, I do realize and acknowledge that because of history, inertia and apathy, loop will continue to remain in widespread use for the foreseeable future, but I hope someday its fate will be similar to COBOL's: still in use in legacy codebases but no longer used in new projects (literally or for all practical purposes). There has been much whining and dissenting about loop, and I hope things can move forward now that there's a really incredibly simple and practical, viable alternative to loop.

Given the fact Loopless just reuses concepts already introduced by Common Lisp and therefore integrates seemlessly with it, I'm hoping some of its utilities will someday be incorporated into whatever standard gets created after Common Lisp, and then why not deprecate loop! To this end, I release this whole library in the public domain to promote maximum possible adoption. If you don't want to adopt Loopless wholesale, you can import any subset you deem useful into your own libraries without licensing problems. Another reason to release this into the public domain is I felt nobody should “own” such low-innovation utilities, what with the wholesale copying of mapcar and dolist and all...

For what it's worth, I use Loopless in virtually all my libraries, and thanks to it I easily got rid of loop almost completely in all my codebases, painlessly. In my experience, it's possible to convert big batches of loops to Loopless style without even having to understand the surrounding code. It's an almost mechanical syntax translation process. I still have a few legacy instances of loop here and there in stale code but they'll get converted whenever I revisit/update that code.

I never stumbled on a loop I couldn't rewrite in a better or at least as good way with Common Lisp plus Loopless. I'm sure it wouldn't be so hard to deliberately construct a contrived, highly unrealistic example of a loop that couldn't satisfactorily be rewritten Loopless style, but that would be missing the point. However, if you do stumble upon a loop instance you believe Common Lisp + Loopless can't handle well, please do let me know via the mailing list or my personal email address! I'll see if I can figure out how to rewrite it or if Loopless really has a blind spot in need of attention.

Lastly, I'll mention Loopless is comprehensively documented!

(The source code itself is deliberately sparsely documented so as to avoid clutter and a resulting drop in my productivity. Moreover, having separate documentation like this allows for a much more thorough treatment.)