To LOOP or not to LOOP

Almost every form in Common Lisp is a list you can read from the inside out. loop isn't. Its arguments are a run of clauses that read almost like English, :for x :in list :collect x, and the macro parses them with a grammar of its own, set out in section 6.1 of the standard.

None of that costs anything when the program runs. Expand a loop and you get ordinary Lisp: a block, a let for the variables, and a tagbody that jumps back to the top with go. The cost is paid by whoever reads the code, and whether it's worth paying depends on the loop.

Here is a small job. From a list of students and their scores, get the names of everyone who scored 90 or more, and the class average.

(defparameter *scores*
  '(("Ada" . 94) ("Grace" . 88)
    ("Alan" . 97) ("Barbara" . 72)))

(loop :for (name . score) :in *scores*
      :when (>= score 90)
        :collect name :into honours
      :sum score :into total
      :count t :into students
      :finally (return (values honours (/ total students))))

("Ada" "Alan")
351/4

That's one pass over the list, three running results and a destructured entry, all in one form. Without loop, the same work needs three variables that you declare and update yourself:

(let ((honours '()) (total 0) (students 0))
  (dolist (entry *scores*)
    (destructuring-bind (name . score) entry
      (when (>= score 90) (push name honours))
      (incf total score)
      (incf students)))
  (values (nreverse honours) (/ total students)))

("Ada" "Alan")
351/4

That version isn't wrong, but its intent is buried in bookkeeping: the push needs an nreverse at the end, and the counter is kept by hand. Several results gathered in one pass is the job loop was made for.

Now change the job: total the scores of the honours students only. The obvious edit is to indent a :sum under the :when.

(loop :for (name . score) :in *scores*
      :when (>= score 90)
        :collect name :into honours
        :sum score :into honours-total
      :finally (return (values honours honours-total)))

("Ada" "Alan")
351

The answer should be 191. It's 351, the total for the whole class. A :when controls exactly one clause. The indentation is only a suggestion, and loop ignores it. To make the :sum conditional as well, you have to chain it with :and:

(loop :for (name . score) :in *scores*
      :when (>= score 90)
        :collect name :into honours
        :and :sum score :into honours-total
      :finally (return (values honours honours-total)))

("Ada" "Alan")
191

Nothing warned about the first version. It compiled, ran and returned a believable number. That's the real cost of loop: its structure lives in keywords, not parentheses, so neither the compiler nor your editor can see it. Structured editing moves whole forms, and a clause isn't a form.

The second trap involves closures. Each function below should remember a different x:

(mapcar #'funcall
        (loop :for x :in '(1 2 3)
              :collect (lambda () x)))

(3 3 3)

On SBCL, loop updates a single variable on every pass instead of binding a new one, so all three closures share it and see its last value. The same code written with dolist returns (1 2 3) on SBCL, and a mapcar always does, because each call to the function gets its own binding. To capture the variable inside loop, rebind it: (let ((x x)) (lambda () x)).

When loop is the wrong tool, it's usually because a simpler one says the same thing in fewer words. For one transformation, one filter or one total, the sequence functions read better, because their names state the intent:

(mapcar #'car *scores*)
;; => ("Ada" "Grace" "Alan" "Barbara")

(reduce #'+ *scores* :key #'cdr)
;; => 351

(every (lambda (entry) (>= (cdr entry) 70)) *scores*)
;; => T, everyone passed

For side effects with no result, dolist and dotimes are shorter and say so. And if you want everything loop does with Lisp's own syntax, there's ITERATE, a library on Quicklisp. Each clause is a form in parentheses, so a when has a body you can see:

(iter (for (name . score) in *scores*)
      (when (>= score 90)
        (collect name into honours)
        (sum score into honours-total))
      (finally (return (values honours honours-total))))

("Ada" "Alan")
191

The trap from the last section can't happen here. Both clauses are inside the when because the parentheses put them there, and your editor can move, wrap or delete each one as a unit. The price is a dependency, and one more library every reader of your code has to know.

Ask what the loop produces, then pick the smallest tool that says it.

The last two rows are where loop is worth its grammar. Everywhere else it's a longer way of saying something Common Lisp already has a word for.