MACROS

This is the </> macro from the almighty-html library. It produces HTML.

(ql:quickload "almighty-html")
(in-package :almighty-html)
(</>
 (div :id "card-container"
   (a :href "/path/to/book-1"
     (div :class "card"
       (h2 :class "card-title" "Book 1")
       (p :class "card-body" "This is book 1 of the series of books.")))))
;  => 

Result<div id="card-container">
  <a href="/path/to/book-1">
    <div class="card">
      <h2 class="card-title">Book 1</h2>
      <p class="card-body">This is book 1 of the series of books.</p>
    </div>
  </a>
</div>

This is a SQL query produced with the sxql library using several macros.

(ql:quickload "sxql")
(in-package :sxql)
(yield (select :*
         (from :users)
         (where (:= :age 30))))

Result"SELECT * FROM users WHERE (age = ?)", (30)

This is example code for Coalton (https://www.youtube.com/watch?v=of92m4XNgrM), a new programming language built on top of Common Lisp. It's "just a macro".

(declare create-account (AccountName -> Balance -> BankM (BankResult Account)))
(define (create-account name initial-balance)
  "Adds an account to the BankState and return the created account."
  (run-resultT
   (do
    (accounts <- (lift (lift get)))
    (ResultT
     (match (get-account name accounts)
       ((Err _) (pure (Ok Unit)))
       ((Ok _) (pure (Err (AccountAlreadyExists name))))))
     (let unvalidated-account = (Account name initial-balance))
     (account <- (ResultT (check-account-is-valid unvalidated-account)))
     (ResultT (set-account account)))))

What we see is that macros are code that is capable of either generating, or fully implementing, entirely other languages. Of course, they can produce Lisp code, as the defun and loop macros do.

The most important thing to know about macros is that they are compiled or expanded before they are evaluated. Because of this expansion step, macros are the only tool programmers can use to modify the evaluation of the code it expands into.

In Lisp, the first symbol in a list is evaluated as a function name, and the rest of the values are evaluated from left to right. Before the function name is looked up, the parameters are evaluated.

(hello my name is "Micah")

The error you get here is about a non-existent my variable--hello will be evaluated after the arguments, and my is the first argument.

But what about defun? The whole point is that the first argument is suppose to be bound to the definition that follows.

(defun hello (name)
  (print name))

hello isn't evaluated as a symbol containing a value–it's used as the name for the code that follows.

And that's how we know defun is a macro. In fact, anything called "special operators" are actually just macros.

We can even view what the defun macro expands to right in Emacs. With the cursor over the part that says defun hello, type SPC m m (macrostep-expand). If you are using SBCL as I suggested at the beginning of the book, you should see something like this:

(progn
 (eval-when (:compile-toplevel) (sb-c:%compiler-defun 'hello t nil nil))
 (sb-impl::%defun 'hello
                  (sb-int:named-lambda hello
                      (name)
                    (block hello (print name)))))

Because macros can expand into more macros, you may be able to expand more macros after the first expansion.

(defun hello (names)
  (loop :for name :in names
        :do (format t "Hello, ~a.~&" name)))
;; expands into
(progn
 (eval-when (:compile-toplevel) (sb-c:%compiler-defun 'hello t nil nil))
 (sb-impl::%defun 'hello
                  (sb-int:named-lambda hello
                      (names)
                    (block hello
                      (loop :for name :in names
                            :do (format t "Hello, ~a.~&" name))))))
;; which further expands into...
(progn
 (eval-when (:compile-toplevel) (sb-c:%compiler-defun 'hello t nil nil))
 (sb-impl::%defun 'hello
                  (sb-int:named-lambda hello
                      (names)
                    (block hello
                      (block nil
                        (let ((name nil)
                              (#:l548
                               (sb-kernel:the* (list :use-annotations t
 :source-form names) names)))
                          (declare (ignorable #:l548)
                                   (ignorable name))
                          (tagbody
                           sb-loop::next-loop
                            (when (endp #:l548) (go sb-loop::end-loop))
                            (sb-loop::loop-desetq name (car #:l548))
                            (sb-loop::loop-desetq #:l548 (cdr #:l548))
                            (format t "Hello, ~a.~&" name)
                            (go sb-loop::next-loop)
                           sb-loop::end-loop)))))))
;; and further...
(progn
 (eval-when (:compile-toplevel) (sb-c:%compiler-defun 'hello t nil nil))
 (sb-impl::%defun 'hello
                  (sb-int:named-lambda hello
                      (names)
                    (block hello
                      (block nil
                        (let ((name nil)
                              (#:l551
                               (sb-kernel:the* (list :use-annotations t
 :source-form names) names)))
                          (declare (ignorable #:l551)
                                   (ignorable name))
                          (tagbody
                           sb-loop::next-loop
                            (if (endp #:l551)
                                (go sb-loop::end-loop))
                            (setq name (car #:l551))
                            (setq #:l551 (cdr #:l551))
                            (format t "Hello, ~a.~&" name)
                            (go sb-loop::next-loop)
                           sb-loop::end-loop)))))))

After calling macroexpand-step the first time, you can expand one more step with e.

You can undo one expansion at a time with u or completely revert all expansions everywhere with q.

As you can see, macros significantly reduce the amount of visual noise in the code and make it easier to understand. But after looking at the code it expands into, you might wonder how the heck you even get started writing a macro at all?

To begin defining a macro, first you need to do some wishful thinking.

• Write the desired call.
• Write the desired expansion.
• Construct macro parameter list from call.

For example, let's say you want to define a when-let macro that combines let and when into one form. First, you begin with the macro call you want to have:

(when-let (user (get-user id))
  (welcome-message user)
  (redirect-to-dashboard user))

Then, you write the code you want it to expand into:

(let ((user (get-user id)))
  (when user
    (welcome-message user)
    (redirect-to-dashboard user)))

Finally, you write the macro definition.

(defmacro when-let ((var expression) &body body)
  `(let ((,var ,expression))
     (when ,var
       ,@body)))

Here you see a backquote in the form `(let ...). Backquotes are like normal single-quotes–they are an abbreviation of quote, which returns the literal representation of some code.

(quote (+ 2 2))

Result(+ 2 2)

'(+ 2 2)

Result(+ 2 2)

`(+ 2 2)

Result(+ 2 2)

Notice that those symbols aren't interpreted. They and the list they are included in are returned as-is.

Backquotes have an important feature: you can tell the interpreter to actually evaluate some of parts of the code.

To tell the interpreter to evaluate parts inside a backquoted form, type , before the part you want evaluated.

(let ((a 2)
      (b 3))
  `((+ ,a ,b) is ,(+ a b)))

Result((+ 2 3) IS 5)

There are some rules you have to follow when using backquotes:

• A comma must follow a backquote.
• A comma can't follow another comma.

For example:

(let ((a 2)
      (b 3))
  `((+ ,a ,b) is ,(+ ,a ,b)))

ResultComma not inside a backquote.
  Stream: #<dynamic-extent STRING-INPUT-STREAM (unavailable) from "(let ((*...">
   [Condition of type SB-INT:SIMPLE-READER-ERROR]

If we add the commas to the second a and b, since the outer (+ ...) form is already preceded with a comma, we get that error.

In the when-let macro definition, we also saw ,@body. The ,@ "splices" the form into the backquoted code–removing the outermost parentheses.

(let ((a 5)
      (b '(1 2 3))
      (c 9))
  `(+ ,a ,@b ,c))

Result(+ 5 1 2 3 9)

,@ is commonly used with macros that take an arbitrary number of arguments and passing those arguments along to another form that also takes an arbitrary number of arguments. That's what the body parameter for when-let is.

After having the call and expansion prepared, you need to look at the parts that are common between them, and then make those parameters in the defmacro lambda-list.

For example, look at user. It's in the expansion code three times:

(let ((user ...))
 (... user
   (... user))

and in the macro call twice:

(when-let (user ...)
  (... user))

Which means a parameter in the when-let definition needs to correspond to this variable.

(defmacro when-let ((var ...)...)
  ...)

…and any instance of var in the body of the macro definition needs to be preceded with a comma.

(defmacro when-let ((var ...)...)
  `(let ((,var ...))
     (when ,var
       ...)))

The same goes for (get-user id). It appears once in the expansion and once in the macro call.

(let ((... (get-user id)))
 (... ...
   (... ...)))

(when-let (... (get-user id))
  ...)

So it gets a spot in the parameter list of the macro:

(defmacro when-let ((... expression) ...)
  ...)

And where expression appears in the macro body, it's preceded with a comma.

(defmacro when-let ((... expression) ...)
  `(let ((... ,expression))
     ...))

The calls to (welcome-message user) and (redirect-to-dashboard user) are actually in a space where we expect any arbitrary number of arbitrary forms. For that, we use a &body parameter and splice it in with ,@.

(defmacro when-let ((... ...) &body body)
  `(let ((... ...))
     (when ...
       ,@body)))

Altogether:

(defmacro when-let ((var expression) &body body)
  `(let ((,var ,expression))
     (when ,var
       ,@body)))

One thing to keep in mind with macros is that if you make a macro, use the macro to write some code and compile that code, if you redefine the macro, you also need to recompile the other code. Otherwise, that code is using the old macro expansion and not the new one.

As I said before, the general rule is to be conservative with your creation of macros. But there are times when you just gotta use the facilities that a macro provides.

There are certain situations under which you absolutely can't use macros. Macros can't be used as data in the same way functions can.

(mapcar #'my-macro data)

Only functions can be passed to higher-order functions like mapcar, reduce, remove-if-not, etc.

And of course, always keep in mind that if you are trying to develop a DSL, you might be overcomplicating things. If you want to collaborate with others, and you want to write DSLs, you better make them good.

Macros can be tough to debug. That's partially because macros are generally harder to read because you have to keep the expansion code and expander code in your head at the same time.

But macros are also more difficult to debug because they are subject to a unique class of bugs called variable capture. The subject of macro hygiene is primarily concerned with avoiding this class of bugs.

The simplest example of a macro that's vulnerable to variable capture is a macro that returns a free variable.

(defmacro might-get-captured1 ()
  '(+ x 1))

Why is this macro vulnerable? Because it doesn't return '(+ x 1). It expands into (+ x 1), which is then evaluated at runtime.

(might-get-captured1)

ResultThe variable X is unbound.
   [Condition of type UNBOUND-VARIABLE]

(let ((x 1))
  (might-get-captured1))

Result2

Try expanding the code with SPC m m (macrostep-expand) to verify.

Compare to a function:

(defun cant-capture-me ()
  '(+ x 1))

Calling it:

(cant-capture-me)

Result(+ X 1)

(let ((x 100))
  (cant-capture-me))

Result(+ X 1)

So it's important to remember that quotes/backquotes in functions return literal representations, whereas macros use quotes/backquotes as templates for expanding into code-to-be-evaluated.

In addition to macros with free variables, macros that bind multiple variables within one let are also vulnerable to capture. Consider the following macro:

(defmacro might-get-captured2 (variable)
  `(let ((x 1)
         (,variable 0))
     (values x ,variable)))

If we call it like this:

(let ((y 5))
  (might-get-captured2 y))

What do you expect the return values to be?

1, 0

Expanded:

(let ((y 5))
  (let ((x 1) (y 0))
    (values x y)))

Or what about this wombo combo:

(let ((x 5))
  (might-get-captured2 x))

ResultExecution of a form compiled with errors.
Form:
  (LET ((X 1) (X 0))
  (VALUES X X))
Compile-time error:
  The variable X occurs more than once in the LET.
   [Condition of type SB-INT:COMPILED-PROGRAM-ERROR]

It returns an error because x occurs twice in the expanded let form.

(let ((x 5))
  (let ((x 1) (x 0))
    (values x x)))

You don't know what symbols could be in a &body parameter of a macro. A symbol used in the macro may get used in the &body as well.

(defmacro might-get-captured3 (&body body)
  `(let ((x 5))
     (+ x ,@body)))

Because the body expands into an environment where x is bound to the value 5, if the symbol x appears in the body, it will likely be overwritten by the macro's let context.

(let ((x 70)
      (y 50))
  (might-get-captured3
    (+ x y)))

Result60

So how do you avoid variable capture?

Like I said, there is much more to say about macros, but this is not the book for and I'm not the guy to talk about macros. Refer to the RESOURCES chapter to find out where to learn more about macros.