Chapter 20 - Macro Etiquette

Macros in Lisp are much more capable than macros in other programming languages. Rather than just providing a simple shorthand notation, Lisp macros give you the capability to truly extend the language. In this chapter we'll learn about the program transforming capabilities of macros as we see how to properly construct macros to solve a wide variety of problems.

Macros are programs that generate programs

Mention macros to most programmers, perhaps even you, and the first image that comes to mind is string substitution -- you use a macro to glue together a few parameters in a new way, maybe with a bit of compile-time decision making thrown in. And because of the typical (in languages other than Lisp) disparity between the macro language and the programming language, the difficulty of writing a macro increases much faster than its complexity.

Lisp macros are Lisp programs that generate other Lisp programs. The generated Lisp code has a fully-parenthesized notation, as does the macro that generates the code. In the simplest case, a macro substitutes forms within a template, clearly establishing a visual correspondence between the generating code and the generated code. Complex macros can use the full power of the Lisp language to generate code according to the macro parameters; often a template form is wrapped in code that constructs appropriate subforms, but even this approach is just a typical use pattern and not a requirement (or restriction) of the Lisp macro facility.

In the following sections, we'll examine the mechanism by which the Lisp system translates code generated by a macro, then we'll see several increasingly sophisticated examples of macros. We'll finish this chapter with a comparison of macros versus the use of inline function declarations.

Close up: how macros work

You define a macro with a DEFMACRO form, like this:

(defmacro name (arguments ...)
  body)

DEFMACRO is like DEFUN, but instead of returning values, the body of the DEFMACRO returns a Lisp form. (As we'll see shortly, there's a very simple way to generate this form with selected subforms replaced by parameters from the macro call or computed by the macro's program.)

Your program "calls" a macro the same way it calls a function, but the behavior is quite different. First, none of the macro's parameters are evaluated. Ever. Macro parameters are bound literally to the corresponding arguments in the macro definition. If you pass (* 7 (+ 3 2)) to a macro, the argument in the body of the macro definition is bound to the literal list (* 7 (+ 3 2)), and not the value 35.

Next, the macro expander is invoked, receiving all of the actual parameters bound to their corresponding arguments as named by the DEFMACRO form. The macro expander is just the body of the DEFMACRO form, which is just Lisp code; the only catch is that the Lisp system expects the macro expander to return a Lisp form.

The Lisp system then evaluates whatever form the macro expander returns. If the returned form is a macro, it gets expanded. Otherwise, the form is evaluated by the rules we first learned in Chapter 3, Lesson 2.

The preceding paragraph is conceptually correct. However, a Lisp implementation may expand macros at different times. A macro could be expanded just once, when your program is compiled. Or it could be expanded on first use as your program runs, and the expansion could be cached for subsequent reuse. Or the macro could be expanded every time it's used. A properly written macro will behave the same under all of these implementations.

In Chapter 21 we'll expand upon some of the things you can express with argument lists.

Backquote looks like a substitution template

The simplest way to generate a form in the body of your macro expander is to use the backquote (`) reader macro. This behaves like the quote (') reader macro, except for when a comma (,) appears in the backquoted form.

A comma is only permitted in a backquoted form. If you use a comma in a quoted form, Lisp will signal an error when it reads the form.

Like quote, backquote suppresses evaluation. But a comma within a backquoted form "unsuppresses" evaluation for just the following subform.

? `(The sum of 17 and 83 is ,(+ 17 83))
(THE SUM OF 17 AND 83 IS 100)

Compare the preceding example, which used backquote, with the similar form using quote (and omitting the comma).

? '(The sum of 17 and 83 is (+ 17 83))
(THE SUM OF 17 AND 83 IS (+ 17 83))

You can probably imagine how backquote and comma provide a template with substitution capabilities. This is just what we need for our macro expander. Here are a couple of simple examples.

; Define the macro 
? (defmacro swap (a b) ; NOTE: This is a restricted version of ROTATEF 
    `(let ((temp ,a))
       (setf ,a ,b)
       (setf ,b temp)))
SWAP

; First invocation 
? (let ((x 3)
        (y 7))
    (swap x y) ; macro call 
    (list x y))
(7 3)
; Let's see the form generated by SWAP: 
? (pprint (macroexpand-1 '(swap x y)))

(LET ((TEMP X))
  (SETF X Y)
  (SETF Y TEMP))

; Second invocation 
? (let ((c (cons 2 9))) ; (2 . 9) 
    (swap (car c) (cdr c))
    c)
(9 . 2)
; And the expansion of its macro call 
? (pprint (macroexpand-1 '(swap (car c) (cdr c))))

(LET ((TEMP (CAR C)))
  (SETF (CAR C) (CDR C))
  (SETF (CDR C) TEMP))

; Here's the second invocation again, "macroexpanded" by hand. 
? (let ((c (cons 2 9)))
    (LET ((TEMP (CAR C)))
      (SETF (CAR C) (CDR C))
      (SETF (CDR C) TEMP))
    c)
(9 . 2)

(PPRINT (MACROEXPAND-1 'macro-call)) is a very handy tool to see what form your macro expander generates. (Don't worry if the output from your Lisp system looks exactly as shown here; there may be some differences in layout.)

As you look at these examples, the important things to note are that:

  1. the macro arguments receive the literal representation of their actual parameters from the macro call, and
  2. macro arguments that are preceded by a comma within a backquoted form are substituted with the literal representation of the parameter from the macro call.

Here are some more macro definitions. Experiment with these in your Lisp system to see what they do.

(defmacro sortf (place)
  `(setf ,place (sort ,place)))

(defmacro togglef (place)
  `(setf ,place (not ,place)))

(defmacro either (form1 form2)
  ; (random 2) returns 0 or 1 
  `(if (zerop (random 2)) ,form1 ,form2))

Beyond the obvious, part 1: compute, then generate

Macros start to get interesting when they do more than a simple textual substitution. In this section, we'll explore a real-world example of using a macro to extend Lisp into the problem domain. In addition to providing a macro expander, our new macro will automatically generate an environment that will be referenced by the expander. Our example will show how to move computations from run-time to compile-time, and how to share information computed at compile-time.

Let's say you're working on an interactive game that makes heavy use of the trigonometric function sine r in computing player motion and interaction. You've already determined that calling the Lisp function SIN is too time-consuming; you also know that your program will work just fine with approximate results for the computation of sine r. You'd like to define a LOOKUP-SIN macro to do the table lookup at runtime; you'd also like to hide the details of table generation, an implementation detail with which you'd rather not clutter your program's source code.

Your macro will be invoked as (LOOKUP-SIN radians divisions), where radians is always in the range of zero to one-quarter pi, and divisions is the number of discrete values available as the result of LOOKUP-SIN. At runtime, the macro expander will just compute the index into a lookup table, and return the value from the table. The table will be generated at compile-time (on most Lisp systems). Furthermore, only one table will ever be generated for a given value of divisions in the macro call.

Here's the code. The comments and documentation strings should help you to understand the code as you read it. I'll provide further explanation below.

;; This is where we cache all of the sine tables generated 
;; during compilation. The tables stay around at runtime 
;; so they can be used for lookups. 
(defvar *sin-tables* (make-hash-table)
  "A hash table of tables of sine values. The hash is keyed
by the number of entries in each sine table.")

;; This is a helper function for the LOOKUP-SIN macro. 
;; It is used only at compile time. 
(defun get-sin-table-and-increment (divisions)
  "Returns a sine lookup table and the number of radians quantized
by each entry in the table. Tables of a given size are reused.
A table covers angles from zero to pi/4 radians."
  (let ((table (gethash divisions *sin-tables* :none))
        (increment (/ pi 2 divisions)))
    (when (eq table :none)
      ;; Uncomment the next line to see when a table gets created. 
      ;;(print '|Making new table|) 
      (setq table
            (setf (gethash divisions *sin-tables*)
                  (make-array (1+ divisions) :initial-element 1.0)))
      (dotimes (i divisions)
        (setf (aref table i)
              (sin (* increment i)))))
    (values table increment)))

;; Macro calls the helper at compile time, and returns an 
;; AREF form to do the lookup at runtime. 
(defmacro lookup-sin (radians divisions)
  "Return a sine value via table lookup."
  (multiple-value-bind (table increment)
                       (get-sin-table-and-increment divisions)
    `(aref ,table (round ,radians ,increment))))

If you still don't see the point of all this code after having read the introduction to this section and the comments in the code, here it is: when your program runs, it executes just AREF (and associated ROUND) to look up the sin r value.

? (pprint (macroexpand-1 '(lookup-sin (/ pi 4) 50)))

(AREF #(0.0 0.0314107590781283 0.06279051952931337 
        [additional entries not shown] 
        0.9980267284282716 0.9995065603657316 1.0)
      (ROUND (/ PI 4) 0.031415926535897934))
;; Note that the macro call makes no mention of a lookup table. 
;; Tables are generated as-needed by (and for) the compiler. 
? (lookup-sin (/ pi 4) 50)
0.7071067811865476

In the macroexpansion, the #(...) is the printed representation of the lookup table for 50 divisions of the quarter circle. This table is stored in the *SIN-TABLES* hash table, where it is shared by every macro call to (LOOKUP-SIN angle 50). We don't even have to do a hash lookup at runtime, because the macro expander has captured the free variable TABLE from the MULTIPLE-VALUE-BIND form in LOOKUP-SIN.

Beyond the obvious, part 2: macros that define macros

Macros that define macros are used infrequently, partly because it's hard to think of a good use for this technique and partly because it's difficult to get right. The following macro, based upon an example in Paul Graham's "On Lisp" book, can be used to define synonyms for the names of Lisp functions, macros, and special forms.

? (defmacro defsynonym (old-name new-name)
    "Define OLD-NAME to be equivalent to NEW-NAME when used in
the first position of a Lisp form."
    `(defmacro ,new-name (&rest args)
       `(,',old-name ,@args)))
DEFSYNONYM
? (defsynonym make-pair cons)
MAKE-PAIR
? (make-pair 'a 'b)
(A . B)

Macros are always a little bit dangerous because code containing a macro call does not automatically get updated if you change the definition of the macro. You can always establish your own convention to help you remember that you need to recompile certain code after you change a macro definition. But there's always the possibility that you'll forget, or make a mistake.

Ultimately, the likelihood that you'll inadvertently end up with code that was compiled with an old version of a macro is directly proportional to how often you're likely to change the macro. I'll probably never need to change the LOOKUP-SIN macro from the previous section once it's defined and working. On the other hand, a macro like DEFSYNONYM practically begs to be used again and again as you generate new code. If you change your mind about the old name to associate with a given new name, all of your previously compiled code will still refer to the old name that you had decided upon previously.

;; WARNING: This example illustrates a practice to avoid! 

;; Here's some core algorithm 
? (defun process-blah-using-algorithm-zark (...) ...)
PROCESS-BLAH-USING-ALGORITHM-ZARK

;; And here's where I use the algorithm, perhaps calling it 
;; from many other places in DO-STUFF besides the one I've shown. 
? (defun do-stuff (...)
    ...
    (process-blah-using-algorithm-zark ...)
    ...)
DO-STUFF
;; Try it out... 
? (do-stuff ...)
[results based upon process-blah-using-algorithm-zark]
;; OK, this looks good.  But I think I'll clean up the 
;; appearance of DO-STUFF by defining an abbreviation 
;; for that really long core algorithm name. 
? (defsynonym process-blah-using-algorithm-zark proc)
PROC
;; Now I'll rewrite DO-STUFF to use the abbreviation. 
? (defun do-stuff (...)
    ...
    (proc ...)
    ...)
DO-STUFF
;; And make sure it still works. 
? (do-stuff ...)
[results based upon process-blah-using-algorithm-zark]

... Some time later ... 

;; Oh, here's a better core algorithm. 
? (defun process-blah-using-algorithm-glonkfarkle (...) ...)
PROCESS-BLAH-USING-ALGORITHM-GLONKFARKLE
;; I'll change the synonym for PROC to 'be' the new algorithm. 
? (defsynonym process-blah-using-algorithm-glonkfarkle proc)
PROC

... Some time later ... 

;; Time to use DO-STUFF again... 
? (do-stuff ...)
[results based upon process-blah-using-algorithm-zark]
;; Hey!! These results don't seem to use the new algorithm. 
;; What could be wrong?  The code LOOKS right... 

The problem, of course, is that the second use of DEFSYNONYM redefined the PROC macro, and I didn't notice that DO-STUFF needed to be recompiled to pick up the changed definition.

My advice: Don't try to be clever by using macros like DEFSYNONYM. Stick with descriptive names that are as long as necessary, and use an editor that supports symbol completion (see Chapter 27). Remember, there's only one way to not abbreviate a name; using abbreviations increases the chance that you'll use the wrong one.

Tricks of the trade: elude capture using GENSYM

You have to be be careful when you define a macro that introduces new variables in its expansion. The REPEAT macro, below, offers us a shorthand way of repeating a body of code a certain number of times.

? (defmacro repeat (times &body body)
    `(dotimes (x ,times)
       ,@body))
REPEAT
? (repeat 3 (print 'hi))

HI
HI
HI
NIL

This seems to do the right thing, but the variable X is going to cause problems. The following example should give us the same results as the last example.

? (setq x 'hi)
HI
? x
HI
? (repeat 3 (print x))

0
1
2
NIL

The variable X in the macro expander shadowed the global X that we tried to reference in the body. Another way to say this is that X is free in the body of the REPEAT form, but it was captured by the definition of X in the macro expander; this prevents the body form from reaching the intended variable X.

The obvious solution is to use a different variable name in the macro expander -- one that won't conflict with any name we'll ever use in our code that calls the REPEAT macro. You might think that some kind of naming convention would work, but there's always the chance that some programmer will come along later and violate the convention. We need a foolproof approach.

Lisp provides a GENSYM function to generate symbols that are guaranteed to be unique. No programmer can ever write a symbol name that conflicts with a symbol created by GENSYM. Here is how we use GENSYM to create a name for the variable needed in the macro expander for the REPEAT macro.

? (defmacro repeat (times &body body)
    (let ((x (gensym)))
      `(dotimes (,x ,times)
         ,@body)))
REPEAT
? x
HI
? (repeat 3 (print x))

HI
HI
HI
NIL

With this new REPEAT macro, we compute a new symbol in the LET form, and substitute this symbol in the macro expander form. To see why this works, let's look at an expansion:

? (macroexpand-1 '(repeat 5 (print x))
(DOTIMES (#:G8524 5) (PRINT X))

#:G8524 is a unique uninterned symbol. You can see that it's uninterned by the #: prefix. But how does Lisp guarantee the uniqueness of this symbol? The Lisp reader guarantees that any symbol it reads with the #: prefix is unique. Compare the following:

? (eq 'a 'a)
T
? (eq '#:a '#:a)
NIL

Even though the #:A symbols print the same, they are different.

Generating variable names to be used in macro expanders has another application. This next macro definition has a subtle problem:

? (defmacro cube (n)
    `(* ,n ,n ,n))
CUBE
? (cube 3)
27
? (let ((n 2))
    (cube (incf n)))
60

In the second case, (INCF N) should have provided the value 3 to CUBE and the result should have been identical to the first test. Let's take a look at the expansion again, to see what happened.

? (macroexpand-1 '(cube (incf n)))
(* (INCF N) (INCF N) (INCF N))

The problem is obvious: CUBE's argument, (INCF N) is being evaluated multiple times. As a rule, this is a bad thing to do, because it violates our assumptions about the way Lisp evaluates forms. We fix this problem by arranging for the macro expander to evaluate CUBE's argument just once.

? (defmacro cube (n)
    (let ((x (gensym)))
      `(let ((,x ,n))
         (* ,x ,x ,x))))
CUBE
? (let ((n 2))
    (cube (incf n)))
27

We created a unique symbol outside of the macro expander, then used this symbol in the expander as the name of a variable to hold the result of evaluating CUBE's argument. The LET form in the macro expander is the only place where CUBE's argument is referenced, so it gets evaluated exactly once for each call to CUBE.

Macros vs. inlining

Lisp allows functions to be inlined by the compiler. In other words, rather than compiling a call to the function, the compiler may substitute the function's body, thus saving the overhead of a function call. Substituting the function's body is generally expensive in terms of space, since a function body's code is usually longer than the code of its calling sequence.

It's important to understand that Lisp allows functions to be inlined. Like all other declarations -- save the SPECIAL declaration -- an INLINE declaration may be treated as advisory or ignored entirely by the compiler.

Here are some examples of how to inline a function. In the first case, function F is inlined everywhere it is used (assuming that the compiler supports inlining). In the second case, function P is compiled with information to support inlining, but is only inlined in the presence of a declaration, as in function Q.

; Case 1 -- F may always be inlined 
(declaim (inline f))
(defun f (...) ...)

(defun g (...)
  (f ...)
  ...)

(defun h (...)
  (f ...)
  ...)

; Case 2 - P may be inlined only following a declaration 
(declaim (inline p))
(defun p (...) ...)
(declaim (notinline p))

(defun q (...) 
  (declare (inline p))
  (p ...) ; inlined 
  ...)

(defun r (...)
  (p ...) ; not inlined 
  ...)

Macros may be used in place of INLINE declarations for cases where code absolutely must be inlined despite the presence (or absence) of compiler support for inlining.

In general, though, you should use macros for language extension, and not for efficiency hacks. The risk of forgetting to recompile after having changed a macro definition can cause hard-to-find bugs that will waste a lot of your development effort.

My advice: Don't use macros as a substitute for inlining unless you can find no other way to achieve desired performance; of course, such efforts should be guided by the results of profiling your code (see Chapter 28) and preferably only when your code is already stable and debugged. You should also reexamine your decision with each new release of your Lisp compiler, and whenever you port your program to another Lisp platform.

Contents | Cover
Chapter 19 | Chapter 21
Copyright © 1995-1999, David B. Lamkins
All Rights Reserved Worldwide

This book may not be reproduced without the written consent of its author. Online distribution is restricted to the author's site.