Chapter 15 - Closures

In this chapter we'll expand upon the discussion of closures that we started in Chapter 11. We'll see again how (and why) closures capture free variables for use in other execution contexts, then we'll see some practical applications. We'll close this chapter with a look at functions that return functions.

Is it a function of the lifetime, or the lifetime of a function?

Common Lisp does not expose closures per se. Recall from Chapter 11 that a closure is a collection of closed-over variables retained by a function. (A closed-over variable is a variable found "free" in the function; this gets "captured" by the closure. We saw some examples of this in Chapter 11; we'll review the details in the next section, in case you've forgotten.) For this reason, Lisp programmers tend to refer to "a function having closed-over variables" as simply "a closure." Or maybe they call it that because it saves them nine syllables.

A closure has to be associated with a function, so it must have the same lifetime -- or extent -- as the function. But all of the closed-over variables come along for the ride -- a closed-over variable has the same extent as the closure. This means that you can close over a lexical variable, which would normally have lexical extent, and give that variable indefinite extent. This is a very useful technique, as we'll see shortly.

How to spot a free variable, and what to do about it.

A variable is free within a function (or within any form, for that matter) if there is no binding occurrence of its name within the lexical scope -- the textual bounds, more or less -- of the function. A binding occurrence is an occurrence of the name that (according to the definition of the form that includes the name) associates storage with the name.

A free variable must be found in one of two places. Either the function is textually wrapped within a form that provides a binding occurrence of the variable, or the variable is special (review Chapter 8) and contained in the global environment. If a free variable is not found in one of these two places, it is unbound (i.e. has no storage associated with the name) and will cause an error when referenced at runtime.

Using closures to keep private, secure information.

If you close over a lexical variable, that variable is accessible only from within the closure. You can use this to your advantage to store information that is truly private, accessible only to functions that have a closure containing your private variable(s).

? (let ((password nil)
        (secret nil))
    (defun set-password (new-passwd)
      (if password
        '|Can't - already set|
        (setq password new-passwd)))
    (defun change-password (old-passwd new-passwd)
      (if (eq old-passwd password)
        (setq password new-passwd)
        '|Not changed|))
    (defun set-secret (passwd new-secret)
      (if (eq passwd password)
        (setq secret new-secret)
        '|Wrong password|))
    (defun get-secret (passwd)
      (if (eq passwd password)
        secret
        '|Sorry|)))
GET-SECRET
? (get-secret 'sesame)
|Sorry|
? (set-password 'valentine)
SECRET
? (set-secret 'sesame 'my-secret)
|Wrong password|
? (set-secret 'valentine 'my-secret)
MY-SECRET
? (get-secret 'fubar)
|Sorry|
? (get-secret 'valentine)
MY-SECRET
? (change-password 'fubar 'new-password)
|Not changed|
? (change-password 'valentine 'new-password)
NEW-PASSWORD
? (get-secret 'valentine)
|Sorry|
; The closed-over lexical variables aren't in the global environment 
? password
Error: unbound variable
? secret
Error: unbound variable
; The global environment doesn't affect the closed-over variables 
? (setq password 'cheat)
CHEAT
? (get-secret 'cheat)
|Sorry|

Functions that return functions, and how they differ from macros.

The preceding example is only good for keeping one secret, because every time we evaluate the outer LET form we redefine all of the functions that close over our "private" variables. If we want to eliminate our dependence upon the global namespace for functions to manipulate our closed-over variables, we're going to have to find a way to create new closed-over variables and return a function that we can save and later use to manipulate the variables. Something like this will work:

? (defun make-secret-keeper ()
    (let ((password nil)
          (secret nil))
      #'(lambda (operation &rest arguments)
          (ecase operation
            (set-password
             (let ((new-passwd (first arguments)))
               (if password
                 '|Can't - already set|
                 (setq password new-passwd))))
            (change-password
             (let ((old-passwd (first arguments))
                   (new-passwd (second arguments)))
               (if (eq old-passwd password)
                 (setq password new-passwd)
                 '|Not changed|)))
            (set-secret
             (let ((passwd (first arguments))
                   (new-secret (second arguments)))
               (if (eq passwd password)
                 (setq secret new-secret)
                 '|Wrong password|)))
            (get-secret
             (let ((passwd (first arguments)))
               (if (eq passwd password)
                 secret
                 '|Sorry|)))))))
MAKE-SECRET-KEEPER
? (defparameter secret-1 (make-secret-keeper))
SECRET-1
? secret-1
#<LEXICAL-CLOSURE #x36AE056>
? (funcall secret-1 'set-password 'valentine)
VALENTINE
? (funcall secret-1 'set-secret 'valentine 'deep-dark)
DEEP-DARK
? (defparameter secret-2 (make-secret-keeper))
SECRET-2
? (funcall secret-2 'set-password 'bloody)
BLOODY
? (funcall secret-2 'set-secret 'bloody 'mysterious)
MYSTERIOUS
? (funcall secret-2 'get-secret 'valentine)
|Wrong password|
? (funcall secret-1 'get-secret 'valentine)
DEEP-DARK

The ECASE form is an exhaustive case statement. In our program, the OPERATION must be found in one of the ECASE clauses, or Lisp will signal an error.

The #'(LAMBDA ... form creates a closure over the free variables PASSWORD and SECRET. Each time we evaluate MAKE-SECRET-KEEPER, the outermost LET form creates new bindings for these variables; the closure is then created and returned as the result of the MAKE-SECRET-KEEPER function.

In pre-ANSI Common Lisp, LAMBDA is merely a symbol that is recognized as a marker to define a lambda expression. By itself, LAMBDA does not create a closure; that is the function of the #' reader macro (which expands into a (FUNCTION ... form).

ANSI Common Lisp defines a LAMBDA macro that expands into (FUNCTION (LAMBDA ..., which you can use in place of #'(LAMBDA wherever it appears in this example. For backward compatibility with pre-ANSI Common Lisp implementations, you should always write #'(LAMBDA ... -- the redundant (FUNCTION ... in the expansion will do no harm.

Within each ECASE clause we extract arguments from the &REST variable ARGUMENTS and then do exactly the same processing as in our earlier example.

Once we have invoked MAKE-SECRET-KEEPER and saved the resultant closure, we can FUNCALL the closure, passing the operation symbol and any additional arguments. Note that each closure created by MAKE-SECRET-KEEPER is completely independent; we've therefore achieved the goal of being able to keep multiple secrets.

Functions that return closures are different from macros. A macro is a function that produces a form; the form is then evaluated to produce a result. A function that returns a closure simply returns an object: the closure. The returned closure is not automatically evaluated by the Lisp evaluator.