Lisp provides a wonderful development environment, as we'll see in the next few chapters. But Lisp would be of little value for some applications without a way to access external programs written in other languages. Fortunately, modern Lisp implementations have a Foreign Function Interface, or FFI for short.
In this chapter I'll describe FFI in general terms. Implementations differ in the details since FFI has not (yet) been standardized. Despite the lack of standardization, current implementations seem to have converged on a similar set of features.
An FFI lets your Lisp program interact with code that is "foreign" -- i.e. not Lisp.
This Lisp-centric view of the world is probably motivated by the Lisp machines, where everything -- even the low-level portions of the OS -- was written in Lisp. A good many of the people involved with Lisp during that time are responsible as well for the development of modern Lisp implementations; hence, the not-so-subtle nod toward the notion of Lisp as the center of the programmer's universe.
A typical FFI provides for both calls from Lisp to separately-compiled code, and from separately compiled code to Lisp. (In the latter case, it is almost always true that the external code must have been called from Lisp; it can then call back into Lisp.) Most often, an FFI supports a C calling convention.
Why is an FFI even necessary? Why can't you link-in separately compiled code as in any other language? The main reason is that Lisp datatypes don't generally have equivalents in conventional languages. For example, C integers typically fill (depending upon declaration) one-half, one, or two machine words and produce mathematically incorrect results when a result exceeds the representational capacity of the integer's storage. A Lisp integer can fit in a machine word, saving a few bits for a type tag. These are called fixnums. Lisp integers having magnitudes exceeding the capacity of the single word representation are converted to a representation that has an unlimited number of bits -- these are bignums. And with a good compiler, you can define subtypes of integers that, when packed into an array, have just enough bits in their representation to handle the declared range of values.
So, one purpose of an FFI is to translate Lisp datatypes to (and from) "foreign" datatypes. Not all conversions are possible -- a good FFI will signal an error when a conversion is not possible at runtime.
When a non-Lisp function accepts or returns values in a record datatype, the FFI must provide a means of constructing appropriate records. Typically, the FFI gives you a way to construct records that are bit-for-bit identical to those that would have been produced by another language. Fields within a record are set and retrieved using specialized Lisp accessors.
An FFI must also support the proper function calling protocol for non-Lisp functions. Protocols differ by platform and by language. Lisp function calling conventions normally differ from those used by other languages. Lisp supports optional, keyword, default, and rest parameters, multiple return values, closures, and (sometimes, depending upon the compiler) tail call elimination; a conventional language might implement tail call elimination.
What else must an FFI do? It loads object files produced by other languages, providing linker functionality within Lisp for these object files. A linker resolves named entries to code in the object file, and fills in machine addresses in the object code depending upon where the code loads into memory.
Finally, an FFI must resolve differences in memory allocation between Lisp and other languages. Most Lisp implementations allow objects to move during operation; this improves the long-term efficiency of memory management and can improve the performance of the program under virtual memory implementations by reducing the size of the working set. Unfortunately, most other languages expect objects to remain at a fixed memory address during program execution. So the FFI must arrange for a foreign function to see Lisp objects that don't move.
All of the above functionality is encapsulated by an FFI wrapper function. All you have to do is define the name, calling sequence and object code file of some foreign function, and the FFI will generate a wrapper that does all of the necessary translations. Once you've done this, the foreign function can be called just like any other Lisp function.
Usually a foreign function is called for its results or to have some effect on the external environment, and a simple call/return sequence is all that's needed. But some foreign functions, particularly those that deal with user interface or device I/O, require access to callback functions during their operation. A callback function is called from the foreign function.
To define a callback function in Lisp, the FFI basically has to solve all of the foreign function problems in the reverse direction. You use the FFI to define a Lisp function that is callable from another language. The result is typically a function object or pointer that can be passed (as a parameter) to a call of a foreign function. When the foreign function is called, it references the callback parameter in the normal manner to invoke the Lisp callback function. The FFI wrapper around the callback translates the foreign calling sequence and parameter values to the corresponding Lisp format, invokes the Lisp callback function, and returns the callback's results after suitable translation from Lisp to foreign formats.
TBCL... For now, use MCL's FFI. Gather examples from several implementations. Perhaps this section should be customized.