Chapter 19 - Streams

All of the I/O functions in Lisp accept a stream argument. In some cases (e.g. READ and PRINT) the stream argument is optional; by default, input comes from the *STANDARD-INPUT* stream (normally connected to the keyboard) and output goes to the *STANDARD-OUTPUT* stream (normally connected to the display). You can redirect I/O by either providing optional stream arguments to READ and PRINT (as well as other I/O functions), or by binding *STANDARD-INPUT* and *STANDARD-OUTPUT* to different streams. (We'll see both of these approaches used in the following examples.)

Streams provide a pipe to supply or accept data

Throughout the preceding chapters of this book, streams have been involved whenever we've seen an example that does input or output -- and all of the examples do I/O, if you count our interactions with the listener. A Lisp stream can provide (source) or consume (sink) a sequence of bytes or characters. (Remember the Lisp definition of byte: a byte can contain any number of bits.)

Some I/O functions accept T or NIL as a stream designator. T is a synonym for *TERMINAL-IO*, a bidirectional (input and output) stream which conventionally reads from *STANDARD-INPUT* and writes to *STANDARD-OUTPUT*. NIL is a synonym for *STANDARD-INPUT* when used in a context which expects an input stream, or for *STANDARD-OUTPUT* when used in a context which expects an output stream.

FORMAT (which we've already seen in several examples, and will examine in depth in Chapter 24) expects as its first argument a stream, a T, a NIL, or a string with a fill pointer. In this case, however, the NIL designator causes FORMAT to return a string, rather than write to *STANDARD-OUTPUT* as is the case for other I/O functions.

The power of streams comes from the ability to associate a stream with a file, a device (such as keyboard, display, or network), or a memory buffer. Program I/O can be directed at will by simply creating the appropriate type of stream for your program to use. The I/O implementation is abstracted away by the stream so your program won't have to be concerned with low-level details.

Lisp also provides a number of special-purpose streams which serve to combine or manipulate other streams in novel ways. A TWO-WAY-STREAM combines a separate input stream and output stream into an I/O stream. A BROADCAST-STREAM sends output to zero or more output streams; think of this as a bit-bucket when used with zero streams, and a broadcaster when used with multiple streams. A CONCATENATED-STREAM accepts input requests on behalf of zero or more input streams; when one stream's input is exhausted, the CONCATENATED-STREAM begins reading from its next input stream. An ECHO-STREAM is like a TWO-WAY-STREAM, with the added feature that anything your program reads from the TWO-WAY-STREAM's input stream automatically gets echoed to the corresponding output stream. Finally, a SYNONYM-STREAM is an alias for another stream; the alias can be changed at runtime without creating a new SYNONYM-STREAM.

Quite a few I/O functions operate directly on streams:

READ-BYTE stream &optional eof-error-p eof-value and READ-CHAR &optional stream eof-error-p eof-value recursive-p

Reads a byte or a character from an input stream.

WRITE-BYTE byte stream and WRITE-CHAR char &optional stream

Writes a byte or a character to an output stream.

READ-LINE &optional stream eof-error-p eof-value recursive-p and WRITE-LINE string &optional stream &key start end

Read or write a line of text, terminated by a newline. (The newline is consumed and discarded on input, and added to output.) The :START and :END keyword arguments let you limit the portion of the string written by WRITE-LINE.

WRITE-STRING string &optional stream &key start end

Like WRITE-LINE, but does not append a newline to output.

PEEK-CHAR &optional peek-type stream eof-error-p eof-value recursive-p

Reads a character from an input stream without consuming the character. (The character remains available for the next input operation.) Optional argument peek-type alters PEEK-CHAR's behavior to first skip whitespace (peek-type T) or to first skip forward to some specified character (peek-type a character).

UNREAD-CHAR character &optional stream

Pushes a character (which must be the character most recently read) back onto the front of an input stream, where it remains until read again.

LISTEN &optional stream

Returns true if data is available (e.g. a yet-to-be-read keystroke or unconsumed file data) on an input stream.

READ-CHAR-NO-HANG &optional stream eof-error-p eof-value recursive-p

If a character is available on the input stream, return the character. Otherwise, return NIL.

TERPRI &optional stream and FRESH-LINE &optional stream

TERPRI unconditionally writes a newline to an output stream. FRESH-LINE writes a newline unless it can determine that the output stream is already at the beginning of a new line; FRESH-LINE returns T if it actually wrote a newline, and NIL otherwise.

CLEAR-INPUT &optional stream

Flushes unread data from an input stream, if it makes sense to do so.

FINISH-OUTPUT &optional stream, FORCE-OUTPUT &optional stream, and CLEAR-OUTPUT &optional stream

These functions flush output buffers if it makes sense to do so. FINISH-OUTPUT tries to make sure that buffered output reaches its destination, then returns. FORCE-OUTPUT attempts to initiate output from the buffer, but does not wait for completion like FINISH-OUTPUT. CLEAR-OUTPUT attempts to discard buffered data and abort any output still in progress.

In the read functions listed above, optional arguments EOF-ERROR-P and EOF-VALUE specify what happens when your program makes an attempt to read from an exhausted stream. If EOF-ERROR-P is true (the default), then Lisp will signal an error upon an attempt to read an exhausted stream. If EOF-ERROR-P is NIL, then Lisp returns EOF-VALUE (default NIL) instead of signalling an error.

Optional argument RECURSIVE-P is reserved for use by functions called by the Lisp reader.

Creating streams on files

The OPEN function creates a FILE-STREAM. Keyword arguments determine attributes of the stream (:DIRECTION, :ELEMENT-TYPE, and :EXTERNAL-FORMAT) and how to handle exceptional conditions (:IF-EXISTS and :IF-DOES-NOT-EXIST). If OPEN is successful it returns a stream, otherwise it returns NIL or signals an error.

Keyword      Value     Stream Direction
----------   -------   -----------------------------
:DIRECTION   :INPUT    input (default)
:DIRECTION   :OUTPUT   output
:DIRECTION   :IO       input & output
:DIRECTION   :PROBE    none, returns a closed stream

Keyword      Value                Action if File Exists
----------   ------------------   ---------------------------------------
:IF-EXISTS   NIL                  return NIL
:IF-EXISTS   :ERROR               signal an error
:IF-EXISTS   :NEW-VERSION         next version (or error)
:IF-EXISTS   :RENAME              rename existing, create new
:IF-EXISTS   :SUPERSEDE           replace file upon CLOSE
:IF-EXISTS   :RENAME-AND-DELETE   rename and delete existing, create new
:IF-EXISTS   :OVERWRITE           reuse existing file (position at start)
:IF-EXISTS   :APPEND              reuse existing file (position at end)

Keyword              Value     Action if File Does Not Exist
------------------   -------   -----------------------------
:IF-DOES-NOT-EXIST   NIL       return NIL
:IF-DOES-NOT-EXIST   :ERROR    signal an error
:IF-DOES-NOT-EXIST   :CREATE   create the file

Keyword         Value               Element Type
-------------   --------------      ------------------------
:ELEMENT-TYPE   :DEFAULT            character (default)
:ELEMENT-TYPE   'CHARACTER          character
:ELEMENT-TYPE   'SIGNED-BYTE        signed byte
:ELEMENT-TYPE   'UNSIGNED-BYTE      unsigned byte
:ELEMENT-TYPE   character subtype   character subtype
:ELEMENT-TYPE   integer subtype     integer subtype
:ELEMENT-TYPE   other               implementation-dependent

Keyword            Value      File Format
----------------   --------   ------------------------
:EXTERNAL-FORMAT   :DEFAULT   default (default)
:EXTERNAL-FORMAT   other      implementation-dependent

Once you've opened a stream, you can use it with appropriate input or output functions, or with queries that return attributes of either the stream or the file. The following queries can be applied to all kinds of streams. All of these accept a stream argument:

Function               Returns
--------------------   -----------------------------------------------------
INPUT-STREAM-P         true if stream can provide input
OUTPUT-STREAM-P        true if stream can accept output
OPEN-STREAM-P          true if stream is open
STREAM-ELEMENT-TYPE    the type specifier for stream elements
INTERACTIVE-STREAM-P   true if stream is interactive (e.g. keyboard/display)

These queries can be applied to file streams. These also accept a stream argument:

Function                 Returns
--------------------     -----------------------------------------------------
STREAM-EXTERNAL-FORMAT   implementation-dependent
FILE-POSITION            current file offset for read or write, or NIL
FILE-LENGTH              length of stream, or NIL

FILE-POSITION returns a byte offset within the stream. This is an exact count for streams of integer subtypes (see below for further description of binary I/O). For streams of character subtypes, the position is guaranteed only to increase during reading or writing; this allows for variations in text record formats and line terminators.

FILE-POSITION can also be called with a second argument to change the file offset for the next read or write. When used for this purpose, FILE-POSITION returns true when it succeeds.

You should always close a stream when you're done using it (except for the interactive streams provided for you use by Lisp, such as *STANDARD-INPUT*, *STANDARD-OUTPUT*, and *TERMINAL-IO*). The "open, process, close" pattern is very common, so Lisp provides macros to make the pattern both easy to code and error-free.

WITH-OPEN-FILE is tailored for file streams. Its arguments are a variable to be bound to the stream, a pathname, and (optionally) keyword arguments suitable for OPEN. The stream is always closed when control leaves the WITH-OPEN-FILE form.

(with-open-file (stream "my-file.dat" :direction :input)
  ... do something using stream ...)

WITH-OPEN-STREAM expects a variable name and a form to be evaluated; the form should produce a stream value or NIL. This macro is commonly used with constructors for specialty streams, such as MAKE-BROADCAST-STREAM, MAKE-ECHO-STREAM, MAKE-TWO-WAY-STREAM, MAKE-CONCATENATED-STREAM, and MAKE-SYNONYM-STREAM.

Creating streams on strings

The data read or written by a stream doesn't have to be associated with a device -- the data can just as well be in memory. String streams let you read and write at memory speeds, but they can't provide either file or interactive capabilities. Lisp provides constructors (MAKE-STRING-INPUT-STREAM and MAKE-STRING-OUTPUT-STREAM), plus macros to support the "open, process, close" pattern.

? (with-input-from-string (stream "This is my input via stream.")
    (read stream))
THIS
? (with-output-to-string (stream)
    (princ "I'm writing to memory!" stream))
"I'm writing to memory!"

These macros accept keyword and optional arguments. WITH-INPUT-FROM-STRING allows :BEGIN and :END keyword arguments to establish bounds on the portion of the string read via the stream. A :INDEX keyword argument lets you name a variable to receive the offset of the next string element to be read -- this is set only upon leaving the WITH-INPUT-FROM-STRING form.

WITH-OUTPUT-TO-STRING allows an optional form, which is evaluated to produce the output string; if this form is missing or NIL, the macro creates a string for you using the :ELEMENT-TYPE keyword argument.

Binary I/O

Lisp supports binary I/O via streams whose element types are finite (i.e. bounded) subtypes of INTEGER. Some examples of appropriate types are:

• Implementation-dependent
  • SIGNED-BYTE
  • UNSIGNED-BYTE
• Range of values
  • (INTEGER 0 31)
  • (INTEGER -16 15)
• Specific number of bits
  • (SIGNED-BYTE 8)
  • (UNSIGNED-BYTE 6)
  • BIT

ANSI Common Lisp implementations should support any of these types for binary I/O. However, the implementation is not required to directly map the specified :ELEMENT-TYPE onto the underlying file system; an implementation is permitted to alter the external format so long as data read from a binary file is the same as that written using the same :ELEMENT-TYPE.