The standard way to detect failures is by way of the assert macro. This macro is also an excellent way of documenting what is known the programming contract, i.e. the agreement between a module and its clients concerning what constitutes valid use cases. A violation of the programming contract is a very common type of failure. The assert macro, in the case of a detected failure, will print an error message somewhat meaningful to a maintainer (but not to the user) and then terminate execution.
Failures that are not detected by the program itself are usually detected by the hardware and/or the operating system in the form of bus errors, segmentation violations, and similar. These failures cause the operating system to terminate the program with a message, usually more cryptic than that from assert, so whenever an assertion can reasonably be used to detect the failure, it is a good idea to do so.
Some failures are not detected at all, and indeed some can not be detected. In this case, the program will simply produce an incorrect result. This situation is not good at all. It is better for a program to terminate execution than not to, and produce incorrect results.
What to do in this situation depends on what kind of program we are dealing with. For a noninteractive program that can usually be restarted (possibly with different inputs and arguments), the easiest solution is to simply print an error message indicating that no memory is left, and then to terminate execution.
For an interactive program, the previous solution is a bit brutal. The best solution is usually to print an error message and then restart the interaction loop, possibly urging the user to take steps to liberate some memory. This solution requires some careful programming however. The execution of the interaction loop must not require additional memory, since we will then end up in an infinite computation.
A typical case would be opening a file that does not exist. In some cases, the file should be created in that case. The program may be better structured if it assumed the the open succeeds, and treat the other case exceptionally.
Consider a low-level library calling malloc. What is the low-level library supposed to do if no memory is available? We don't know, since it depends on the type of application (interactive or not) in which the library is used, perhaps even on how the application was executed (some applications can be run either interactively or noninteractively).
Thus, low-level code must somehow communicate the problem to high-level code, for it to decide what to do. How does this communication take place?
This kind of error handling has been common since the days of early programming languages that did not have more advanced mechanisms to handle such situations.
In Pascal, for instance, you make sure the procedure that discovers the exceptional situation is nested inside the procedure that handles it, and simply use an ordinary goto instruction to a label inside the procedure that handles the situation. This way of doing it imposes additional structure on the program that might not be wanted, but at least it is not impossible to deal with. Note that Turbo Pascal (at least in early versions) did not allow nonlocal gotos, which made it necessary to resort to return codes.
In the C language, we use two standard library routines, setjmp and longjmp. These two routines do not behave like ordinary C programs. They both use a small data structure called a jump buffer, or jmp_buf for short. The first routine, setjmp is given an object of type jmp_buf and saves the current execution point in the program in the data structure, and finally returns the integer value zero. The second routine, longjmp also takes a jmp_buf and an integer (which should be different from zero), restores the execution point to the one saved, effectively making the call to setjmp return again, but this time not with a zero value, but with the value given to longjmp. To handle an error then, a procedure would call setjmp and then test the return value. If that return value is zero, it is the normal program flow, so execution of the code that can provoke an exception can start. If not, the execution of the code resulted in an exceptional situation which should be handled properly. This way of handling exceptional situations is not great either, since both routines (the one that discovers the exceptional situation and the one that must handle it) must usually share a common variable, namely the jump buffer. Only occasionally is it practical to use this method directly. The good way is instead (at least for nontrivial programs) is to use these routines to build a real exception mechanism.
While it is preferable to use a programming language that has a built-in exception mechanism, sometimes you are stuck with C. Rather than rewriting an exception mechanism each time you write an application, you might be able to use the exception library written by Jean-Pierre Braquelaire. This library lets you use a simple form of exceptions with the C language.