Lisp as a Shared Library (original) (raw)

Allegro CL supports the use of lisp via a shared library by C/C++ programs in the same manner as they use any other function libraries. This document describes how Allegro Common Lisp developers on UNIX machines can create applications with Lisp as a shared library. See <dll.html> for information on creating similar files on Windows.

This functionality makes use of posix threads and is supported on all Unix-style Allegro CL platforms.

1.0 Why an updated interface

The previous Lisp as a shared library example was written at a time when good thread library support was not available on many platforms. As a result, the example was available on very few platforms. Further, the example was built in a way that was not compatible with how we build our lisp binaries, link in the allegro shared library, and load in the lisp environment at startup. This often led to problems in customer applications that were difficult to debug. Lastly, many changes have gone into our lisp environment since the example was written. A refactoring of the Lisp as a shared library interface allows us to provide a more simplified and concise API.

Many of the steps involved in loading the lisp environment need not be exposed to the user, nor clutter their code. This updated interface hopes to better separate lisp startup code from user application code. A single call is needed to start lisp initialization. This call handles choosing the appropriate allegro shared library, spawning a thread, and returning control to the user once initialization is complete. Various hooks, described below, allow you to perform app specific initialization at various points, as well as synchronize access to the Lisp control thread, from which calls into the lisp environment can be made.

The following files are needed to build a lisp shared library application.

Also in examples/unix-shared-library/ is code for a sample C application that calculates factorials by calling into lisp. An architecture specific makefile is also provided with the recommended build commands for compiling and linking user applications. Typing make in this directory will build the factorial example (ftest), placing it in the subdirectory fact/.

Run fact/ftest to test the demo. You will likely need to set your dynamic library search path environment variable to include this directory. See OS Specific Library Search Path for what environment variable is expected to be set.

The components of a Lisp as a shared library application are as follows:

Foreign Code

Lisp code

The basic flow of control of a Lisp as a shared library application appears as follows:

MAIN THREAD

- user defined C main()
  - calls initialize_lisp(...)
    - spawns a new thread to load and initialize lisp
    - (wait for lisp_init_complete() signal)     
  - ...continue app execution... (see FOREIGN THREAD below)       

LISP THREAD

- lisp initialization begins, normal lisp startup sequence begins.
  - *restart-init-function* is called 
    (user:initialize-lisp, from lnkacl.fasl)

    - internal set up (such as define callback routine 
      lisp_lookup_address() )
    - call *restart-app-function*, a user defined function 
      to perform necessary lisp setup for user application.
    - call lisp_ready_hook()
      - signal init complete by calling lisp_init_complete()
      - user-defined code to handle requests to be handled by lisp
      - perform any application setup as necessary.
      - (wait for work)
      - call lisp routine. lisp_lookup_address() takes a string 
        as argument and returns a function pointer to the 
        associated lisp callback function.
      - pass return value(s) back to calling thread.
      - resume (wait for work) or return, which will terminate 
        the lisp thread.
   - exit_routine(int value)
     cleanup as necessary for shutting down lisp environment.

FOREIGN THREAD (MAIN or other)

    - contact lisp thread to request work be performed.
    - wait for (user-defined) synchronization event signifying
      completion of lisp work.
    - use value
    ... continue app execution ...

5.0 A note about using Foreign Functions on non-os-thread platforms

On non-os-thread platforms, only a single thread (the thread spawned by initialize_lisp()) can call into the lisp environment. As a result, multi-threaded applications must synchronize with this thread in order to dispatch work performed by lisp. This is done either from within lisp in the application's *restart-init-function* or via foreign code in the lisp_ready_hook()).

See Foreign functions and multiprocessing in <foreign-functions.html>.

Synchronization with the lisp thread is left to the application developer. A factorial example is provided which uses one possible method.

6.0 C API (routines and data structures)

The following functions must be defined (except where labeled optional).

int initialize_lisp(char **envp, char *image_arg, int libtype, 
                   int suppress_io,
                   void (*exit_routine) (int),
                   struct shlib_library_item **shlib_items)

This existing function must be called to set up the lisp environment. The arguments are:

The Allegro shared library, and the image_arg if specified with no path component, are searched for at startup using the OS specific library search path environment variable. See OS Specific Library Search Path to find out what the search path is for your target platform.

This function returns 0 on success, or a negative value when an error occurs.

void *lisp_lookup_address(char *func_name)

This existing function is a simplified routine for finding entry points (function pointers) into lisp. This is provided as an alternative to the index based lookup API provided by register-foreign-callable and lisp_call_address(). As of Allegro CL 8.1, the address returned by register-foreign-callable is allocated in Allegro's C heap, and is static even across dumplisps. As a result, it is possible to map directly from function name to function pointer without the index redirection.

The arguments are:

This function returns a function pointer to the indicated function or 0 if not found.

This existing function is called from the callback routine lisp_ready_hook(). This signals initialize_lisp() that the lisp environment has been initialized and control can be returned to it's calling function.

This function returns a value as pthread_cond_signal().

This function must be defined. The user must establish a C function that will be called into by the lisp thread, once initialization is complete. The body of this function must call lisp_init_complete(). The rest of the code in this routine should contain whatever application specific synchronization is required to communicate with the lisp thread.

The function has no return value. When this function returns, the *restart-init-function* in the lisp environment calls (exit) (see exit). This will then cause your exit routine, if specified, or the default exit routine, to be invoked.

Optional

exit_routine(int exit_value)

A default exit function is defined, but users may define their own to perform whatever cleanup of the lisp environment is desired. The exit-rountine argument to initialize_lisp specifies what exit routine to call. The default exit routine calls pthread_exit().

If you define an exit routine and expect your application to continue execution after the lisp thread is complete, it must call pthread_exit(). If the exit routine returns, exit() will be called, stopping the entire running process.

7.0 Lisp API

The lisp API is almost completely user-defined. After the lisp environment is loaded, the following values should be set in order to set up the lisp as a shared library application environments.

8.0 Compilation and Delivery

examples/unix-shared-library/makefile contains the recommended architecture specific compile/link commands for building your application. It also contains the recommended build script for generating your lisp application.

In addition, please note the following:

When building your lisp application,

See generate-application and <delivery.html> for general help with building lisp deliverables.

9.0 OS-Specific Library Search Path

On most *nix platforms, the dynamic loader library search path can be modified by setting the value of the environment variable LD_LIBRARY_PATH, with the following exceptions.

macOS

Apple uses DYLD_LIBRARY_PATH for finding dependent shared libraries. In some cases, this variable is removed from the environment across calls to fork() and exec() (the system calls used to create a new process). If the first idiom below (for the BASH shell) does not work, then you must use the second:

$ export DYLD_LIBRARY_PATH=...
$ allegro

This is more cumbersome, but it will work when the above does not:

$ env DYLD_LIBRARY_PATH=... allegro

The above instructions apply to allegro as well as mlisp, alisp or any application built with Allegro CL.

Copyright (c) Franz Inc. Lafayette, CA., USA. All rights reserved.