CLPython Manual (original) (raw)

1. Installing CLPython

1.1 Supported Common Lisp implementations

CLPython is developed with portability in mind. The latest status of the supported Common Lisp implementations is as follows:

Your help is appreciated in making CLPython work on even more implementations, see Task: Porting CLPython.

[back to top]

1.2 Library dependencies

CLPython depends on the following libraries:

The libraries above are all dependencies to run CLPython in Allegro CL. But in other Lisp implementations the following libraries are also required:

[back to top]

1.3 Loading CLPython

CLPython is loaded with ASDF:

(asdf:operate 'asdf:load-op :clpython)

If this leads to an error that the "clpython component can not be found", it might be possible to continue by loading the system definition file manually:

(load "clpython.asd")
(asdf:operate 'asdf:load-op :clpython)

Upon loading of CLPython a "quick start guide" is printed. It shows a few ways in which Python source code can be evaluated:

CLPython quick start guide: Run a string of Python code: (clpython:run "for i in range(4): print i") Run a Python file: (clpython:run #p"~/example/foo.py") Start the Python "interpreter" (REPL): (clpython.app.repl:repl) To start mixed Python/Lisp input mode: (clpython:enter-mixed-lisp-python-syntax) Run the test suite: (asdf:operate 'asdf:test-op :clpython)

[back to top]

1.4 Running the test suite

The test suite can be run to verify that CLPython runs fine in your Lisp implementation. It generates a lot of output followed by the number of test successes and failures. There can be errors, but they should all be marked "known failures".

(asdf:operate 'asdf:test-op :clpython)
... a lot of output ...
End CLPython test
Errors detected in this test: 4 (all known failures)
Successes this test: 1305

[back to top]

2. Running Python code

2.1 A note on completeness

While many built-in Python classes, functions, and modules are implemented in CLPython, there is also a lot of functionality not available yet. Running valid Python code in CLPython will often fail due to missing functionality. Please see Completeness and compatibility for the details. Of course it would be appreciated if you would contribute missing functionality.

[back to top]

2.2 Evaluating a Python expression

The simplest way to execute a string of Python code is by calling generic function run on it:

(clpython:run "2 + 3")
5

As for the return value of run: this depends on whether the source code is a statement or an expression. Statements are used for definitions and control flow, like: if, def, class, try/except, import. Statements have expressions inside them. Expressions are forms that evaluate to a value, like: a+b, [a,b], f(1,2,3).

If the source supplied to run is an expression then its value is returned, like 5 in the example above. In case of statements the return value is an undefined implementation detail. Below is an example of running a statement:

(run "for i in range(4): print i")
0
1
2
3
nil

(Please see Mixed-syntax Lisp/Python mode on how to run Python code directly in the Lisp listener, as alternative to using run.)

[back to top]

2.3 Executing a Python module

Given a file foo.py with this contents:

def f():
print 'hello world'
f()

It can be executed using run passing the filename as absolute or relative pathname:

(run #p"foo.py")
hello world
None

To load a "package", which is basically a directory containing an __init__.py file, load that file:

(run #p"/tmp/zut/init.py")

Relative filenames will be looked up in the directories in *clpython-module-search-paths* and the Python value sys.path.

Filenames should always be passed as pathnames, not strings, because strings are treated directly as source code (see § 2.2) leading to errors like:

(run "foo.py")
NameError: Variable `foo' is unbound.

[back to top]

2.4 Python runtime environments

By default every time a Python expression is executed with run a fresh, empty execution environment is used. Referring to a variable bound in a previous call won't work, for example:

(run "a = 3")

(run "print a")
Error: NameError: Variable `a' is unbound.

This can be made to work if the calls share the execution environment, called habitat. If *habitat* is bound, it is used as execution environment, otherwise a new habitat is created and used:

(setf habitat (make-habitat))
#<habitat stdin=nil stdout=nil stderr=nil #loaded-modules=0 cmd-line-args=nil search-paths=#(".") @ #x2292eb52>

(run "a = 3") ;; stores the variable binding in habitat

(run "print a") ;; looks up the variable in habitat
3

Moreover you can use different habitats concurrently, and explicitly pass the one to use to run:

(defvar other-habitat (make-habitat))
other-habitat

(run "a = 4" :habitat other-habitat)

(run "print a" :habitat other-habitat)
4

And back to the orignal environment:

(run "print a" :habitat habitat)
3

(run "print a")
3

[back to top]

2.5 Compiling versus interpreting

Function run takes the keyword argument compile which determines whether the source should be compiled before being run. This options is only relevant if the Lisp implementation has both an interpreter and compiler, which is the case in e.g. LispWorks and Allegro CL. In compiler-only implementations like SBCL this option has no effect.

This option is in particular relevant when running benchmarks:

(run #p"/usr/lib/python2.6/test/pystone.py" :compile nil :args '("pystone.py" "3000"))
Pystone(1.1) time for 3000 passes = 26.709991
This machine benchmarks at 112.31752 pystones/second
None

(run #p"/usr/lib/python2.6/test/pystone.py" :compile t :args '("pystone.py" "3000"))
Pystone(1.1) time for 3000 passes = 0.11001587
This machine benchmarks at 27268.793 pystones/second
None

[back to top]

2.6 Python interpreter

A common way to evaluate Python expressions is by typing them into an "interpreter" or "read-eval-print loop" (REPL). CLPython comes with a REPL that acts like the regular Python interpreter:

(clpython:repl)
Welcome to CLPython, an implementation of Python in Common Lisp.
Running on: International Allegro CL Free Express Edition 8.2 [Windows] (May 16, 2010 12:01)
REPL shortcuts: :q' = quit, :h' = help.
Using 1 default module search paths set in clpython-module-search-paths

1 + 23 24 print 'hello Lisp world!' hello Lisp world!

For more information, please see the output of typing :h in the repl, Handling of errors, and the documentation for function repl.

This interpreter is useful for evaluating Python code, but not so practical when it comes to mixing Python and Lisp code. Use the more advanced mixed-mode interpreter for that instead .

[back to top]

2.7 Mixed-syntax Lisp/Python mode

CLPython is able to turn a regular Lisp listener (REPL) into a "mixed-mode" listener that supports both Lisp and Python source as input:

clpython(213): (clpython:enter-mixed-lisp-python-syntax)
; The mixed Lisp/Python syntax mode is now enabled; ; Lispy *readtable* is now set.
clpython(214): print 123 * 2
246
clpython(215): range(100)[98:2:-2]
#(98 96 94 92 90 88 86 84 82 80 ...)
clpython(216): (+ 1 2)
3

It supports multi-line Python statements as long as the next lines are properly indented:

clpython(70): for i in range(4): print i, print i*2
0 0 1 2 2 4 3 6

See the documentation for macro enter-mixed-lisp-python-syntax for all "mixed-mode" options, and also see Handling of errors.

This mode can also be enabled in Lisp source files:

TODO

You may find this mixed-mode handy for Python development using the Slime IDE.

[back to top]

2.8 Handling of errors

In case an unhandled error happens during evaluation of Python code, the behaviour of CLPython is different from standard Python. You may find yourself suddenly in the Lisp debugger without a clear idea of what is happening or how to continue.

In standard Python, when an unhandled error occurs the error message is printed and the stack is unwound:

123 + 4/0
Traceback (most recent call last):
File "", line 1, in
ZeroDivisionError: integer division or modulo by zero

CLPython implements Python exceptions as Common Lisp errors (see Python object representation). The Common Lisp way to deal with errors is:

  1. The error is reported to the user;
  2. Options are presented on how to proceed: so-called "restarts";
  3. The user chooses an option by "invoking a restart" and execution will continue.

The same error in the CLPython REPL:

123 + 4/0
ZeroDivisionError: Attempt to divide 4 by zero.

Restart actions (select using :continue):
0: Use another divisor instead of zero
1: Retry evaluating this top-level form in module __main__' 2: Skip this top-level form in module main'
3: Retry the expression: "123 + 4/0" (:re).
4: Return to Python top level (:pt).
5: Return to Top Level (an "abort" restart).
6: Abort entirely from this (lisp) process.
[1] cl-user(30):

The execution is now paused, not aborted. The most interesting restarts in the list are:

The exact way to invoke a restart is implementation-dependent:

[back to top]

3. Accessing Lisp from Python

3.1 Lisp functions

In the Mixed-syntax Lisp/Python mode you can access Lisp functions in Python code directly:

clpython(26): vectorp( [1,2,3] )
t

clpython(27): vectorp( (1,2,3) )
nil

If the Lisp name contains dashes, which makes them invalid Python identifiers, replace the dashes by underscores in the Python code:

clpython(28): make_hash_table
#

clpython(29): make_hash_table()
#<eql hash-table with 0 entries @ #x218e1a82>

An example use of calling print-object on a Python object and the Python standard output:

clpython(30): print_object
#

clpython(31): import sys
#

clpython(32): sys.stdout
#<py-hash/py-==->lisp-val hash-table with 0 entries @ #x21678432>

clpython(33): print_object( {}, sys.stdout)
#<py-hash/py-==->lisp-val hash-table with 0 entries @ #x219305a2>

[back to top]

3.2 Lisp classes

TODO

[back to top]

3.3 Lisp packages

TODO

[back to top]

4. Accessing Python from Lisp

4.1 Python values and functions

In the Mixed-syntax Lisp/Python mode you can access Python values (like global variables, functions, and modules) from Lisp directly:

clpython(70): def py(n): print "python got: %s" % n
#<python-function py (interpreted) (__main__/py) @ #x218e6b52>

clpython(71): (py (+ 1 2))
python got: 3

For Python expressions that are not a simple identifier there is the ~ (tilde) reader macro. Think of the tilde as a snake! It works for attribute lookup:

clpython(72): import math
#

clpython(73): (list cl:pi ~math.pi)
(3.141592653589793d0 3.141592653589793d0)

And it works for object subscription:

clpython(74): x = [1,2,3]
#(1 2 3)

clpython(75): (list ~x[1])
(2)

[back to top]

4.2 Python class instantiating

Section § 4.1 showed how to access Python values including class objects. Python classes are instantiated using py-call:

clpython(76): class C: pass
#<class C @ #x21eca942>

clpython(77): (setq x (py-call ~C))
#<C @ #x21f1d9e2>

clpython(78): x.a = 3

clpython(79): print x.a
3

[back to top]

9. Compiler implementation details

9.1 Python object representation

Python objects are represented by an equivalent Lisp value where possible, and as CLOS instances otherwise:

Python data type Representation in CLPython
Class CLOS class
Instance of user-defined class CLOS instance
Exception Condition
Function Function
Generator function Closure
Dict Hashtable
(Unicode) String Unicode string
List Adjustable vector
Tuple Consed list
Long, Integer, Boolean Integer
Float Double-float
Complex Complex

[back to top]

9.2 Compilation by macroexpansion

CLPython first translates Python code into an abstract syntax tree (AST), and then translates the AST into Lisp code. A lot of work is carried out by macros. Example:

if 4 > 3: print 'y' else: print 'n')

This is parse into the following abstract syntax tree:

clpython(1): (defvar source " if 4 > 3: print 'y' else: print 'n'")
source

clpython(2): (defvar ast (parse source))
ast

clpython(3): (with-ast-user-pprinter () (prin1 ast) (values))
([module-stmt] ([suite-stmt] (([if-stmt] ((([comparison-expr] [>] ([literal-expr] :number 4) ([literal-expr] :number 3)) ([suite-stmt] (([print-stmt] nil (([literal-expr] :string "y")) nil))))) ([suite-stmt] (([print-stmt] nil (([literal-expr] :string "n")) nil)))))))

Here [if-stmt] is a macro that expands a Python if statement into a Lisp cond form:

(with-ast-user-pprinter () (prin1 (macroexpand-1 '([if-stmt] ((([comparison-expr] [>] ([literal-expr] :number 4) ([literal-expr] :number 3)) ([suite-stmt] (([print-stmt] nil (([literal-expr] :string "y")) nil))))) ([suite-stmt] (([print-stmt] nil (([literal-expr] :string "n")) nil)))))) (values))
(cond ((clpython::py-val->lisp-bool ([comparison-expr] [>] ([literal-expr] :number 4) ([literal-expr] :number 3))) ([suite-stmt] (([print-stmt] nil (([literal-expr] :string "y")) nil)))) (t ([suite-stmt] (([print-stmt] nil (([literal-expr] :string "n")) nil)))))

The expansion of [if-stmt] thus contains a call to function py-val->lisp-bool. Expansion of the [comparison-expr] form includes a call to py->, and the expansion of [print-stmt] calls py-print.

These functions implement the Python semantics for truth, comparison and printing, which can be hairy at times. They are part of the CLPython runtime. The generated Lisp code is thus not self-contained and independent, but requires CLPython to be loaded every time the code is executed.

[back to top]

10. Completeness and compatibility

10.1 Compatibility target

When it comes to compatibility, the moving target is the most recent 2.x release of CPython, which is currently Python 2.7.1.

[back to top]

10.2 Python-the-language

Many commonly used language features are implemented, including:

Missing:

[back to top]

10.3 Built-in functions

The following built-in functions are implemented:

TODO

Missing:

TODO

[back to top]

10.4 Built-in classes and methods

TODO

[back to top]

10.5 Built-in modules

TODO

[back to top]

10.6 Compatibility with CPython C extensions

There are many extension libraries written in C for the standard Python. One example is NumPy. These modules can not currently be used in CLPython, as CLPython does not implement the Python C API. Interesting is that IronPython, the .Net implementation of Python, has started to support C extensions with a component called IronClad, with apparently most of NumPy already working (implementation details). That they got it working is a nice surprise. But I don't know yet if their approach would work for CLPython.

Other extensions are based on the ctypes module. It seems doable to port this module using CFFI.

[back to top]

11. Contributing

11.1 CLPython source on Github

The source code of CLPython is hosted in a repository at GitHub. There you can easily clone the project, start hacking, share patches, and report issues.

[back to top]

11.2 Announcement mailing list

To be informed about new releases, please subscribe to clpython-announce.

[back to top]

11.3 Developement mailing list

Please send questions and feature requests to clpython-devel.

[back to top]

11.4 Bug tracker

Please report issues in the issue tracker at GibHub.

[back to top]

11.5 Task: Implementing builtin modules

Probably the first reason some Python code does not run on CLPython, is that some built-in module is partly or completely missing. Extending suppport for built-in features is a task that can always use help.

[back to top]

11.6 Task: Porting CLPython

The intention is to support all Common Lisp implementations over time. Many implementations are already supported. Implementations that are not yet supported are e.g. ABCL and CLISP.

The first requirement on new platforms is that the libraries Closer to MOP and CL-Yacc can be run on them. Porting CLPython can then best started by just loading the system, resolving the roadblocks as they appear. The parts in the code prefixed by reader conditionals (like #+allegro) are obvious cases that need to be extended to support other Lisp implementations. Once CLPython can be loaded successfully, running the test suite without unexpected errors is the next goal.

Please mention porting attempts and progress on clpython-devel.

[back to top]

11.7 Task: Supporting C Extensions

Your help is welcome in investigating whether CLPython can support C extension modules, see Compatibility with CPython C extensions.

[back to top]