DSLs in Lisp, an example for an advanced DSL development technique in Lisp. (original) (raw)

Rainer Joswig
Hamburg, Germany
Email: joswig@lisp.de

Home > Lisp News > 8. July 2005

DSLs in Lisp, an example for an advanced DSL development technique in Lisp.

I've read the article by Martin Fowler about Domain Specific Languages (DSL). Though he mentions Lisp, he doesn't give a Lisp example. I feel so sorry for developers that they have to use a combination of C# or Java together with XML. The code I see continuously fails to impress me. The code lacks basic attributes I would find important: readability is low, entering the code is a pain, development style is non-interactive, the languages they combine don't really fit together and more. (You could think of online source archives like sourceforge as a hall of shame.) Unfortunately most 'researchers' and software developers nowadays can't read or understand a single line of Lisp.

Below is Martin Fowler's example in a primitive and straight forward Lisp version. I'm taking advantage of methods dispatching on symbols and a code generating macro that makes it easy to create the necessary classes and methods. Note that this is not production quality code, it just shows how you can create a small DSL-like extension to Common Lisp and have it generate classes and methods.

Lot's of Lisp code is using this technique. A prominent example is the CL-HTTP web server. CL-HTTP uses these code generating macros all over the place, combined with consing-free operations. If you want to study applied advanced Lisp coding techniques, CL-HTTP is a good start. You need a bit experience though to understand it. A good Lisp environment helps a lot to not be overwhelmed by complex code. Make the editor, listener and the debugger be your friends. Try to understand interactive and incremental development styles. Unfortunately these development styles are hardly anywhere documented or even explained in detail. I would propose that some next Lisp conference has a track on Lisp development styles with live demonstrations.

There are a few Lisp-based development techniques I find totally amazing and I guess few people have seen them. Here is one that works together with DSLs in Lisp:

Imagine you want to implement some protocol (like POP3) from some RFC document. I've seen the following development style:

Putting parentheses around the specification and make it run.

Open a listener and an editor.
Open the RFC document in the editor and save a copy as a Lisp file.
Read the document.
Now start to put parentheses around the relevant sections, examples and pseudo code snippets. Convert these regions into Lisp-based DSL-code.
Implement the DSL with a few classes, functions, methods.
Define the mapping from the DSL code to the DSL implementation with the necessary macros.
Convert the text of the RFC to comment sections and reuse them for the comments of your DSL implementation classes, functions, macros, ... directly from the RFC text.
Add lots of tracing to the DSL implementation. TRACE is your friend.
Now go to the listener. Connect the listener to a POP3 port.
Enter a Lisp form that talks to the POP3 port. Make this Lisp form print lots of diagnostics to the Listener and return interesting values. You even can let it create Inspector windows with intermediate values. Everytime you enter a POP3 command in the listener and press return, your form takes the input, talks to the POP3 server and returns the results. So you have made the Lisp listener a DSL listener. When your DSL implementation has an error, you will find yourself in the debugger underneath and you can fix your DSL implementation. If your DSL implementation has the necessary error handlers and restarts, you can resume from the debugger and never leave the DSL listener loop.
Now try out the protocol and complete the implementation incrementally piece by piece.

I call this development style: 'putting parentheses around the specification and make it run'. It is only possible because Lisp development systems are giving you the combination of easy DSL integration (due to a programmable programming language), an interactive development system (with REPLs (Read Eval Print Loop) and incremental compilation) and robust error handling (with the condition system). I've also seen very advanced editor (Zmacs, the editor of the Lisp Machine, in this case) usage to manipulate the spec source text and the DSL implementation to evolve during the development phase. Since Zmacs can be extended with macros on the fly, you can also change your Lisp source code with code manipulating Lisp code.

Hint: if you can set up a development session like that, you have a good chance to impress your friends.
Additionally: if you use a presentation based Listener, you can make all your output (dsl statements, dsl implementation objects, tracing output, ...) mouse sensitive. You can also write your own DSL-based command loop in CLIM.

; the example from Marting Fowler in Lisp. The rough version.

(defmacro defmapping (name mapping &body; fields) `(progn (defclass ,name () ,(loop for (nil nil slot) in fields collect slot)) (defmethod find-mapping-class-name ((mapping (eql ',(intern mapping)))) ',name) (defmethod parse-line-for-class (line (class-name (eql ',name))) (let ((object (make-instance class-name))) (loop for (start end slot) in ',fields do (setf (slot-value object slot) (subseq line start (1+ end)))) object))))

; EXAMPLE

(defmapping service-call "SVCL" (04 18 customer-name) (19 23 customer-id) (24 27 call-type-code) (28 35 date-of-call-string))

(defmapping usage "USGE" (04 08 customer-id) (09 22 customer-name) (30 30 cycle) (31 36 read-date))

(defparameter test-lines "SVCLFOWLER 10101MS0120050313......................... SVCLHOHPE 10201DX0320050315........................ SVCLTWO x10301MRP220050329.............................. USGE10301TWO x50214..7050329...............................")

(map nil 'describe (with-input-from-string (stream test-lines) (loop for line = (read-line stream nil nil) while line collect (parse-line-for-class line (find-mapping-class-name (intern (subseq line 0 4)))))))

I made a movie showing how to develop the above example. For the development session I was using LispWorks on my PowerMac G5 (2 * 2.5 Ghz). While later thinking about it I was making the following observation: to write that code I would have been around 30 percent faster on my Apple Quadra (40 Mhz, 68040) using MCL. LispWorks is a productive IDE, too. But there is definitely room for improvement. I'd say LispWorks has more advantages if one writes more complex and larger code. On my MacIvory I would have been around 2 times faster. For larger and and more complex code this could easily go up to 5 times faster development speed. Why is that? The Lisp Machine development environment is just much more focused on making the developer productive and lets him go very short ways from code to documentation to running code. Zmacs is more productive, since you have to leave Zmacs less often and you can reuse code more easily in Zmacs.

Links:

Keywords:

Terms of use: no crawlers, no wget, no site copying, use of pictures and text only with permission. No excessive rss feed checking.