Literate Programming in XML

In a nutshell, literate programming allows an author to develop a program as a logical set of “program fragments”. Each program fragment contains some small, functional unit of thoroughly documented processing logic. Program fragments can refer to each other, allowing the author to construct complex fragments from simple ones. The author is free to organize these fragments in any order that facilitates their documentation.

The web of fragments and fragment references forms the complete program. Given a starting point, the web can be tangled together into an executable program (with the program fragments reordered to suit the requirements of the language processor rather than the requirements of prose exposition) or woven into documentation.

The sophistication of tangle and weave varies between literate programming systems. Some tangle processors, for example, take care to adjust identifier names so that they will be both short and unique (this was a necessity for the original Web system where the language processor was a Pascal compiler that did not accept identifiers longer than seven characters). Knuth's tangle also performed a systematic source-code “uglification” to discourage maintainance programmers from working directly on the tangled code instead of the Web sources. Similarly, weave programs can provide quite sophisticated facilities such as pretty printing of the source code fragments and the generate of special indexes and concordances.

Typical literate programming systems are quite complex. They are built on top of some underlying documentation system (such as TeX) and described in terms of the macros and other documentation markup required to describe an xweb document.

Indeed, my first forays towards an XML literate programming system were built with the expectation that DocBook would be the underlying documentation format. However, it quickly became apparent that this was unnecessary. By stipulating that the documentation format would include two new (namespaced) elements, and the xweb just a few more, it was possible to implement literate programming in XML on top of any format that the author chooses: DocBook, TEI, XHTML, you name it.

This is possible because a literate programming system only needs to be able to do four things:

XML makes each of these tasks easy.

The literate programming system described in this paper uses two elements to achieve these ends: src:fragment identifies a fragment and src:fragref identifies a cross-reference to a fragment.

The src:fragref uses XML ID/IDREF to point to the src:fragments. The fragment where processing begins is identified by an ID. The default ID is “top”.

Figure 1 shows a literate programming example^[2]. It is a Perl program for calculating members of the Fibonacci series.

<article xmlns:src="http://nwalsh.com/xmlns/litprog/fragment"
         xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<articleinfo>
<title>Calculating Members of the Fibonacci Series</title>
<author>
  <firstname>Norman</firstname>
  <surname>Walsh</surname>
</author>
</articleinfo>

<para>This trivial document describes a simple, recursive
implementation of the Fibonacci series in Perl. The principal
motivation for this document is to demonstrate the use of Literate XML.
</para>

<section><title>Recursive Definition</title>

<para>The Fibonacci series begins: 1 1 2 3 5 8 13... Each member of
the series, after the first two, is the sum of the preceding two
members.</para>

<para>This can be implemented recursively by calculating the preceding
two members of the series and returning their sum:</para>

<src:fragment id="sub.fib.recursion">
&amp;fib($n-2) + &amp;fib($n-1);
</src:fragment>
</section>

<section><title>The <function>fib</function> Function</title>

<para>The heart of this program is the recursive function that
calculates the members of the Fibonacci series.</para>

<para>The first and second members of the Fibonnacci series are
<quote>1</quote>; all other values are calculated recursively.</para>

<src:fragment id="sub.fib">
sub fib {
    my $n = shift;

    if ($n &lt;= 2) {
        return 1;
    } else {
        return <src:fragref linkend="sub.fib.recursion"/>
    }
}
</src:fragment>
</section>

<section><title>Code Preamble</title>

<para>The program preamble simply establishes a default location for
the Perl executable and informs the interpreter that we want to use
the strict pragma.</para>

<src:fragment id="preamble">
#!/usr/bin/perl -w

use strict;
</src:fragment>
</section>

<section><title>Argument Checking</title>

<para>This program expects its argument on the command line and it expects
that argument to be an unsigned decimal integer.</para>

<src:fragment id="argcheck">
my $num = shift @ARGV || die;

die "Not a number: $num\n" if $num !~ /^\d+$/;
</src:fragment>
</section>

<section><title>The Program</title>

<para>The program prints out the Fibonacci number requested:</para>

<src:fragment id="top">
<src:fragref linkend="preamble"/>
<src:fragref linkend="argcheck"/>

print "Fib($num) = ", &amp;fib($num), "\n";

<src:fragref linkend="sub.fib"/>
</src:fragment>
</section>

</article>

This xweb can be tangled into Perl, as shown in Figure 2, or woven into DocBook that can be subsequently transformed into HTML, or PDF or used for any other sort of DocBook processing.

#!/usr/bin/perl -w

use strict;


my $num = shift @ARGV || die;

die "Not a number: $num\n" if $num !~ /^\d+$/;


print "Fib($num) = ", &fib($num), "\n";


sub fib {
    my $n = shift;

    if ($n <= 2) {
        return 1;
    } else {
        return 
&fib($n-2) + &fib($n-1);

    }
}

An xweb file can be written in any XML vocabulary (or mixture of vocabularies) that is convenient for the author. The system described in this paper relies on the ability to add a few new elements to the markup vocabulary from the “http://nwalsh.com/xmlns/litprog/fragment” namespace. In this article, we use the namespace prefix “src:” to identify elements from that namespace.

The most important elements are src:fragment and src:fragref.

<src:fragment id="preamble">
Some lines of code.
Some more code.
</src:fragment>

<src:fragment id="preamble">Some lines of code.
Some more code.</src:fragment>

In the past few years, the number of XML vocabularies has exploded. Where there used to be just a few, there are now hundreds. In addition, many of these new vocabularies have all sorts of sophisticated processing expectations: XSLT, W3C XML Schema, RELAX NG, Schematron, SAML, SVG, and MathML just to name a small handful.

Luckily, Literate Programming with XML applies equally well to XML, so it is possible to apply a literate programming methodology to the development of XML vocabularies.

In fact, the system described by this paper has been used not only to develop itself, but also in the development of the DocBook XSL Stylesheets. In fact, one of the motivating factors in the development of this system was to provide a better mechanism for building and documenting the DocBook stylesheets.

Writing Literate XML is easy. Figure 3 shows a a simple W3C XML Schema implemented in the literate style. This example uses XHTML as its documentation vocabulary. Like Figure 1, this xweb can be tangled into XSD or woven into documentation that can be subsequently transformed into HTML.

Weaving is still a two step process because weave produces an XML document that contains embeded src:fragment and src:fragref elements. Even in the case where the surrounding XML markup is XHTML, these elements need to be styled. It would be relatively easy to make a one-step weave stylesheet for the special case where the source was XHTML, but that would need to be adjusted for each user that wanted to change the way fragments are formatted. The burden of maintaining these extra weave stylesheets seems to outweigh the convenience.

<html xmlns="http://www.w3.org/TR/xhtml-basic"
      xmlns:xs='http://www.w3.org/2001/XMLSchema'
      xmlns:src="http://nwalsh.com/xmlns/litprog/fragment"
      xmlns:ex='urn:publicid:-:Norman+Walsh:Schema Example:EN'>
<head>
<title>Document Schema</title>
</head>
<body>

<div>
<h1>A Simple Document W3C XML Schema</h1>

<p>This schema defines the
<tt>urn:publicid:-:Norman+Walsh:Schema Example:EN</tt> namespace
by defining several elements and their complex types.
</p>

<p>Documents that conform to this schema have the general form:</p>

<pre>&lt;doc xmlns="urn:publicid:-:Norman+Walsh:Schema Example:EN">
  &lt;title>Sample Document&lt;/title>
  &lt;para>Some paragraphs.&lt;/para>
&lt;/doc></pre>
</div>

<div>
<h1><a name="types"/>The Types</h1>

<p>This schema only defines three element types: <tt>doc</tt>,
<tt>title</tt>, and <tt>para</tt>.</p>

<div>
<h2>The <tt>doc</tt> Type</h2>

<p>This is a document.</p>

<src:fragment id="doc.type">
  <xs:complexType name='doc'>
    <xs:sequence>
      <xs:element ref="ex:title" minOccurs='0' maxOccurs='1'/>
      <xs:choice minOccurs='1' maxOccurs='unbounded'>
        <xs:element ref='ex:para'/>
      </xs:choice>
    </xs:sequence>
  </xs:complexType>
</src:fragment>
</div>

<div>
<h2>The <tt>title</tt> Type</h2>

<p>This is a title.</p>

<src:fragment id="title.type">
  <xs:complexType name='title' mixed="true">
    <src:fragref linkend="role.attrib"/>
    <xs:anyAttribute namespace="##other" processContents="lax"/>
  </xs:complexType>
</src:fragment>
</div>

<div>
<h2>The <tt>para</tt> Type</h2>

<p>This is a paragraph.</p>

<src:fragment id="para.type">
  <xs:complexType name='para' mixed="true">
    <src:fragref linkend="role.attrib"/>
    <xs:anyAttribute namespace="##other" processContents="lax"/>
  </xs:complexType>
</src:fragment>
</div>
</div>

<div>
<h1>The <tt>role</tt> Attribute</h1>

<p>Each of the complex types in this schema allows an optional role attribute.
The role attribute is simply a string.</p>

<src:fragment id="role.attrib">
    <xs:attribute name="role" type="xs:string"/>
</src:fragment>
</div>

<div>
<h1>The Elements</h1>

<p>This schema defines one element of each <a href="#types">complex
type</a>.</p>

<src:fragment id="elements">
  <xs:element name="doc" type="ex:doc"/>
  <xs:element name="para" type="ex:para"/>
  <xs:element name="title" type="ex:title"/>
</src:fragment>
</div>

<div>
<h1>The Schema</h1>

<p>The schema wrapper surrounds all these definitions.</p>

<src:fragment id="top" mundane-result-prefixes="ex xs">
<xs:schema xmlns:xs='http://www.w3.org/2001/XMLSchema'
           xmlns:ex='urn:publicid:-:Norman+Walsh:Schema Example:EN'
           targetNamespace='urn:publicid:-:Norman+Walsh:Schema Example:EN'
           elementFormDefault='qualified'>
  <src:fragref linkend="doc.type"/>
  <src:fragref linkend="title.type"/>
  <src:fragref linkend="para.type"/>
  <src:fragref linkend="elements"/>
</xs:schema>
</src:fragment>
</div>

</body>
</html>

<src:fragment id="top">
<src:passthrough>&lt;!DOCTYPE foo PUBLIC "foo" SYSTEM "http://example.com/foo"&gt;
</src:passthrough>

<foo>...</foo>
</src:fragment>

<src:fragment id="equation">
  <xsl:template match="eq">
    <m:math xmlns:m="http://www.w3.org/1998/Math/MathML">
      <xsl:apply-templates/>
    </m:math>
  </xsl:template>
</src:fragment>

<src:fragment id="equation">
  <xsl:template xmlns:xsl="http://www.w3.org/1999/XSL/Transform" match="eq">
    <m:math xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
            xmlns:m="http://www.w3.org/1998/Math/MathML">
      <xsl:apply-templates xmlns:xsl="http://www.w3.org/1999/XSL/Transform"/>
    </m:math>
  </xsl:template>
</src:fragment>

<src:fragment id="equation" mundane-result-prefixes="xsl">
  <xsl:template match="eq">
  ...

<src:fragment id="equation">
  <xsl:template match="eq">
    <m:math xmlns:m="http://www.w3.org/1998/Math/MathML">
      <xsl:apply-templates/>
    </m:math>
  </xsl:template>
</src:fragment>

The system is quite small, it consists principally of three XSL Stylesheets. Literate Programming with XML is implemented using literate programming. Consequently, each of these stylesheets is available as a xweb document and as published documentation.

There are some additional support tools for DocBook-based xwebs:

Weaving is simply a matter of processing the xweb document with the appropriate variant of the weave stylesheet. The resulting XML document can then be processed by whatever documentation processing system you choose.

Tangling is about the same, except that if the top fragment in your xweb is not identified with the id “top”, you will need to pass the appropriate value in the $top parameter.

If you're interested in downloading these tools, see http://docbook.sourceforge.net/projects/litprog/.

Several people saw early drafts of this work and made many insightful and helpful comments. In particular, I would like to thank Tony Coates, Chris Maden, Eve Maler, and Michael Sperberg-McQueen.