To organize computational scientific research and hence to conveniently transfer our technology, we impose a simple filing discipline on the authors in our laboratory. A document's makefile includes laboratory-wide standard rules that offer readers these four standard commands: make burn removes the document's result figures, make build recomputes them, make view displays the figures, and make clean removes any intermediate files. Although we developed these standards to aid readers we discovered that authors are often the principal beneficiaries.
In the mid 1980's, we noticed that a few months after completing a project, the researchers at our laboratory were usually unable to reproduce their own computational work without considerable agony. In 1991, we solved this problem by developing a concept of electronic documents that makes scientific computations reproducible. Since then, electronic reproducible documents have become our principal means of technology transfer of scientific computational research. A small set of standard commands makes a document's results and their reproduction readily accessible to any reader. To implement reproducible computational research the author must use makefiles, adhere to a community's naming conventions, and reuse (include) the community's common building and cleaning rules. Since electronic reproducible documents are reservoirs of easily maintained, reusable software, not only the reader but also the author benefits from reproducible documents.
On average, two PhD students graduate each year from our laboratory,
the
Indeed, the problem occurs wherever traditional methods of scientific publication are used to describe computational research. In a traditional article the author merely outlines the relevant computations: the limitations of a paper medium prohibit a complete documentation including experimental data, parameter values, and the author's programs. Consequently, the reader has painfully to re-implement the author's work before verifying and utilizing it. Even if the reader receives the author's source files (a feasible assumption considering the recent progress in electronic publishing), the results can be recomputed only if the various programs are invoked exactly as in the original publication. The reader must spend valuable time merely rediscovering minutiae, which the author was unable to communicate conveniently.
To facilitate efficient technology transfer, our laboratory developed
the concept of a
A makefile contains the commands to build a software package and is a
standard UNIX utility for software maintenance. More
powerful than a simple script of commands, a makefile has a notion that result
files (
The second component in making scientific computations reproducible is
the consistent availability of a small set of standard make rules.
These rules allow a reader to interact with the document
without knowing the underlying application-specific commands or files.
The inclusion of an universal set of rules in every makefile
ensures the rules' consistency across documents and enables authors to
concentrate exclusively on the application-specific part of the makefile.
Furthermore, a central set of rules accumulates the wisdom of a
community on how to organize a reproducible electronic document. Our
laboratory offers about 100 lines of GNU make code (
The third component of a reproducible document system is its naming conventions for all files. These conventions allow a community to formulate universal rules that recognize a file's type and handle it appropriately. For example, a cleaning rule removes all intermediate files. The rule identifies intermediate files based on a community's naming conventions: typically files with suffix .o (object files) or files with the name stem junk (temporary files) are removed. A cleaning rule is important since a cleaned directory is more accessible and inviting to a reader than a cluttered uncleaned one. Furthermore, a cleaning rule saves resources such as disk memory by removing superfluous files. Naming conventions are also needed for result files. The rules for displaying, removing, or recomputing a result file are based on the result's file name. For example, at our laboratory result files (which are invariably figures) can have various formats such as postscript or gif. Our laboratory's naming conventions require the author to indicate the result file's format by a suffix, such as .ps or .gif. Consequently our laboratory can supply a universal format-independent rule for displaying result files: The rule identifies the result file's suffix, concludes the file's format, and invokes the appropriate viewing program such as ghostview for postscript or xview for gif files.
ReDoc rules are easy to implement. An author who already uses makefiles only needs to adhere to the ReDoc naming conventions and include the ReDoc rules to make a traditional document reproducible. Our laboratory distributes its ReDoc rules, this article, and the accompanying example on its World Wide Web site (Schwab and Claerbout, 1996) (see Figure 1). A different community may need to adapt these ReDoc rules to its own peculiarities and naming conventions.
At our laboratory the software readily accessible to any researcher has increased tremendously. Today students commonly take up projects of former students, starting by easily removing and recomputing the original result files. Students who graduated and left our laboratory were able to seamlessly continue their own research at their new locations.
We successfully employed the ReDoc rules in our laboratory's most recent sponsor report (14 articles by 15 authors) and three of Jon Claerbout's textbooks on seismic imaging. These documents contain a total of 483 result files: 276 easily reproducible, 21 conditionally reproducible, and 186 non-reproducible figures. Before publication automatic scripts removed and rebuilt all 276 easily reproducible result files (see Figure 2). We use the same scripts and documents to benchmark computer platforms. Additionally, our laboratory published 12 PhD theses that use an earlier version of the reader interface based on an dialect of make called cake (Somogyi, 1984). These electronic documents are available on CD-ROMs (Claerbout, 1996).
We chose to implement the current reader interface in GNU make, since
it is platform-independent, excels in the efficient maintenance of
even complex software packages, and is equipped with a special
mechanism to handle intermediate files, which we will discuss later
(see section
What is next? Of course, we want to publish our results on the
World Wide Web. The Web conveniently distributes the
combination of reading material for researchers and software for
computers. Ideally, each computed figure in a future World Wide
Web document should be accompanied by a
Unfortunately, popular make dialects generally do not support rules to keep documents simultaneously clean and up to date. These dialects consider a result file out of date when certain intermediate files are missing (for example, GNU make considers a result file out of date if a file is absent whose dependency is formulated by a non-pattern rule). Such treatment of intermediate files is convenient for software maintenance but not suitable for reproducible electronic documents.
The cake dialect of make was designed for document maintenance
rather than for software maintenance. cake assumes
Today's ReDoc rules are able to maintain a document clean and
up to date while being formulated in the very popular GNU make
dialect. At our request, Richard Stallman recently enhanced GNU make
to adequately handle the ReDoc rules' intermediate targets. (Footnote:
GNU make refers to intermediate files as
The author of a reproducible document has to list each result file as either easily, conditionally, or non-reproducible. For every easily or conditionally reproducible result, the author has to supply a rule that generates the result file. Furthermore, the author needs to specify a cleaning rule that removes the intermediate files. The ReDoc rules offer the author a comprehensive default cleaning rule, jclean.
Since the author's rules deal with the document's application, the
effort required of an author is best illustrated by an example. The electronic
version of this article is accompanied by a subdirectory called
SEPINC = ../Rules
include ${SEPINC}/Doc.defs.top
RESULTSER = frog dot
col = 0.,0.,0.-1.,1.,1.
${RESDIR}/frog.ps ${RESDIR}/frog.gif: frog.x
frog.x > junk.pgm
pgmtoppm ${col} junk.pgm > junk.ppm
ppmtogif junk.ppm > ${RESDIR}/frog.gif
pnmtops junk.pgm > ${RESDIR}/frog.ps
objs = copy.o adjnull.o rand01.o wavecat.o \
pressure.o velocity.o viscosity.o wavesteps.o
frog.x: ${objs}
dot.build ${RESDIR}/dot.txt : dot.x
dot.x dummy > ${RESDIR}/dot.txt
dot.view: ${RESDIR}/dot.txt
cat ${RESDIR}/dot.txt
dot.burn:
rm ${RESDIR}/dot.txt
dot.x : ${objs}
clean: jclean
include ${SEPINC}/Doc.rules.red
include ${SEPINC}/Doc.rules.idoc
include ${SEPINC}/Prg.rules.std
The variable RESULTSER contains the list of the document's
easily reproducible results, frog and dot.
The next rule contains the commands to build the postscript and gif version of the frog result. Such a rule is application-specific and cannot be supplied by included default rules. The target names comprise the directory RESDIR, in which the result files reside, and file suffixes (.ps, .gif), which indicate the files' formats. The rule depends on an executable frog.x, which it executes during the computations of the result.
Default rules for compilation and linking of executables such as frog.x are supplied by a shared include file, Prg.rules.std (Footnote: Compilation and link rules are compiler dependent. In the Frog example, we include some generic FORTRAN rules; at our laboratory compilation rules depend on an environment variable indicating the compiler type.). The dependency of the executable on its subroutine object files, as in the case of frog.x, needs to be defined by the author of the makefile, since it depends on the application-specific file names.
In the case of the dot result file, the rules supplied by the author reflect the commands of the reader interface: dot.build creates the result file, dot.view displays it, and dot.burn removes it.
Finally, the target clean invokes the included default target jclean. The targets remove intermediate files based on our laboratory's naming conventions.
An author or reader does not need to know the implementation details of the ReDoc rules to use the rules (most researchers at our laboratory have never inspected the ReDoc rules). But you may wonder how the rules operate and how you may have to adapt the rules for your community's computational environment.
burn: burnER
burnER: ${addsuffix .burn, ${RESULTSER}}
burnCR: ${addsuffix .burn, ${RESULTSCR}}
%.burn:
${foreach sfx, ${RES_SUFFIXES} , \
if ${EXIST} ${RESDIR}/$*${sfx} ; then \
${RM} ${RESDIR}/$*${sfx} ; fi; \
}
The burn target invokes its dependency burnER. The
burnER rule selects the easily reproducible result files for
removal. The burnER rule uses GNU make's built-in function
addsuffix to generate its dependency list. Each entry of
burnER's dependency list is a concatenation of the name of an easily
reproducible file and the suffix .burn. In the Frog
example, burnER depends on frog.burn and dot.burn.
The dependency frog.burn invokes the pattern rule %.burn,
which removes the result files corresponding to the result frog.
At our laboratory a single result name such as frog usually
denotes several result files of identical contents but differing
format, e.g. postscript or gif. The %.burn rule scans a list
of possible suffixes (RES_SUFFIXES = .ps .gif) and removes all
related result files: frog.ps and frog.gif.
Since text result files, such as dot.txt, are rare at our laboratory, the ReDoc rules do not contain laboratory-wide rules for handling them. Consequently the author of the Frog document supplies an explicit dot.burn rule in the makefile. This explicit dot.burn rule overrides the default %.burn pattern rule, which generates postscript result files.
The standard burn rule exclusively removes the easily reproducible result files. A reader can remove any existing, conditionally reproducible result files by invoking make burnCR. A reader can exclusively remove the result files related to the frog result by invoking make frog.burn.
build: buildER
buildER: ${addsuffix .build, ${RESULTSER}}
buildCR: ${addsuffix .build, ${RESULTSCR}}
%.build: ${RESDIR}/%.ps
At our laboratory, almost every result file is a figure that exists in
postscript format. Consequently the ReDoc %.build rule
updates the postscript version of any easily reproducible result,
such as ${RESDIR}/frog.ps. Additional versions of the
result (e.g. frog.gif) are usually generated as a side effect of
the rule that computes the postscript version (frog.ps).
As in the case of dot.burn, the nonstandard text result dot (since it is not a figure) requires the author to supply an explicit dot.build rule.
view : ${addsuffix .view, ${RESULTSALL}}
%.view: FORCE
if ${CANDO_GIF} ; then \
${MAKE} $*.viewgif ; \
elif ${CANDO_PS} ; then \
${MAKE} $*.viewps ; \
else \
echo "can't make $*.viewps $*viewgif";\
fi
RESULTSALL lists all result files and is defined as the
concatenation of RESULTSNR, RESULTSCR, and RESULTSER.
At our laboratory the %.view rule checks for the various formats of a result and chooses the first version the makefile knows how to generate. The variable CANDO contains the return value of a recursive gmake -n call. This return value indicates if that particular version of the result can be built. Having found a version that can be built, the %.view rule invokes another rule (%.viewgif or %.viewps) that updates and displays the result file:
%.viewgif : ${RESDIR}/%.gif FORCE
${XVIEW} ${UXVIEWFLAGS} ${RESDIR}/$*.gif
%.viewps : ${RESDIR}/%.ps FORCE
${GVIEW} ${UGVIEWFLAGS} ${RESDIR}/$*.ps
In the Frog example, the frog.view rule finds a rule for
computing a .gif version of frog. Consequently, it invokes
the gif rule frog.viewgif. In return frog.viewgif executes
xview to display the result file frog.gif. If your
computer system does not support the gif viewer xview, then you
will need to supplement the %.viewgif rule with your own
display command. Alternatively, frog.viewps executes
ghostview to display the result file frog.ps.
clean: jcleanSome authors at our laboratory append the default jclean with a command to remove some additional files that do not adhere to the standard naming conventions. Only very few authors ignore the jclean target (and its communal wisdom) and design their own rule.
Since the author of the Frog example adheres strictly to the ReDoc naming conventions for files, the default jclean mechanism suffices to remove the intermediate files:
jclean : klean.usual klean.fort ;
KLEANUSUAL := core a.out paper.log *.o *.x *.H *.ps *.gif
klean.usual :
@-${TOUCH} ${KLEANUSUAL} junk.quiet
@-${RM} ${KLEANUSUAL} junk.*
FORT_FILES = $(patsubst %.f,%,$(wildcard *.f)) junk.quiet
klean.fort:
@\
for name in ${FORT_FILES} ; do \
if ${EXIST} $${name}.r ; then \
${TOUCH} $${name}.f ; \
${RM} $${name}.f ; \
fi ; \
done
The jclean target uses two methods to identify intermediate
files. The first method, klean.usual, simply removes files whose
names fit one of the rule's name patterns: e.g. the executable
frog.x, or the intermediate bitmap files junk.pgm and
junk.ppm. The second method, klean.fort, removes FORTRAN
files, such as frog.f, if RATFOR versions of the program, such
as frog.r, exist.
We are currently collaborating with Richard Stallman of the Free Software Foundation to develop an alternative, more reliable cleaning mechanism. This alternative mechanism would free the author from naming the intermediate files according to the community's naming conventions. The anticipated cleaning mechanism analyses the makefile's rules and dependencies to identify the intermediate files. Fastidious authors would have the option of automatically removing all files that are neither source files nor result files.
Fig 1: The GNU make code that implements the ReDoc rules, this article, and its example are freely available on our World Wide Web site: http:/sepwww.stanford.edu/pub/redoc/.
Fig 2: A concrete test of a document's reproducibility is a cycle of burning and rebuilding its results. A simple script can implement such a reproducibility test by invoking the ReDoc rules described in this article. The ReDoc rules remove and regenerate the document's results independent of the document's content. The graph above plots the successfully reproduced figures versus the series of tests that removed and rebuilt the figures. The document contained 14 articles with 112 easily reproducible figures by 15 authors. After each test the authors were given time for corrections. After the first test, only 60% of the document's easily reproducible figures were in fact reproducible. After the fourth test, almost all figures were reproducible and the document was published.
Fig 3: The reader interface for reproducible research is only one
component of SEP's current computational research environment: A
research document at SEP is written in LaTeX (visible in the
background to the left). Using SEP's own LaTeX macros, a
push-button in each figure caption invokes a graphic user interface
(written in a script language called
Claerbout, J. F., 1996, CDROMs of the Stanford Exploration Project: http://sepwww.stanford.edu/office/sepcd.html/.
Cole, S. P., and Nichols, D., 1996, The Xtpanel page: http://sepwww.stanford.edu/oldsep/dave/xtpanel/
Oram, A., and Talbott, S., 1991, Managing projects with make: O'Reilly & Associates, Inc.
Schwab, M., and Claerbout, J. F., SEP's Reproducible electronic documents: http://sepwww.stanford.edu/idoc/.
Somogyi, Z., 1984, Cake: a fifth generation version of make: http://munkora.cs.mu.oZ.au/~zs/
Stallman, M. and McGrath, R., 1991, GNU Make: Free Software Foundation.
SUN, 1996, Java: Programming for the Internet: http://java.sun.com/
