REPRODUCIBLE COMPUTATIONAL RESEARCH:
A history of hurdles, mostly overcome

I discovered reproducibility in computational research when I learned about makefile syntax and how to use it to incorporate figures in documents. Here I summarize the reproducibility obstacles I faced writing textbooks, teaching reproducibility, how SEP has set up it's reproducibility rules, and how it uses them. An unanswered question is what we can do to enable reproducible research to spread more widely throughout the community. The next "killer application" will be "reproducible lectures."

EXPERIENCE WITH BOOKS

FGDP, 1976

My first book, FGDP, was published in 1976. It was produced by a big company from my typewritten manuscript. Most illustrations were made by a draftsman. I had the good fortune to have access to a photographic seismic section plotting machine at Chevron. With access to that, my book additionally contained numerous wave propagation illustrations, a spectacular advance for the time. Each figure was a treasure. I never thought of reproducing anything. Having gotton it once on paper -- that was enough.

IEI, 1985

My second book was published in 1985. By then SEP was well underway. Much different than the first book, my second book benefited immensely from the many students at SEP.

Rob Clayton

introduced SEP to the typesetting software troff. He also wrote the first parameter fetching program, getpar. Rob wrote our first program for plotting on a raster plotter. Rob and

Jeff Thorson

assembled plotting programs for many long forgotton devices. These programs were device specific. When a new plotting machine came in, a new plot program needed to be written.

Dave Hale

assembled the first plot library called vplot. Now plots could actually be saved in a file, and shown again later on a different plotting machine. Dave also wrote us a contour plot program still in use today.

Rick Ottolini

wrote innumerable plot programs leading to his ultimate 3-D viewer. Some sponsors joined SEP just to get access to the Rickmovie program. This program is still the heart of our daily research. Rick also wrote a "Window" program for taking subsets of seismic data. Rick's Window program distinguished input data and output data using different parameter names. This drove me crazy so,

Jon Claerbout (me)

constructed the first version of SEPLIB. Early versions of SEPLIB were so ideosyncratic that the senior students at SEP refused to touch it. Luckily,

Stew Levin

cleaned up SEPLIB thereby making it work in a reliable manner and adding many new features.

Joe Dellinger

devised the first vplot interpretation tool that could direct vplot files to various devices. He also extended the vplot language to include raster files so we could show seismic sections as variable brightness, not just wiggly lines. Joe's vplot filters have been the basis for all SEP publication for maybe two decades now. Hooray for Joe!

One thing Joe missed (which I failed to properly recoup) was the notion of a plot matrix. Think of tables in html. A table in html can contain text or an image or another table. Html knows how to resize and reshape a table.

Another thing Joe missed (because it hadn't been invented yet) is user interaction with plots.

Shuki Ronen

wrote a plot program that had hidden lines.

John Etgen's wife Jenny

organized our plot programs into a suite that all understood the same definitions of inputs.

I mention all these people because of the debt I owe them, and because they all contributed in a major way to each of us being able to reproduce our own work. We had not yet reached the higher goal of wanting to enable others to reproduce our work.

PVI, 1991

When I returned from sabbatical leave in 1988 I saw the students had figured out how to automatically include figures in the typesetting language. Hooray!

Kamal AlYahya

introduced SEP to the LaTeX word processing system.

Steve Cole

wrote the filter that converted vplot to postscript. When we at SEP wrote a plot, we could view it both on a screen and on paper. Hooray! This made us the envy of almost everyone else. Much later, users of Matlab and Mathematica got this capability.

Biondo Biondi

finished my saw and sat preprocessors that gave allocatable memory and seismic data I/O features to Fortran 77 users.

Dave Nichols

wrote the filter that converted vplot to Xwindow. He also introduced cake to simplify the maintenance of our growing numbers of computers. This variation on the UNIX make utility was a great step forward in logic. Using this tool I first grasped the concept of reproducible research. Change anything, a program, a parameter, a data set, words in a document; type cake, sit back and watch while changed figures are rebuilt and inserted in your changed document.

Martin Karrenbach

and I came up with the basic four rules for research reproducibility (burn, clean, build, view). Martin also assembled our first research reports on CD-ROM. In those days computer memories were smaller than a CD-ROM; a CD-ROM is a read-only memory; so Martin had many issues to address. Martin also set up SEP's first web server. It was 1994. In the many years since 1994 we have had books and reports on the web, but we have never learned how to extend our reproducibility features over the web.

GEE

My current book, GEE, in the making for about 12 years is not intended for formal publication as the web is more practical. More economical for the user, and permits me continual upgrades. With time, building it has become more fragile. I feel with time my books will become unbuildable in their present form. Bob Clapp and Sergey Fomel (see below) could explain this.

SOFTWARE

XTEX

Martin and Dave Nichols installed and adapted some software named XTEX. In this document viewing program we could push a button and up would pop a window with four buttons for the four basic choices (build, view, burn, clean). You could press burn; refresh the document page and see the document with a blank space where the illustration had been; press build; watch the reconstruction of the figure; and see the document restored. Nowdays with Acrobat we can merely push a button in the figure caption which fires up the view option. This is wonderful feature for lectures.

I am reminded to state what we learned about interactive programs. A user should be able continue viewing from where he, she, or someone else left off from an earlier session.

Interactive programs should always be able to save their state so they can restart. Otherwise, dependence on an interactive program can be a form of slavery (nonreproducible research).

In 1990 (SEP-67) I set for SEP a goal of reproducible documents and results. The basic idea we had is that anyone should be able to reproduce our research results with the software on our CD-ROM, no matter if their computer was a Sun, an HP, or an IBM. Many people thought we were reaching too high. I recall some thought we were braggarts, liars, or crazy.

Reality showed us it wasn't as easy as it seemed. Many things could go wrong. Here just a few that I can remember:

1. You might have a different complier version.
2. Your version of UNIX shell might be different.
3. Your version of Xwindow might be different.
4. You needed to install all of SEP software, starting with the cake program itself.

GMAKE

Although people admired our reproducible research goals, few attempted to adopt them. Matthias Schwab felt our inability to make a big impression on the world was due to our requiring people to first install the free utility cake. Matthias felt we should convert to the standardized building routine GNU make (gmake). There was a defect in gmake, however. It did not have the ability to consider a target to be up-to-date if any intermediate files were absent. Matthias and I convinced Richard Stallman, author of GNU, to make this addition to gmake. Stallman fulfilled our request. Matthias organized all SEP's old reports on the web and with help he hired for a summer, he converted my book PVI from cake to gmake and taught everyone at SEP to use gmake.

Unfortunately, the needs of program maintainers and research archivists do not wholly coincide. Our needed feature later became broken. Only after a long delay were we at SEP able to convince the owners of gmake to repair it.

PEOPLE

Matthias and I wrote an article we were pretty proud of for the magazine, "Computers in Physics," They rejected our paper. The editors said, "Well, if everyone used Microsoft software, reproducibility would be trivial."

That magazine soon went bankrupt, was taken over by IEEE, and it was renamed "Computers in Science and Engineering". They kept our paper in their files and some years later they changed their minds about our article and wanted to publish it. The only problem was, they didn't like one of our figures because it was plotted at low resolution. Could we replot it at a better resolution? By that time Matthias had left, and the answer was, embarassingly enough, "no, I cannot reproduce that figure at higher resolution."

What about the inexperienced and the reluctant?

Converting everyone at SEP to reproducible documents was not a trivial task either. More than once I would need to nag saying, "It's not really for the benefit of other people. Experience shows the principal beneficiary of reproducible research is you the author yourself." I won't name the foot draggers, but there were some pretty valuable people who really didn't want to take the time go along. Part of the solution was simply to graduate the senior students. Another part of the solution is to see to it that we actually do test all the documents coming out of SEP. This could not have been fun. I should thank those who did it. Martin Karrenbach was our first tester. Matthias Schwab took over next. Marie Prucha Clapp tested for many years.

The most common pitfall is people referring to files outside the archival area, for example, files in their home directory. Bob Clapp wrote our reproducibility testing script, so nowdays the testing itself is more automated.

Bob's testing script became increasingly important as more people at SEP worked with truly large data sets. When a project is truly large, we actually do not test the reproducibility, but we do test the presence of the makefile commands to reproduce the result.

Our operational definition of reproducibility

Here is a link to the preface of any recent SEP report page.

Bob Clapp takes over

Bob has been maintaining an increasingly huge software base (in addition to a huge number of machines and other enterprises that I am not fully aware of.)

Sergey Fomel takes over

Sergey has assembled an attractive system for maintenance of documents and software that I cannot begin to describe.

Why doesn't reproducible research spread?

We are deeply and certainly aware of the benefits of our definition of "reproducible research". Additionally, many people have admired it. This raises the question, "Why has it not spread far and wide?" We should discuss this topic. Not only would reproducible research be a huge benefit to the community, but we would benefit from a larger community participating in its evolution.

THE KILLER APPLICATION OF THE FUTURE:
Reproducible presentations

A future is coming that will be much different from the past. I can see it. It is an easy extension from where we are. I don't know why I can't convince the big software companies to provide it. What is it? Reproducible presentations. Imagine you have just given a talk using MS Powerpoint, Adobe Reader, or Apple Keynote. You would like for the talk to be in a package so the package can be put on the web, or on a CD or DVD. What is needed to be stored? Basically three things:

1. The slides.
2. A sound file containing the words you said while showing the slide.
3. The pointer location on the screen while you were talking.

Observe the bandwidth requirements are not onerous. Why do I feel this be so revolutionary? I feel it will put out of business poor teachers all over the world. This is my amazing and very strong claim!

Besides replaying the lecture, viewers should be able to jump around from slide to slide, listening only to slides of interest. You can't do that with video tape. Lecturers would be able to swap out slides or replace the oral stream belonging to any slide. Readers could also gather the slides they wish to return to. Authors could reference slides they wish to quote.

I'm told that since Adobe purchased Flash, that we might hope to see something like this. Additionally, I feel Microsoft inevitably must add this feature to PowerPoint.

Unfortunately, SEP would like a bit more than the above. Some of us make good use of interactive graphics during our talks. For most of us, most of the time, however, all we need is "blink movies" (alternation between two frames, or running a short sequence of frames in a loop). Even now, many of us do blink movies within PowerPoint (awkwardly).