CD-ROM technology has remained static from 1990 till now while web browsing technology has exploded. Economics now favors the web. SEP has learned how to deliver reproducible research on CD-ROM but not yet on the web.
It may seem strange to put the author's own name at the top of the list to whom we wish to provide the reproducible research, but it often seems that the greatest beneficiary of preparing the work in a reproducible form is the original author! Authors lose reproducibility of their work by the passage of time exacerbated by their departure from Stanford.
The CD/reproducibility effort has two parts: The first part is defining and learning the standards for reproducible research. The second part is assembling and building the CD-ROM. These efforts are independent. With the system we have in place, the researcher does almost none of the CD-ROM work beyond the work required to prepare reproducible research and a paper document. The researcher only copies his/her work to a staging area.
Space and ``read-only'' limitations may also create a few extra minor chores. The CD-ROM being ``read only'' motivates enforcing the expectation (which is good practice anyway) that researchers have in place a rule for removing all ``intermediate'' files, namely, those files that are created by programs from other files. This clean up is required for the CD-ROM to enable push-button reproducibility. It also makes revisiting an old directory less onerous for humans. The required existance of a makefile with a clean rule for intermediates and a burn rule for targets defines the test for reproducible research.
We have had problems with our CD-ROMs not working on many systems. Our CD-ROMs have source code, documents, a few data sets, fonts, word processing software, and binary programs that we hoped would work on popular computers as well as all workstations manufactured by our sponsoring companies. The problems that we have had include
We might be able to cut costs by not mastering the CD-ROMs but instead making many singles. We could get by with about 45 singles (one for each sponsor and one for each researcher). I will hope about 50 singles would cost about $40/each for about $2000. That assumes we do not buy equipment but continue to use outside suppliers. Lower single incremental costs would apply if we purchased equipment at about $2-3000, and UNIX software estimated at $5-10,000.
For $2500 we could purchase a 9Gb disk and half fill it with our 10 previous CD-ROMs. Many machine dependent binaries and fonts could be thrown away, meaning the cost for keeping this info on the web would be about $1500 for the lifetime of the disk. The disk should be made read only. Material could be partitioned into public areas and restricted areas. We should probably do this anyway, regardless of our CD-ROM plans.
People would not be able to directly do reproducible work on our archival read-only disk file system. They would need to make copies and work from the copies. Naturally, the difficult chore of building a complete environment, as done on our CD-ROM needs to be done if reproducibility is to be achieved. We have not demonstrated a prototype for such local reproducibility but it seems possible. For people at remote sites to be able to build the required environment is even more problematic but not an unrealistic goal.
My goal is for people who have graduated and left Stanford for a couple of years to be able to rebuild the illustrations in their thesis. Turns out that by the time you reach that goal you have approximately met the goal of providing reproducibility for colleagues and sponsors and the general public. The University long-range position is that the research results should be reported in such a way that the work is reproducible by others. SEP is the first place where we have interpreted this quite literally.
We have had sponsors request web serving in preference to the CD-ROM. We don't know how many sponsors have web access but we are certain that some do not. In a few years we expect that most will have web service and we can supply the remaining with magnetic tape.