Testing of documents

Here I suggest a suite of tests which an author should run on each chapter after finishing his research (or maybe even while he is building his new document). The same tests are run when the document is tested for a cdrom.

• make clean removes all intermediate files. Make sure clean visits all subdirectories. Intermediate files not removed by clean will prevent a successful reproduction of your results on a read-only cdrom. Take a careful look if all intermediate files have been removed. Be careful when using someone else's: what he considers junk files maybe precious to you. Make sure you remove all executable files.
• Wastestat looks for files which are junk. If you find a file that is not needed, then remove it. If it is an intermediate file generated by your clean rule, then include it into your clean rule. I suggest you run the different options of Wastestat separately, so that you know why the script considers the file a junk file.
• Execstat lists all executables in your chapter. Your chapter should probably not include any executables but source code. The reader creates the needed executable when needed.
• Resstat computes three lists: figures in your RESULTS statement, figures in your paper.tex plot statements, and files in your figure directory Fig. It then compares the three lists for consistency. At this point of our testing suite, Resstat should not show any inconsistencies and the summary statement at then end should count all figures.
• make burn burns all reproducible figures at once.
• Resstat lists now all reproducible figures as missing. The summary statistics should list no reproducible figures.
• make build up-dates all figures and rebuilds the missing ones. This command should finish without interference by you. A common mistake is to display a figure after having reproduced it. This prevents the command from running efficiently in a batch-mode as it is necessary when testing an entire document: reproducing and displaying a figure should be done by two separate make rules.
• Resstat should yield a result identical to the first Resstat: no inconsistencies.
• make clean removes all intermediate files.
• make build checks all files if they are up-to-date. No action should result from this command. A common problem arises when your hierarchy of make rules contains a rule without dependencies. All the files depending on such a rule are always out-of-date.

Once you have tested each chapter, you can test the entire document using the script Tour (see selfdoc) and a similar cycle of commands as above. Try to write your make rules so that they produce little or no output when they execute successfully. A small output log during testing is much easier to check for problems.

The structures and tests described in the previous paragraphs help develop any self-contained reproducible and interactive document. The observations are independent from the documents medium such as on-line, cdrom, or on-the-web. To store such a document on a read-only cdrom adds a few constraints which I will discuss next.