Testing reproducibility

Each SEP report is tested for reproducibility. Each paper is tested in four categories: ER reproducibility, CR reproducibility, clean rules, and independence. ER figure reproducibility is tested by deleting all ER figures and attempting to rebuild them using the author's makerules. All CR figures are checked to make certain that they do have accurate makerules. The clean rule check means that no files except the ER figures should be created in the build process, no files other than the results exist in the results directory, and no object or binary files exist after the clean rules are run. Finally, no makerules may attempt to access binaries outside of the paper directory. These requirements are considerably stricter than they have been in past years.