Table of Contents
SEP report reproducibility test guideline
Requirement for repro test (scroll down to see the testing result SEP148)
If you have more complex reproducing strategies rather than typing make. You need to include a readme file or put the instructions at the top of make file as comment.
The quality of reproducibility will be evaluated in five aspects:
1. All Code compiles on SEP computing environment
2. flow for CR figures is present in makefile
3. ER figures can be reproduced
4. Makefile is written in a clear/concise/neat manner
5. CR figures can be reproduced.
The last two are optional, but rest are mandatory(everyone needs to pass in order to pass the test). Also we will check that proper reproducibility label are assigned to each figures.
It is not rare that your makefile will have dozens of entries. We are not able to check all of these. Therefore, we require the makefile to have several common targets, i.e
1. make burn; burn all the figures in the paper.
2. make clean; burn all figures, and also clean the executables and the intermediate result file (.H).
3. make NR; build NR figures; (this could be a simple copy from a backup folder).
4. make ER; build ER figures
5. make CR; build CR figures
Again, you can back up your figures to a separate folder, but it is not OK to just copy them back to directory Fig in those figures making rules.
1. What is the rule of thumb of choosing NR/ER/CR labels for a plot?
ER denotes Easily Reproducible and are the results of processing described in the paper. The author claims that you can reproduce such a ﬁgure from the programs, parameters, and makeﬁles included in the electronic document. The data must either be included in the electronic distribution, be easily available to all researchers (e.g., SEG-EAGE data sets), or be available in the SEP data library2. We assume you have a UNIX workstation with Fortran, Fortran90, C, C++, X-Windows system and the software downloadable from our website (SEP makerules, SEPlib, and the SEP latex package), or other free software such as SU. Before the publication of the electronic document, someone other than the author tests the author’s claim by destroying and rebuilding all ER ﬁgures. Some ER ﬁgures may not be reproducible by outsiders because they depend on data sets that are too large to distribute, or data that we do not have permission to redistribute but are in the SEP data library.
CR denotes Conditional Reproducibility. The author certiﬁes that the commands are in place to reproduce the ﬁgure if certain resources are available. The primary reasons for the CR designation is that the processing requires 20 minutes or more, MPI or CUDA based code, or commercial packages such as Matlab or Mathematica.
NR denotes Non-Reproducible ﬁgures. SEP discourages authors from ﬂagging their ﬁgures as NR except for ﬁgures that are used solely for motivation, comparison, or illustration of the theory, such as: artist drawings, scannings, or ﬁgures taken from SEP reports not by the authors or from non-SEP publications.
Please refer to the up-to-date instructions @ vostok: net/server/wrk/sep147/Adm/preface.tex for the most updated details.
2. Do I need to include all the .H files I used in the figures in the local directory?
You are not forbidden from doing that just for backup purpose. However, your makefile flows for building these figures should start from the raw input data file, perform the computations and then generate the figures. The only exception is plotting the raw data, since no computation needs to be performed. Simply grabbing backed-up intermediate results (.H files) and plotting them is not considered a valid reproducible flow, unless it is NR.
3. My makefile rules take a long time to compute, how can I test them more efficiently?
'-n' option in Make would show the user nominally all the cmds GNUmake would execute for this target, but without actually executing them. e.g
make -n target.H rather than
make target1.H. This would help you diagnose many of the mistakes you might have in your make file in advance. For example, a filename typo in the dependence list.
1. put a dash '-' at the beginning of the command, if you want Make to ignore the encountered error and continue the flow. For eg. (remove the double quotes below)
"-" mkdir bin make bin/exe.x
Testing score sheet SEP 148
|code||CR Fig Flows||ER figs can||Makefile||CR figs can|
|Papers:||Filename||compiles||present||be rebuilt||Clarity||be rebuilt||note|
-: Not applicable. (for example, if one paper has no CR figures)
Paper1: Taylor Dahlke and Jon Claerbout, “Ricker wavelet deconvolution of Western Australia data in the radial/time domain”
Paper2: Huy Le and Stewart A. Levin, “Practically stable unstable orthorhombic finite differences”
Paper3: Jason P. Chang, Sjoerd de Ridder, and Biondo Biondi, “Noise characterization and ambient noise cross-correlations at Long Beach”
Paper4: Musa Maharramov, “One-way extrapolation of seismic waves in elastic isotropic media with mode conversions”
Paper5: Yi Shen, “Wave-equation migration Q analysis (WEMQA)”
Paper6: Ali Almomin and Biondo Biondi, “Tomographic full waveform inversion (TFWI) by successive linearizations and scale separations”
Paper7: Ali Almomin, “Accurate implementation of two-way wave-equation operators”
Paper8: Yang Zhang, Biondo Biondi, and Robert Clapp, “Accelerating residual moveout based wave-equation migration velocity analysis with compressed-sensing”
Paper9: Yang Zhang and Guojian Shan, “Wave-equation migration velocity analysis using partial stack-power-maximization”
Paper10: Ohad Barak and Shuki Ronen, “Applications for rotational seismic data”
Paper11: Chris Leader and Robert Clapp, “Large scale linearised inversion with multiple GPUs”
Paper12: Chris Leader and Biondo Biondi, “Extended image space separation of continuously recorded seismic data”
Paper13: Yunyue (Elita) Li, Biondo Biondi, Dave Nichols, Gary Mavko, and Robert Clapp, “Stochastic rock physics modeling for seismic anisotropy”
Paper14: Yunyue (Elita) Li, “Image-guided WEMVA for azimuthal anisotropy”
Paper15: Mandy Wong, “Handling salt reflection in least-squares RTM with salt-dimming”
Paper16: Mandy Wong, “Least-squares RTM with wavefield decomposition”
Paper17: Adam Halpert, “Efficient velocity model evaluation: 2D and 3D field data tests”
Paper18: Adam Halpert, “Salt delineation via interpreter-guided 3D seismic image segmentation”
Paper19: Sjoerd A.L. de Ridder, “Multicomponent ambient seismic noise correlations and tomography at Valhall - Part I: Characterization and correlation of microseism noise in the ambient seismic field”
Paper20: Sjoerd A.L. de Ridder, “Multicomponent ambient seismic noise correlations and tomography at Valhall - Part II: Group-velocity tomography comparisons between 2004, 2005 and 2010”
Paper21: Sjoerd A.L. de Ridder, “PhD geneaology of Jon Claerbout: Ancestry and legacy”
Paper22: Xukai Shen, “Anti-noise wave-equation traveltime inversion and application to salt estimation”
Paper23: Jon Claerbout, “Ricker-compliant and pseudo-unitary decon”
Paper24: Jon Claerbout, “t-squared gain for deep marine seismograms”
Paper25: Biondo Biondi and Ali Almomin, “Tomographic full waveform inversion (TFWI) by extending the velocity model along the time-lag axis”
Paper26: Robert G. Clapp, “Angle gather recovery using Iterative Soft Thresholding”
Paper27: Robert G. Clapp, “Synthetic model building using simplified basin modeling approach”
Paper28: Stewart A. Levin, “SEP3D Output—A ProMAX/SeisSpace Utility”