SEP report reproducibility test guideline

Requirement for repro test (scroll down to see the testing result SEP148)

If you have more complex reproducing strategies rather than typing make. You need to include a readme file or put the instructions at the top of make file as comment.

The quality of reproducibility will be evaluated in five aspects:

1. All Code compiles on SEP computing environment

2. flow for CR figures is present in makefile

3. ER figures can be reproduced

4. Makefile is written in a clear/concise/neat manner

5. CR figures can be reproduced.

The last two are optional, but rest are mandatory(everyone needs to pass in order to pass the test). Also we will check that proper reproducibility label are assigned to each figures.

makefile format

It is not rare that your makefile will have dozens of entries. We are not able to check all of these. Therefore, we require the makefile to have several common targets, i.e

1. make burn; burn all the figures in the paper.

2. make clean; burn all figures, and also clean the executables and the intermediate result file (.H).

3. make NR; build NR figures; (this could be a simple copy from a backup folder).

4. make ER; build ER figures

5. make CR; build CR figures

Again, you can back up your figures to a separate folder, but it is not OK to just copy them back to directory Fig in those figures making rules.

FAQs

1. What is the rule of thumb of choosing NR/ER/CR labels for a plot?

ER denotes Easily Reproducible and are the results of processing described in the paper. The author claims that you can reproduce such a figure from the programs, parameters, and makefiles included in the electronic document. The data must either be included in the electronic distribution, be easily available to all researchers (e.g., SEG-EAGE data sets), or be available in the SEP data library2. We assume you have a UNIX workstation with Fortran, Fortran90, C, C++, X-Windows system and the software downloadable from our website (SEP makerules, SEPlib, and the SEP latex package), or other free software such as SU. Before the publication of the electronic document, someone other than the author tests the author’s claim by destroying and rebuilding all ER figures. Some ER figures may not be reproducible by outsiders because they depend on data sets that are too large to distribute, or data that we do not have permission to redistribute but are in the SEP data library.

CR denotes Conditional Reproducibility. The author certifies that the commands are in place to reproduce the figure if certain resources are available. The primary reasons for the CR designation is that the processing requires 20 minutes or more, MPI or CUDA based code, or commercial packages such as Matlab or Mathematica.

NR denotes Non-Reproducible figures. SEP discourages authors from flagging their figures as NR except for figures that are used solely for motivation, comparison, or illustration of the theory, such as: artist drawings, scannings, or figures taken from SEP reports not by the authors or from non-SEP publications.

Please refer to the up-to-date instructions @ vostok: net/server/wrk/sep147/Adm/preface.tex for the most updated details.

2. Do I need to include all the .H files I used in the figures in the local directory?

You are not forbidden from doing that just for backup purpose. However, your makefile flows for building these figures should start from the raw input data file, perform the computations and then generate the figures. The only exception is plotting the raw data, since no computation needs to be performed. Simply grabbing backed-up intermediate results (.H files) and plotting them is not considered a valid reproducible flow, unless it is NR.

3. My makefile rules take a long time to compute, how can I test them more efficiently?

'-n' option in Make would show the user nominally all the cmds GNUmake would execute for this target, but without actually executing them. e.g make -n target.H rather than make target1.H. This would help you diagnose many of the mistakes you might have in your make file in advance. For example, a filename typo in the dependence list.

Tips

1. put a dash '-' at the beginning of the command, if you want Make to ignore the encountered error and continue the flow. For eg. (remove the double quotes below)

buildx:

  "-" mkdir bin
  make bin/exe.x

Testing score sheet SEP 148

code CR Fig Flows ER figs can Makefile CR figs can
Papers: Filename compiles present be rebuilt Clarity be rebuilt note
Paper1 Taylor1
Paper2 Huy1
Paper3 Jason1
Paper4 Musa1
Paper5 yishen1
Paper6 Ali1
Paper7 Ali2
Paper8 yang1
Paper9 yang2
Paper10 Ohad1
Paper11 Chris1
Paper12 Chris2
Paper13 elita1
Paper14 elita2
Paper15 mandy1
Paper16 mandy2
Paper17 adam1
Paper18 adam2
Paper19 sjoerd1
Paper20 sjoerd2
Paper21 sjoerd3
Paper22 xukai1
Paper23 jon1
Paper24 jon2
Paper25 biondo1
Paper26 bob1
Paper27 bob2
Paper28 stew1

Y: Yes/Passed

N: No/Failed

-: Not applicable. (for example, if one paper has no CR figures)

Paper Titles


Paper1: Taylor Dahlke and Jon Claerbout, “Ricker wavelet deconvolution of Western Australia data in the radial/time domain”

Paper2: Huy Le and Stewart A. Levin, “Practically stable unstable orthorhombic finite differences”

Paper3: Jason P. Chang, Sjoerd de Ridder, and Biondo Biondi, “Noise characterization and ambient noise cross-correlations at Long Beach”

Paper4: Musa Maharramov, “One-way extrapolation of seismic waves in elastic isotropic media with mode conversions”

Paper5: Yi Shen, “Wave-equation migration Q analysis (WEMQA)”

Paper6: Ali Almomin and Biondo Biondi, “Tomographic full waveform inversion (TFWI) by successive linearizations and scale separations”

Paper7: Ali Almomin, “Accurate implementation of two-way wave-equation operators”

Paper8: Yang Zhang, Biondo Biondi, and Robert Clapp, “Accelerating residual moveout based wave-equation migration velocity analysis with compressed-sensing”

Paper9: Yang Zhang and Guojian Shan, “Wave-equation migration velocity analysis using partial stack-power-maximization”

Paper10: Ohad Barak and Shuki Ronen, “Applications for rotational seismic data”

Paper11: Chris Leader and Robert Clapp, “Large scale linearised inversion with multiple GPUs”

Paper12: Chris Leader and Biondo Biondi, “Extended image space separation of continuously recorded seismic data”

Paper13: Yunyue (Elita) Li, Biondo Biondi, Dave Nichols, Gary Mavko, and Robert Clapp, “Stochastic rock physics modeling for seismic anisotropy”

Paper14: Yunyue (Elita) Li, “Image-guided WEMVA for azimuthal anisotropy”

Paper15: Mandy Wong, “Handling salt reflection in least-squares RTM with salt-dimming”

Paper16: Mandy Wong, “Least-squares RTM with wavefield decomposition”

Paper17: Adam Halpert, “Efficient velocity model evaluation: 2D and 3D field data tests”

Paper18: Adam Halpert, “Salt delineation via interpreter-guided 3D seismic image segmentation”

Paper19: Sjoerd A.L. de Ridder, “Multicomponent ambient seismic noise correlations and tomography at Valhall - Part I: Characterization and correlation of microseism noise in the ambient seismic field”

Paper20: Sjoerd A.L. de Ridder, “Multicomponent ambient seismic noise correlations and tomography at Valhall - Part II: Group-velocity tomography comparisons between 2004, 2005 and 2010”

Paper21: Sjoerd A.L. de Ridder, “PhD geneaology of Jon Claerbout: Ancestry and legacy”

Paper22: Xukai Shen, “Anti-noise wave-equation traveltime inversion and application to salt estimation”

Paper23: Jon Claerbout, “Ricker-compliant and pseudo-unitary decon”

Paper24: Jon Claerbout, “t-squared gain for deep marine seismograms”

Paper25: Biondo Biondi and Ali Almomin, “Tomographic full waveform inversion (TFWI) by extending the velocity model along the time-lag axis”

Paper26: Robert G. Clapp, “Angle gather recovery using Iterative Soft Thresholding”

Paper27: Robert G. Clapp, “Synthetic model building using simplified basin modeling approach”

Paper28: Stewart A. Levin, “SEP3D Output—A ProMAX/SeisSpace Utility”

sep/internal/reprotest148.txt · Last modified: 2015/05/27 02:06 (external edit)
CC Attribution-Share Alike 4.0 International
Driven by DokuWiki Recent changes RSS feed Valid CSS Valid XHTML 1.0