History files and datapaths

When you did ``Wiggle < Txx.H | Tube'' before some floating-point data obviously was plotted, but that data was not contained within the file Txx.H. You can verify this yourself by doing ``more Txx.H'', which shows that Txx.H is just a short ASCII file:

Mallick:   joe@montebello   Sat Feb 29 23:59:53 HST 1992
     output() : sets next in="./Data/Txx.H@"
     data_format="xdr_float"
     n1=1024
     n2=20
     n3=1
     esize=4
     o1=0  d1=.002 label1="Time, seconds"
     o2=.1 d2=.1   label2="Offset, kilometers"
     title="Txx"

Files such as Txx.H are called ``history'' files because they document the ``history'' of the data in the associated data file. (They are also called ``header'' files in some documents.) Programs append information onto the history file; it thus contains a history of the programs run (and often of all the parameters specified, so that from the history file you could recreate the file from scratch).

In our example so far only one program has been run. The line

Mallick:   joe@montebello   Sat Feb 29 23:59:53 HST 1992

shows that this particular data originated in its SEPlib incarnation from a program called ``Mallick'', which was run by a ``joe@montebello'' at just before midnight on a Saturday night.

The most important part of the history file is the line

in="./Data/Txx.H@"

The parameter in tells where the data associated with the file Txx.H actually is. This can be a relative pathname (begins with ``./'') as in our example here, or an absolute one (begins with ``/''). The rest of the file gives other important information:
esize=4 indicates the data consists of elements 4 bytes long;
data_format="xdr_float" indicates these 4-byte long elements are in fact machine-independent IEEE floating-point numbers.
n1=1024 n2=20 indicates the data is in a 2-dimensional array with a fast axis 1024 elements long and a slow axis 20 elements long.
o1=0 d1=.002 label1="Time, seconds" indicates the fast axis has dimensions of seconds, with the first element corresponding to time 0, the second element time .002, the third .004, etc.
o2=.1 d2=.1 label2="Offset, kilometers" indicates the slow axis has dimensions of kilometers, with the first element corresponding to an offset of .1 kilometers, the second element .2, the third .3, etc.

So far we have displayed our data without creating any new files at all. We did this by using UNIX pipelines (``|''). These tell the operating system we want to pass the information (both data and parameters) from one program to another, and we don't want to be bothered with having to keep track of any temporary intermediate files ourselves.

We could have done it differently by saving the output of the programs. Let's see how our favorite example

Wiggle < Txx.H | Tube

can be split up into two separate steps:

Wiggle < Txx.H  > Out.H
Tube < Out.H

where Out.H is the output of Wiggle. (Do ``man vplot'' if you are curious what sort of output Wiggle writes.)

OK, you have the history file of the output. But where has the actual data gone? It might be a very fascinating question if you are playing around with files of a size of several 100MBytes, and we all have to face the bitter truth that free diskspace is always limited.

There are three options for directing your output:

By default SEPlib will attempt to put your data files in a subdirectory with your username under the system-default SEPlib ``scratch'' directory. If you tried the commands above and got an error message something like
output(): No such file or directory
Wiggle: output() unable to open output file /sepcd2/scr/joe/_Out.H@
It means your default directory did not exist. Look to see what directory Wiggle was trying to use, and create it if you wish (and have sufficient permissions to do so) using the UNIX command ``mkdir''; then try our example again. Be warned that to keep such ``public'' data areas from filling with junk, they are usually subject to swift and merciless disk mowing.

Alternatively, you can specify where you want the data files to go with a command-line parameter, for example:

Wiggle < Txx.H  > Out.H out=./Data/Out.datafile

but it might get tiring to do this every single time! (For a quick experiment you might want to try the above example with and without the leading ``./'' in the out= argument, and note what in= gets set to in Out.H in each case. Unless the SEPlib output subroutine sees a leading ``./'', it automatically expands output file names to fully qualified paths.)

The best solution is for you to create a personal directory to keep your data in somewhere, and tell SEPlib that's where you want it to put your data by default. Let's suppose you create a directory called ``Data'' under your home directory. You then tell SEPlib to put data files there by doing:

setenv DATAPATH ~/Data/

(Note in this example the leading ~ will get expanded to your full home directory name by the csh before the DATAPATH variable is set.) Remember that binary data files will accumulate in the directory given by your DATAPATH if you are not careful. Make sure to use Rm, not rm, to delete SEPlib files! (Examples of using Rm to remove SEPlib files can be found later in this document.)

Note that the DATAPATH is simply prepended as an arbitrary string to a slightly modified version of the history filename to get a name for the data file, so you probably want your DATAPATH to end with a ``/'', like the example above does. You can also set your datapath by creating a file called ``.datapath'' in your home directory (or in your current directory, with the one in your current directory taking precedence). The .datapath file should contain a line looking something like

datapath=/home/montebello/joe/Data/

Note in the file you have to expand out your full home-directory name yourself.