SEP has a lot of computing power and a lot of computers. It is easy to get confused when the senior students start talking about computers, calling them by name as if they were friends. This page will tell you about the computers and printers at SEP, what they are used for, and about the directories you will be working in.
SEP's computers currently are named after moods and lost cities. You'll learn these names gradually. The computers that you needed to know to keep up with some of the conversations around here are:
- vostok, our NFS server, has all of the things that are crucial to make SEP run.
- Because this is our most important machine, under no circumstances should you be sending jobs to vostok, because if it goes down, all of the other machines are in trouble.
- When ssh-ing in to SEP from the outside world, you should connect through vostok and then ssh into another machine to run jobs.
- sthelens is our machine for shuffling around large 3D datasets.
- Because this is our 3D machine, people working with 3D datasets (for their thesis requirements) have priority.
- It also has the Exabyte and SDLT tape drives attached to it, along with 250GB of “data” space and 750GB of scratch.
- If you are bringing data in from a portable hard drive, you should connect it here.
- zapad is our web server.
- The lost cities (pompei, sodom, gomorrah)
- These shared memory machines are the workhorses for the group. Each of them has 16 AMD 2.2GHz x86_64 processors and 32GB shared memory.
- The moods (mad, sad, glad)
- These machines are similar to lost cities, but have fewer processors and less memory. Each machine has eight Intel Xeon 1.8GHz x86_64 processors and 8GB shared memory.
- * GPU machines (tesla1~4, fermi1~4)
- These machines are cpu machines that have GPUs connected to them , teslas have 2 GPUs for each CPU node, fermis have 8 GPUs for each CPU node, GPU code have higher priority on those machines, CPU only code should not be running on those machines.
- The bricks (brick1,brick2,brick3,brick4)
- Our gluster file system is a parallel file system with up to 16TB disk capacity. They are connected to the lost cities through Infiniband and other machines (the moods, sep200s, sep400s and sep500s) through Ethernet.
- The Clusters.
- We currently have 4 clusters, with the following imaginative names:
- sep100s - Donated by Chevron, 34 nodes x 2 (x 2 hyper-threads)Xeon processors, 4GB RAM, 140GB RAID 0 disk, 1Gb Ethenet (sep101-134). [Dead node: sep104]
- sep200s - 32 nodes x 2 (x 2 hyper-threads) Xeon processors, 2GB RAM, 100GB disk, 1Gb Ethernet (sep201-232). Will be replaced in one year
- sep300s - 8 nodes x 2 (x 2 hyper-threads) Xeon processors, 6GB RAM, 100GB disk, 1Gb Ethernet (sep301-308). They are currently turned off, we'll probably reinstall the system and make them up running again.
- sep400s - 32 nodes x 2 Xeon x86_64 processors, 24 x 4GB RAM (sep401-424), 8 x 8GB RAM (sep425-432), 250GB disk, 1Gb Ethernet. [Dead nodes: sep401, sep403, sep422, sep428].
SEP has several printers. You will learn how to maintain them, but first learn their names.
- sephp: laser printer in the computer room
- sephp2: laser printer on the 3rd floor
- gpcolor: department color laser printer in the corner of the 4th floor. You should do your printing from your desktop machine, which should have CUPS installed. An lpr -P sephp file.ps should be all it takes to print a postscript file.
Home directories /homes/sep/[username] is your home directory. You can keep small, important files in it that are difficult to reproduce. Examples are:
- computer code: C, C++, Fortran, Java, etc.
- papers you write
- Makefiles and history files relating to your research Home directory space is limited to 300MB per person. This is done because this directory is backed up semi-religiously. Individual files in your home directory should be less than 2.5MB. You should not have data or executables in your home directory! That is what your personal device is for.
Personal devices /net/server/[username] is your personal device. You can store large files such as data and executables here if you intend to keep them for more than a few weeks.To make space for your executables, you should create a /net/server/[username]/bin directory. In this directory create sub-directories called LINUX (PC running Linux), or “all” (for machine-independent executables like shell scripts,perl, awk, and sed programs).
Scratch space /net/[computer name]/scr[number of disk] is where you can keep large files that only need to be around for a little while. For example, /net/pompei/scr1 is the first scratch device on pompei. The .datapath file tells SEPlib which scratch device to use when you are creating new files. The “.datapath” file exists in your home directory, but you can be override it by placing another “.datapath” file in the current directory (so that you have the binaries on the same machine as the executable, for minimizing network traffic).Be aware that we run eraser scripts that remove files that have not been used for over two weeks. You could also receive a “space hog” notice if you use too much of the scratch device - if you do, clean up!
Work directories Work directories are named in the same manner as scratch directories so /net/koko/wrk1 is the first work directory on koko. These directories are only to be used for projects that you are working on with other people and need “neutral ground” for. For example, the report is always put together in /net/koko/wrk1/sep1??.
- /usr/local/XXX: contains directories trees associated with different software packages. This directory is a link to /linux_local on koko.
- /usr/local/src: location for shared source files
- /our: SEP source code
- /gnu: GNU source code
- /pub: public domain source code
- /their: 3rd party source code that is not publicly distributed
- /net/sepdata and /net/sepdata2: datasets to test your code with
We currently have 2 methods of backing up home directories and personal devices:
- Nightly backups are stored in /backup on koko, and weekly backups are written (on Wednesday nights) to tape, as described in the “backup” section.
- Koko mirrors all of its information to an identical set of internal disks. The home directories are mirrored every night, while the personal devices are mirrored once a week. Remember that if you erase a file that you need, it will be removed from the mirror overnight while the script runs. It is also strongly recommended that you set up a mirroring script on your own machine. This can be done by taking the mirroring script shown here, altering it for your home directory, personal device and desktop machine, and then adding it to the crontab on your desktop machine. You can add it to your crontab by typing the command “crontab -e” and inserting the following line:0 3 * * * python /homes/sep/bill/adminstuff/mirror_personal.py »/dev/nullwhere you change the path of the python mirroring script and the time (currently set for nightly at 3AM) to what best suits you.Again, the important thing to realize is that you should not trust any SEP backup measures, and should have at least one failsafe that is your own responsibility.