Oklahoma OURRstore Users' Guide

This document is a DRAFT.
OURRstore is in Friendly User Mode, not in full production mode.
Expect any of the information below to change.

Description of OURRstore

The OU & Regional Research Store (OURRstore) is a very large scale storage instrument being deployed and maintained by the OU Supercomputing Center for Education & Research (OSCER) at the University of Oklahoma (OU).

OURRstore will be in production for 8+ years, and will have capacity for well over 100 Petabytes of tape (that is, well over 100,000 TB).

OURRstore will enable faculty, staff, postdocs, graduate students and undergraduates at the University of Oklahoma (OU), and at institutions across Oklahoma, as well as in other Great Plains Network (GPN) and Established Program to Stimulate Competitive Research (EPSCoR) states and territories, to build large and growing data collections, and to share these datasets with collaborators and others.

Via an innovative, low cost business model, researchers will buy their own tape cartridges, good for 8+ years, and pay zero usage charges (just tape cartridge and shipping costs).

OURRstore is expected to have hundreds of users.

OURRstore is intended for bulk data archiving, meaning small numbers of large files, not for zillions of tiny files. We can help you with that.

OURRstore is funded by a National Science Foundation Major Research Instrumentation grant (“Acquisition of a Regional Resource for Long-term Archiving of Large Scale Research Data Collections,” OAC-1828567, $967,755, 9/1/2018 - 8/31/2021, Principal Investigator Henry Neeman).

OURRstore is NOT a backup (here, "backup" means "automatically copy new or changed files nightly and all files occasionally," which ISN'T what the PetaStore is for).

What are the major components of OURRstore?

Tape Library:
~11000 tape cartridge slots
~5000 for new OU
~5000 for new non-OU
~1000 for data moved-in from PetaStore project
6 Tape drives (LTO-8 type, but compatible with IBM LTO-7M media and IBM LTO-8 media)
Note: Usable capacity does not equal raw capacity
Track-mounted Robot
Disk subsystem
~half a PetaByte, shared amongst all users
This space is used as a temporary "Landing Pad"
This landing pad space may (no guarantees) be extensible -- feel free to inquire.
Software
Scripts
Data Transfer Nodes provided by OU and OFFN

How do I get started?

Pre-requisite 1: Have an OSCER Account
Pre-requisite 2: Sign and Submit a PetaStore Acceptable Use Agreement
Eventually, an OURRstore Acceptable Use Agreement
Co-requisite 3: Purchase Media
Usable capacity does not equal raw capacity
Expect 15% overhead for formatting and fragmentation, and 100% overhead for each redundant tape.
Co-requisite 4: Non-OU users: Purchase Mailers

What are the system rules?

Rule 1: Minimum file size is 1 GB (1 Billion -- or 10^9 -- Bytes)

Applies to every individual file, not folders.
OSCER can help develop a workflow for compliance.

Rule 2: Maximum file size is 1 TB (1 Trillion -- or 10^12 -- Bytes)

Applies to every individual file, not folders.
Again, OSCER can help develop a workflow for compliance.

Rule 3: Contact OSCER about purchasing media

Currently, it's IBM LTO-7 Media (6TB raw), which we will format as LTO-7 type M (9TB raw)
LTO-8 (12TB raw) Media is acceptable
Again, usable capacity does not equal raw capacity.
(The decision between LTO-7M and LTO-8 is largely driven by customers' cost per TB)

Terminology:

Resident
A file is resident if its data is physically on the staging disk, and NOT on any tape.
Pre-migrated
A file is pre-migrated if its data is physically on both the staging disk AND on at least one tape.
Migrated
A file is migrated if its data is physically NOT on disk, but IS on at least one tape.
Stub-out
To stub-out a file is to replace a regular file on disk with a small stub of the file. The stub file may be accessed as if it is the regular file, and the OURRstore system will transparently recall the data from tape.
Staging Area or Landing Pad
The disk subsystem, used for temporary storage and for cached access.

Methods to access OURRstore

scp/sftp (and frontends, e.g. remote rsync / FileZilla / WinSCP)

Current: schooner dtn2.oscer.ou.edu
Future: sooner data mover nodes, offn dtns

Interactive linux console (ssh) (local commands, e.g. "cp" and local "rsync")

For OU-Users, (or anyone during friendly user mode)
Current: schooner dtn2.oscer.ou.edu
Future: sooner data mover nodes

Globus

Globus collection name: OURRstore or "U Oklahoma OURRstore"
Connectivity may be through OU or OFFN

Auto-archive (future)

from OSCER's OURdisk
from OSCER's Supercomputer /scratch

Information Lifecycle:

Upload/Ingest

I: Customer uploads file to Disk
(File ends in Resident state)
II: System Copy file from Disk to Tape
(File ends in Pre-migrated state)
III: System copies file from Disk to 2nd (or 3rd) tape:
(File will end in Pre-migrated state, regardless of which one of the following 3 methods is used):
- A. 2nd/3rd copy may (likely) be at the same time as the 1st
  III.A will happen if there is not much contention for tape drives
- B. 2nd/3rd copy may (unlikely) be later than the 1st
  III.B may happen if there is contention for tape drives
- C. 2nd/3rd copy may (very unlikely) be MUCH later than the 1st, after it has reached Migrated state
  III.C may happen if both there is tape drive contention and pressure to free up "landing pad" space
IV: Free up disk space, stub-out
(File ends in Migrated state)
Order of operations:
Usually, step III.A or III.B happen, so the order is I, II, III.(A or B), IV
Rarely, if Step III.C is to be taken, then the order is I, II, IV, III.C

Retrieval

No need to "trigger" the tape library to read or stage the data
Transparent recall
May be cached May experience delay
Recall caused a file's state to temporarily change from Migrated (on tape only) to Pre-Migrated (on disk and on tape)

Future: Publishing

What are the replication policies, and where are they?

On Globus, you can find your project at Collection "OURRstore" path "/projectname"
On OSCER Supercomputer, you will find it at "/archive/ourrstore/projectname"
note: After friendly user mode, this path will be "/archive/projectname"
Within your project, you will have at least one "replication policy" folder:
.../projectname/tape_2copies
(Available to everyone)
.../projectname/tape_3copies
(Automatically available for non-OU users)

How can I stay on top of how much OURRstore space I have, verify OURRstore is properly migrating?

Reporting:

(future) monthly report
Includes project's Space Available, project's Space Purchased
Includes project's space usage and file counts by each user of the project
Includes a compressed attachment containing detailed file report.
(md5 checksum, [other checksums ...]), Fst, (tape1 barcode [, tape2 barcode [, tape3 barcode]]), size_in_bytes, PATH/file.name
(future) OURRstore will send you summaries of ingested files, digested on a nightly/weekly cadence
Includes count of files, amount of data ingested in bytes, list of pending tasks.
Includes a compressed attachment containing detailed file ingest report.
(same format as monthly attachment)

Special commands on interactive linux console

"archive report"
(past. Replacement command TBD)

"dls"
(past. Replacement command TBD)

What are Some Best Practices for archiving?:

Verify, using standard tools from your discipline that your files are complete and not corrupt
Generate a checksum of each of your files
tar or zip your data
Generate a checksum of your tar/zipped file
Extract your data
Generate checksums of extracted data
Verify correct extraction of your data by comparing original checksums
Upload your data
Store your checksum information in a safe place outside of OURRstore.
At a minimum, e-mail it to yourself
Verify your checksums of your tar/zipped file with those contained in the e-mail report from OSCER