This structure is relevant only for moriah cluster.
It's important to understand the folder structure in the cluster so you will know where to save your files.
¶ Backup vs Snapshot
First you need to understand the difference between a backup and snapshots.
Here are short simplified explanations:
¶ Backup
A copy of the data in an external system with version history.
Need IT support to recover.
¶ Snapshots
A copy of the data in the current storage.
Good for fast recovery of accidently deleted or wrongly edited files.
¶ Snapshots Cycles in Cluster
- One snapshot every 4 hours (0, 4, 8, 12, 16, 20) -- cycles out every 24 hours.
- One snapshot at midnight -- cycles out after two weeks.
- One snapshot at 1 a.m. on the 1st of every month -- cycles out every 3 months.
See this wiki page on recovering files from snapshots.
¶ Storage Tiers and Symlinks
Each lab member will have their own subfolder under their PI folder in the labs shared storage.
For your convenience, symbolic links to personal folders under each storage tier are automatically created there:
icore-home and icore-data folders
| Volume | Details | Path |
| Fast, snapshots, no backup | Main working storage for computation work | /sci/labs/<PI login> |
| Fast, no snapshots or backup | Fast storage for temporary files (without any backup options) | /sci/nosnap/<PI login> |
| Fast with full backup | Fast storage for a full backup service | /sci/backup/<PI login> |
| Archive | Long term storage. Cannot be used for computation | /sci/archive/<PI login> |
| Archive with backup | Archive storage + full backup | /sci/backup/Archive/<PI login> |
¶ Home Folder
Each user has a home folder with a free quota of 5GB. This folder has backup and snapshots.
The path of the folder is:
/sci/home/<your_username>Even if you don't use this folder directly, it tends to get filled with temporary files and various software packages for R, Python, Matlab, etc.
If you are not working within a virtual environment (using conda, python venvs or any other way), then installing python packages using “pip install” or the R built-in package manager will put those packages somewhere in your home folder.
To check the status of your home folder, we recommend using the ncdu tool, available by loading the ncdu module:
your_shell_prompt> module load ncdu
After which calling the ncdu command (from your home-folder! type cd ~ to get there if you arent) will open an interactive directory tree, sorted by the size of your home folder's sub-directories:
Make sure to run the command from your home folder to get the relevant information!
Using the arrow keys (right to dive into a folder, left to jump back out), you can get inside the ncbi folder and see that I don't care about the data that's in it, using the d key, I tell the program that I wan't to delete it.
¶ Group Folder
Each PI/Lab has a group folder.
This is where you save files that you use for computation work in the cluster.
Each member of a group has a folder in the group folder.
All data saved under the member folder HAS snapshots but NO backup.
Quota: Default is 1TB per lab folder. PIs can requested to increase/decrease their quota by filling this Google form.
Location:
/sci/labs/<pi_username>/<member_username>Inside each group member folder there are 4 symlinks (symbolic links) to special folders: backup, nosnap , archive and archive_backup.
See details on each folder below.
¶ Backup
All data saved in this folder IS backed up to an external backup system.
To recover data from this backup you must contact the support team.
¶ Nosnap
All data saved in this folder has NO backup or snapshots.
¶ Archive
Data saved in this folder is for long term storage and cannot be used for computational work in the cluster.
If you wish to work on data that is stored here then you should first move it to your lab folder.
¶ Archive Backup
Data saved in this folder is for long term storage and is backed up to an external system.
¶ Data Folder
These folders contain data that is used by several groups of researchers and students.
¶ NextSeq
This folder contains the NextSeq systems sequencing output.
Folder path:
/sci/data/nextseq