commit
efa580644c
16 changed files with 876 additions and 0 deletions
@ -0,0 +1,17 @@ |
||||
Using Torch with R |
||||
================== |
||||
|
||||
|
||||
Reference Literature |
||||
-------------------- |
||||
|
||||
### Book: "Deep Learning and Scientific Computing with R torch" |
||||
|
||||
- Author: Sigrid Keydana |
||||
- Publish-date: 2023-04-01 |
||||
- Publisher: CRC Press |
||||
- Publisher-version: https://doi.org/10.1201/9781003275923 |
||||
- License: Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License |
||||
- Github-source: https://github.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch |
||||
- Blog: https://blogs.rstudio.com/ai/posts/2023-04-05-deep-learning-scientific-computing-r-torch/ |
||||
- Free-web-book: https://skeydan.github.io/Deep-Learning-and-Scientific-Computing-with-R-torch/ |
||||
@ -0,0 +1,20 @@ |
||||
Overview of the Carpentries (Software Carpentry, Data Carpentry, ...) |
||||
===================================================================== |
||||
|
||||
|
||||
"Software Carpentry September 2012 Introduction" |
||||
https://www.youtube.com/watch?v=hIGweDdrZ20&ab_channel=softwarecarpentry |
||||
|
||||
* Greg Wilson's intro on the essentiality of computing skills for researchers. |
||||
* How effective learning works for novice programmers |
||||
(short- and long-term memory, chunking, ...). |
||||
|
||||
This seems to be part of a series of videos in this playlist: |
||||
|
||||
"Live Lecture September 2012" |
||||
https://www.youtube.com/watch?v=hIGweDdrZ20&list=PLhFTuW7KWApwoo2DHzpWLhA19zenWF1LF&ab_channel=softwarecarpentry |
||||
(28 video snippets) |
||||
|
||||
|
||||
|
||||
|
||||
@ -0,0 +1,9 @@ |
||||
Platforms and Solutions for AI/ML in the Cloud |
||||
============================================== |
||||
|
||||
|
||||
H2O.ai (India-based) |
||||
https://h2o.ai/ |
||||
|
||||
Solutions / example use cases: https://h2o.ai/solutions/use-case/ |
||||
|
||||
@ -0,0 +1,30 @@ |
||||
Basil: A Tool for Semi-Automatic Containerization, Deployment, and Execution of Scientific Applications on Cloud Computing and Supercomputing Platforms |
||||
------------------------------------------------------------------------------------------------------------------------------------------------------- |
||||
|
||||
Project site: https://icompute.us/entry |
||||
|
||||
|
||||
NSF Award: 2314203 |
||||
|
||||
"[CSSI?] Elements: Basil: A Tool for Semi-Automatic Containerization, |
||||
Deployment, and Execution of Scientific Applications on Cloud |
||||
Computing and Supercomputing Platforms" |
||||
https://www.nsf.gov/awardsearch/showAward?AWD_ID=2314203 |
||||
|
||||
From the abstract: |
||||
|
||||
> "...this project is developing BASIL - a tool for |
||||
> semi-automatically containerizing the scientific applications, |
||||
> frameworks, and workflows. This project will deliver BASIL through |
||||
> a web portal, as a command-line tool, and through APIs. BASIL has |
||||
> a broad applicability across multiple domains of deep societal |
||||
> impact such as artificial intelligence, drug discovery, and |
||||
> earthquake engineering. By enabling the preservation of valuable |
||||
> legacy software and making them usable for several years in |
||||
> future, BASIL will save cost and time in software rewriting and |
||||
> software installations, and thus contribute towards advancing the |
||||
> prosperity of the society. The project will result in educational |
||||
> content on ?Introduction to Containerization? and students engaged |
||||
> in the project will develop valuable skills in the areas of |
||||
> national interest such as supercomputing/High Performance |
||||
> Computing (HPC) and cloud computing." |
||||
@ -0,0 +1,32 @@ |
||||
Finetuning Large Language Models |
||||
================================ |
||||
|
||||
|
||||
"Finetuning Large Language Models" |
||||
https://www.deeplearning.ai/short-courses/finetuning-large-language-models/ |
||||
https://learn.deeplearning.ai/finetuning-large-language-models/lesson/1/introduction |
||||
Taught by Sharon Zhou |
||||
|
||||
Free course, registration required. |
||||
|
||||
From the course website: |
||||
|
||||
"Join our new short course, Finetuning Large Language Models! Learn |
||||
from Sharon Zhou, Co-Founder and CEO of Lamini, and instructor for the |
||||
GANs Specialization and How Diffusion Models Work. |
||||
|
||||
When you complete this course, you will be able to: |
||||
|
||||
* Understand when to apply finetuning on LLMs |
||||
* Prepare your data for finetuning |
||||
* Train and evaluate an LLM on your data |
||||
|
||||
With finetuning, you're able to take your own data to train the model |
||||
on it, and update the weights of the neural nets in the LLM, changing |
||||
the model compared to other methods like prompt engineering and |
||||
Retrieval Augmented Generation. Finetuning allows the model to learn |
||||
style, form, and can update the model with new knowledge to improve |
||||
results." |
||||
|
||||
|
||||
|
||||
@ -0,0 +1,13 @@ |
||||
PyTorch Installation |
||||
==================== |
||||
|
||||
|
||||
Selecting PyTorch target (CPU/GPU) for conda install |
||||
---------------------------------------------------- |
||||
|
||||
See: https://pytorch.org/get-started/locally/https://pytorch.org/get-started/locally/ |
||||
|
||||
There are a selector box in the page above to select which commands to |
||||
use to install Torch. |
||||
|
||||
|
||||
@ -0,0 +1,37 @@ |
||||
Tools and Utilities for BibTeX Citations |
||||
======================================== |
||||
|
||||
[2023-02-17] |
||||
|
||||
|
||||
Utilities for Generating BibTeX Citations |
||||
----------------------------------------- |
||||
|
||||
When a lot of citations are involved, this becomes a troublesome task. |
||||
Having some tools would make our lives easier. |
||||
|
||||
|
||||
### Converting from text-plain citations to BibTeX entries |
||||
|
||||
|
||||
https://text2bib.economics.utoronto.ca/ |
||||
|
||||
From the site: |
||||
|
||||
> This site converts a list of references in a wide range of styles to BibTeX. |
||||
> Minimal requirements for input file: |
||||
> |
||||
> * Either references are separated by blank lines or each line |
||||
> is a separate reference or each reference starts with |
||||
> `\bibitem{}`, `\bibitem{<label>}`, `\item`, `\bigskip`, or `\smallskip`. |
||||
> * Each reference either starts (after possibly one of the separator strings |
||||
> mentioned in the previous point) with a list of authors, |
||||
> which is followed by either a year or a title, or starts with a year; |
||||
> if the authors are followed by a year, the next string is the title. |
||||
> |
||||
> The conversion is not perfect, but is very good for many reference styles |
||||
|
||||
|
||||
### Harvesting DOI Metadata as BibTeX Entries |
||||
|
||||
https://www.doi2bib.org/ |
||||
@ -0,0 +1,16 @@ |
||||
*Automatic) Grading Tools |
||||
========================= |
||||
|
||||
Submitty |
||||
-------- |
||||
https://github.com/Submitty/Submitty |
||||
|
||||
Summary: "Homework Submission, Automated Grading, and TA grading system" |
||||
|
||||
Some capabilities: |
||||
|
||||
* Common multiple-choice answers |
||||
* Simple-response answers (text, numeric) |
||||
* PDF submission + annotating capability |
||||
* Notebook submission (support codes) |
||||
|
||||
@ -0,0 +1,17 @@ |
||||
Notes on NVIDIA-SMI |
||||
=================== |
||||
|
||||
|
||||
NVIDIA-SMI for NVLink query |
||||
--------------------------- |
||||
|
||||
"Exploring NVIDIA NVLink `nvidia-smi` Commands" |
||||
https://www.exxactcorp.com/blog/HPC/exploring-nvidia-nvlink-nvidia-smi-commands |
||||
|
||||
Example commands to consider: |
||||
|
||||
``` |
||||
$ nvidia-smi nvlink --help |
||||
$ nvidia-smi nvlink --status |
||||
``` |
||||
|
||||
@ -0,0 +1,14 @@ |
||||
NSF-funded Dashboards & Portals |
||||
=============================== |
||||
|
||||
"NSF-powered dashboards you should bookmark" |
||||
-------------------------------------------- |
||||
|
||||
https://beta.nsf.gov/science-matters/nsf-powered-dashboards-you-should-bookmark |
||||
2022-11-03 - Jason Bates |
||||
|
||||
|
||||
|
||||
## Impact of insects and diseases on forest health |
||||
|
||||
Hosted by Purdue University, the Alien Forest Pest Explorer combines information from multiple sources to show the impact of different forest insects and diseases -- and the potential for further damage. The dashboards, also supported by the U.S. Forest Service, overlay pest data with related data about the status and health of the host tree species in forests and has resolution down to the county level. |
||||
@ -0,0 +1,80 @@ |
||||
Python Tutorials & Short Courses (vol. 1) |
||||
========================================= |
||||
|
||||
Mosh's Python for Beginners - Learn Python in 1 Hour |
||||
---------------------------------------------------- |
||||
|
||||
https://www.youtube.com/watch?v=kqtD5dpn9C8&ab_channel=ProgrammingwithMosh |
||||
|
||||
Length: 1 hr |
||||
|
||||
Key contents: |
||||
* Download & install Python |
||||
* Download & install PyCharm as Python IDE |
||||
* Three basic data types: strings, numbers, boolean (True/False) |
||||
* Variables |
||||
* Simple input (from terminal) |
||||
* String concatenation (`+` operator) |
||||
* Type conversion |
||||
* Simple calculator program |
||||
* String methods |
||||
|
||||
|
||||
|
||||
"Python Tutorial - Python Full Course for Beginners" |
||||
https://www.youtube.com/watch?v=_uQrJ0TkZlc&ab_channel=ProgrammingwithMosh |
||||
|
||||
TABLE OF CONTENT |
||||
``` |
||||
00:00:00 Introduction |
||||
00:01:49 Installing Python 3 |
||||
00:06:10 Your First Python Program |
||||
00:08:11 How Python Code Gets Executed |
||||
00:11:24 How Long It Takes To Learn Python |
||||
00:13:03 Variables |
||||
00:18:21 Receiving Input |
||||
00:22:16 Python Cheat Sheet |
||||
00:22:46 Type Conversion |
||||
00:29:31 Strings |
||||
00:37:36 Formatted Strings |
||||
00:40:50 String Methods |
||||
00:48:33 Arithmetic Operations |
||||
00:51:33 Operator Precedence |
||||
00:55:04 Math Functions |
||||
00:58:17 If Statements |
||||
01:06:32 Logical Operators |
||||
01:11:25 Comparison Operators |
||||
01:16:17 Weight Converter Program |
||||
01:20:43 While Loops |
||||
01:24:07 Building a Guessing Game |
||||
01:30:51 Building the Car Game |
||||
01:41:48 For Loops |
||||
01:47:46 Nested Loops |
||||
01:55:50 Lists |
||||
02:01:45 2D Lists |
||||
02:05:11 My Complete Python Course |
||||
02:06:00 List Methods |
||||
02:13:25 Tuples |
||||
02:15:34 Unpacking |
||||
02:18:21 Dictionaries |
||||
02:26:21 Emoji Converter |
||||
02:30:31 Functions |
||||
02:35:21 Parameters |
||||
02:39:24 Keyword Arguments |
||||
02:44:45 Return Statement |
||||
02:48:55 Creating a Reusable Function |
||||
02:53:42 Exceptions |
||||
02:59:14 Comments |
||||
03:01:46 Classes |
||||
03:07:46 Constructors |
||||
03:14:41 Inheritance |
||||
03:19:33 Modules |
||||
03:30:12 Packages |
||||
03:36:22 Generating Random Values |
||||
03:44:37 Working with Directories |
||||
03:50:47 Pypi and Pip |
||||
03:55:34 Project 1: Automation with Python |
||||
04:10:22 Project 2: Machine Learning with Python |
||||
04:58:37 Project 3: Building a Website with Django |
||||
``` |
||||
|
||||
@ -0,0 +1,25 @@ |
||||
Python Containers on ODU HPC |
||||
============================ |
||||
|
||||
This article contains the list of Python containers available on ODU HPC platform for all HPC users. |
||||
|
||||
> The background for the containerized Python can be found on the main documentation page for [Python on HPC Environment](/Software/Python). If |
||||
|
||||
|
||||
Python containers with TensorFlow and PyTorch |
||||
--------------------------------------------- |
||||
|
||||
Since TensorFlow and PyTorch have become wildly popular with the adoption of AI/ML into many scientific disciplines, RCS provides a handful of TensorFlow and PyTorch containers for general uses. |
||||
|
||||
|
||||
|
||||
Looking for containers with newer Python? |
||||
----------------------------------------- |
||||
|
||||
We have a few modules that contain newer versions of Python: |
||||
|
||||
| Module name | Current Python version | |
||||
|---------------------------|-------------------------| |
||||
| `python3/2023.2-py39` | Intel Python 3.9.16 | |
||||
| `python3/2023.2-py310` | Intel Python 3.10.11 | |
||||
|
||||
@ -0,0 +1,135 @@ |
||||
CFL3D Installation Notes - ODU Wahab Cluster |
||||
============================================ |
||||
|
||||
About the CFL3D Software |
||||
------------------------ |
||||
|
||||
* Software home page: https://software.nasa.gov/software/LAR-16003-1 |
||||
* Git repo: https://github.com/nasa/CFL3D |
||||
* Documentation: https://nasa.github.io/CFL3D/ |
||||
|
||||
Installation on ODU Cluster |
||||
--------------------------- |
||||
|
||||
* Base container: intel/2023.0 (ICC + Intel MPI) |
||||
|
||||
* Following build instruction: https://nasa.github.io/CFL3D/Cfl3dv6/cfl3dv6_build.html#make |
||||
|
||||
* Configuration: (This is called "Installation" stage in their lingo.) |
||||
From the `build` subfolder, issue: `./Install -noredirect -linux_compiler_flags=Intel` |
||||
|
||||
* Build: |
||||
|
||||
- `make cfl3d_seq` |
||||
- `make cfl3d_mpi` |
||||
- ... and so on. see the help doc issued by `make` with no target. |
||||
|
||||
|
||||
Usage Instruction |
||||
----------------- |
||||
|
||||
(Initially written for Dr. Adem Ibrahim, 2023-05-31) |
||||
|
||||
Dr. Ibrahim, |
||||
|
||||
Below is an instruction to run CFL3D on our cluster: |
||||
|
||||
The software is currently installed in your home directory at the following path: |
||||
|
||||
~/CFL3D/bin |
||||
|
||||
**Prerequisites for running CFL3D** |
||||
|
||||
This software was built on top of the "intel/2023.0" container, |
||||
so the first thing you must do is to invoke the following commands on the shell: |
||||
|
||||
```bash |
||||
module load container_env intel/2023.0 |
||||
``` |
||||
|
||||
For serial runs, the main input file MUST be named to `cfl3d.inp`. |
||||
Assuming that this input file has existed in the current directory, |
||||
you will run the *serial* CFL3D software in this way: |
||||
|
||||
```bash |
||||
crun.intel ~/CFL3D/bin/cfl3d_seq |
||||
``` |
||||
|
||||
There is an MPI (parallel) version of CFL3D, called `cfl3d_mpi` |
||||
that has been installed into the the same folder. |
||||
|
||||
|
||||
This is an example of SLURM job script to run CFL3D in serial (sequential) mode: |
||||
|
||||
```bash |
||||
#!/bin/bash |
||||
#SBATCH --job-name cfl3d |
||||
#SBATCH --ntasks 1 |
||||
|
||||
module load container_env intel/2023.0 |
||||
crun.intel ~/CFL3D/bin/cfl3d_seq |
||||
``` |
||||
|
||||
CFL3D has a lot of sample calculations located here: |
||||
https://nasa.github.io/CFL3D/Cfl3dv6/cfl3dv6_testcases.html |
||||
|
||||
### Demo: Flat Plate Steady Flow |
||||
|
||||
Source: https://nasa.github.io/CFL3D/Cfl3dv6/cfl3dv6_testcases.html#flatplate |
||||
|
||||
Here are the commands I invoked: |
||||
|
||||
```bash |
||||
module load container_env intel/2023.0 |
||||
|
||||
mkdir -p ~/LIONS/Cfl3dv6/examples |
||||
cd ~/LIONS/Cfl3dv6/examples |
||||
|
||||
# download and unpack the input files |
||||
wget https://nasa.github.io/CFL3D/Cfl3dv6/2DTestcases/Flatplate/Flatplate.tar.Z |
||||
tar xvf Flatplate.tar.Z |
||||
cd Flatplate/ |
||||
|
||||
# split the input files and generate the unformatted grid file, |
||||
# which is grdflat5.bin |
||||
crun.intel ~/CFL3D/bin/splitter < split.inp_1blk |
||||
|
||||
# copy the main input file as "cfl3d.inp" before running: |
||||
cp grdflat5.inp cfl3d.inp |
||||
srun crun.intel ~/CLF3D/bin/cfl3d_seq |
||||
``` |
||||
|
||||
Files will be unpacked to a subfolder called `Flatplate`, |
||||
and this folder is also where the calculation is taking place. |
||||
The main output file will go to a file named `cfl3d.out`. |
||||
|
||||
> FIXME: Run in parallel. Still has issue on Wahab. |
||||
|
||||
### Update 2023-06-02 |
||||
|
||||
A few notes: |
||||
|
||||
1. The CFL3D software can only run on Wahab as the hardware was new enough |
||||
for the instruction sets used in the code. |
||||
Please do not run this on Turing as it will quit with an error message. |
||||
|
||||
2. With version 6, there is no more need to recompile CFL3D |
||||
every time you want to run with a different physical system (model). |
||||
The code now allocates arrays dynamically, so `precfl3d` is not needed anymore. |
||||
|
||||
3. I included the source code in <<#TODO>> directory in case you want to play |
||||
around and modify the source code. |
||||
|
||||
4. The code was built to NOT read from stdin. Please do not run in this way: |
||||
|
||||
``` |
||||
crun.intel ~/CFL3D/bin/cfl3d_seq < MY_INPUT.inp ### WON'T WORK |
||||
``` |
||||
|
||||
Instead, run it in two steps: |
||||
|
||||
``` |
||||
cp MY_INPUT.inp cfl3d.inp |
||||
crun.intel ~/CFL3D/bin/cfl3d_seq |
||||
``` |
||||
|
||||
@ -0,0 +1,159 @@ |
||||
# Snakemake: A General Introduction |
||||
|
||||
|
||||
Main-website: |
||||
https://snakemake.github.io/ |
||||
|
||||
Snakemake Documentation |
||||
https://snakemake.readthedocs.io/en/stable/ |
||||
|
||||
Tutorials: |
||||
|
||||
* Official tutorial (complete steps) |
||||
https://snakemake.readthedocs.io/en/stable/tutorial/tutorial.html#tutorial |
||||
|
||||
- Basic tutorial: |
||||
https://snakemake.readthedocs.io/en/stable/tutorial/basics.html#tutorial-basics |
||||
|
||||
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~` |
||||
|
||||
|
||||
|
||||
Standalone |
||||
|
||||
|
||||
This is the easiest way to run it, you will be allocated to a single node, and all resource is allocated to you, there is no cross-node parallelization in this way, you can use job script below: |
||||
|
||||
|
||||
|
||||
#!/bin/bash |
||||
|
||||
|
||||
|
||||
#SBATCH --exclusive |
||||
|
||||
|
||||
|
||||
enable_lmod |
||||
|
||||
module load container_env snakemake |
||||
|
||||
|
||||
|
||||
crun snakemake |
||||
|
||||
|
||||
|
||||
Break rules to their individual Slurm Jobs |
||||
|
||||
|
||||
This is my recommended way of running this package, it utilizes resource most effectively, however it is a little bit more complicated, so I have written some wrapper script to do most of legging work, you can just launch it like this: |
||||
|
||||
|
||||
|
||||
#!/bin/bash |
||||
|
||||
|
||||
|
||||
#SBATCH -c 2 |
||||
|
||||
|
||||
|
||||
enable_lmod |
||||
|
||||
module load container_env snakemake |
||||
|
||||
|
||||
|
||||
snakemake.helper -j 10 |
||||
|
||||
|
||||
|
||||
Please note that this script should only take 1 or 2 core to run, it is a master script, it does not do the real work, additional jobs will be launched by this job. |
||||
|
||||
Instead of run "crun snakemake" directly, you run "snakemake.helper" instead, it's the helper script I mentioned, it will launch job within the cluster for you. |
||||
|
||||
|
||||
|
||||
For cluster resource, I only enforce "threads" in snake rules, any other resource (mem,disk,….) might or might not be enforced by snakemake (I am not sure), at least I will not enforce it on the scheduler level. |
||||
|
||||
|
||||
|
||||
rule map_reads: |
||||
|
||||
input: |
||||
|
||||
"data/genome.fa", |
||||
|
||||
"data/samples/{sample}.fastq" |
||||
|
||||
output: |
||||
|
||||
"results/mapped/{sample}.bam" |
||||
|
||||
threads: 2 |
||||
|
||||
shell: |
||||
|
||||
"bwa mem {input} | samtools view -b - > {output}" |
||||
|
||||
|
||||
|
||||
When setting threads in a rule, I will launch Slurm job with same "--cpus-per-task" configuration to match it. When not explicitly set, it will always be single thread. |
||||
|
||||
|
||||
|
||||
My snakemake module also support MPI mode, but from what I observe your tasks usually involving running some code on a lot of input, instead of running a single multiple node code on a single input, so mode 2 should be most useful to you. If you do need to run MPI somehow, please let me know it will require some additional setup, especially when combined with "--use-conda" . |
||||
|
||||
|
||||
|
||||
Snakemake conda is supported and tested, you should be able to install any conda package you want to. Unless it requires MPI, it usually works. When using conda, please make sure environment file is given and launch snakemake with "—use-conda": |
||||
|
||||
|
||||
|
||||
# Snakefile |
||||
|
||||
|
||||
|
||||
rule map_reads: |
||||
|
||||
input: |
||||
|
||||
"data/genome.fa", |
||||
|
||||
"data/samples/{sample}.fastq" |
||||
|
||||
output: |
||||
|
||||
"results/mapped/{sample}.bam" |
||||
|
||||
threads: |
||||
|
||||
2 |
||||
|
||||
conda: |
||||
|
||||
"envs/mapping.yaml" |
||||
|
||||
shell: |
||||
|
||||
"bwa mem {input} | samtools view -b - > {output}" |
||||
|
||||
|
||||
|
||||
# job_script.sh |
||||
|
||||
|
||||
|
||||
crun snakemake --use-conda # for standalone |
||||
|
||||
snakemake.helper -j 10 --use-conda # for running in scheduler mode |
||||
|
||||
|
||||
|
||||
Running other container modules is also possible, please just let me know what you need. If you can install what you need with conda, please use conda first since you can do it yourself. |
||||
|
||||
|
||||
|
||||
You can find my sample job script in /home/jsun/snakemake , please let me know if you have any questions or issues. |
||||
@ -0,0 +1,248 @@ |
||||
# Snakemake: A General Introduction |
||||
|
||||
Snakemake is a workflow tool for managing and executing |
||||
interrelated sets of computation/analysis steps. |
||||
|
||||
For more information, please visit |
||||
[Snakemake documentation](https://snakemake.readthedocs.io/en/stable/) |
||||
|
||||
|
||||
## Snakemake in a Nutshell |
||||
|
||||
Imagine that you have a complex computation that consists of a series |
||||
of tasks that must be executed in order |
||||
(each taking the output from the previous step as its input): |
||||
|
||||
input preprocessing => simulation => output postprocessing |
||||
|
||||
In the example above, a complete computation consists of: |
||||
|
||||
* step 1: input preprocessing of some sort, such as generating |
||||
a set of input files from a few parameters; |
||||
* step 2: compute-heavy simulation, potentially taking hours or days |
||||
on many CPU cores and/or GPUs; |
||||
* step 3: output postprocessing, such as calculating some statistics |
||||
from the simulation, determining molecular properties from |
||||
the simulation, creating a report with graph panels. |
||||
|
||||
If there is only one computation to do, then it's no brainer to do the |
||||
three steps "by hand", i.e. creating up to three SLURM job scripts and |
||||
running them in sequence. |
||||
What if there are 1000 of such computations to perform? |
||||
What if the set of steps are complex (e.g. one step requires a few |
||||
inputs from different prior computations), or long? |
||||
Is there a better way than sitting in front of terminal submitting |
||||
1000+ jobs *in a particular order*? |
||||
|
||||
The answer is, **yes!** |
||||
This is where we need *workflow tools* such as Snakemake. |
||||
In Snakemake, a complete workflow is stored in a specially formatted text |
||||
file named `Snakefile`. |
||||
Each processing step is expressed as a *rule* inside the snakefile. |
||||
Each rule may depend on one or more other rules; the input and output |
||||
file declaration determines this dependency rule. |
||||
|
||||
|
||||
> A more thorough introduction to Snakemake is beyond the scope of this article. |
||||
Readers are referred to some articles and tutorials linked at the end |
||||
of this article to learn more. |
||||
We assume that you have a basic idea of workflow tools in order to use Snakemake. |
||||
{.is-info} |
||||
|
||||
|
||||
## Snakemake on ODU HPC |
||||
|
||||
On Wahab and Turing, there are two modes to use Snakemake: |
||||
|
||||
Snakemake is installed, there are two modes to run this software: |
||||
|
||||
- Cluster (job-based) execution |
||||
- Single-node (standalone) execution |
||||
|
||||
We will describe cluster execution first, which is more scalable and powerful. |
||||
|
||||
|
||||
### Mode 1: Cluster Execution (recommended) |
||||
|
||||
In the [Cluster (job-based) execution](https://snakemake.readthedocs.io/en/stable/executing/cluster.html), |
||||
Snakefile will turn each rule into an individual Slurm job and execute |
||||
them via the job scheduler. |
||||
This will allow the HPC resources to be used most efficiently, |
||||
therefore we recommend using this way to run your workflow, whenever possible. |
||||
We created a helper script called `snakemake.helper` to help executing |
||||
a snakefile workflow through a job scheduler. |
||||
Here is an example invocation (written as a Slurm job script): |
||||
|
||||
``` |
||||
#!/bin/bash |
||||
#SBATCH --cpus-per-task 2 |
||||
#SBATCH --job-name Example_snakemake_cluster |
||||
|
||||
enable_lmod |
||||
module load container_env snakemake |
||||
|
||||
snakemake.helper -j 10 |
||||
``` |
||||
|
||||
This script should only take 1 or 2 core to run, since it is *only* a master script. |
||||
It does not do the real work (e.g. it does not run the rule in this script). |
||||
When processing the input snakefile, Snakemake will spawn additional Slurm jobs |
||||
(where one snakefile rule == one Slurm job) |
||||
and monitor their completion in order to push the workflow forward to its completion. |
||||
|
||||
Pros: |
||||
|
||||
- Very scalable, this method can use as many CPU cores and compute nodes as |
||||
specified in the snakefile (or by the cluster policy); |
||||
- Complete control over resource specification per rule; |
||||
|
||||
Cons: |
||||
|
||||
- May not be efficient for small/simple rules. |
||||
|
||||
|
||||
> Important: In the cluster execution model, do not run `crun snakemake` directly! |
||||
You run `snakemake.helper` helper script instead. |
||||
{.is-info} |
||||
|
||||
> For cluster resources, we only enforce `threads` in the snakefile rules. |
||||
Any other resource (memory, disk, ..) might or might not be enforced by |
||||
Snakemake. |
||||
{.is-info} |
||||
|
||||
|
||||
## Mode 2: Single-node (stand-alone) execution |
||||
|
||||
In the single-node execution mode, you will acquire all the CPU |
||||
resources in a single node and can use all the cores to perform as |
||||
many tasks as can be parallelized to shorten the execution time. |
||||
(Snakemake will figure out what rules can be executed in parallel and |
||||
try to execute them concurrently, subject to the CPU core |
||||
constraints). |
||||
|
||||
Use the following template to start your own job running Snakemake in |
||||
the single-node mode: |
||||
|
||||
``` |
||||
#!/bin/bash |
||||
#SBATCH --exclusive |
||||
#SBATCH --job-name Example_snakemake_1node |
||||
|
||||
enable_lmod |
||||
module load container_env snakemake |
||||
|
||||
crun snakemake |
||||
``` |
||||
|
||||
Pros: |
||||
|
||||
- The easiest way to run Snakemake (no need to think about threads, etc.); |
||||
- You will have complete access to all compute resources on a single node |
||||
(CPU, memory, ...). |
||||
|
||||
Cons: |
||||
|
||||
- No parallelization across multiple nodes, |
||||
thus limiting the parallel scalability; |
||||
- Rules will be invoked within the same container as the Snakemake program; |
||||
if your program requires software in other containers, this will not work. |
||||
(Currently a containerized program [those with `crun` in its invocation] |
||||
cannot execute another program located inside a different container). |
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
(I am not sure), at least I will not enforce it on the scheduler level. |
||||
|
||||
|
||||
rule map_reads: |
||||
|
||||
input: |
||||
|
||||
"data/genome.fa", |
||||
|
||||
"data/samples/{sample}.fastq" |
||||
|
||||
output: |
||||
|
||||
"results/mapped/{sample}.bam" |
||||
|
||||
threads: 2 |
||||
|
||||
shell: |
||||
|
||||
"bwa mem {input} | samtools view -b - > {output}" |
||||
|
||||
|
||||
|
||||
When setting threads in a rule, I will launch Slurm job with same "--cpus-per-task" configuration to match it. When not explicitly set, it will always be single thread. |
||||
|
||||
|
||||
|
||||
My snakemake module also support MPI mode, but from what I observe your tasks usually involving running some code on a lot of input, instead of running a single multiple node code on a single input, so mode 2 should be most useful to you. If you do need to run MPI somehow, please let me know it will require some additional setup, especially when combined with "--use-conda" . |
||||
|
||||
|
||||
|
||||
|
||||
- |
||||
|
||||
is installed, there are two modes to run this software: |
||||
|
||||
|
||||
|
||||
Standalone |
||||
|
||||
|
||||
|
||||
|
||||
Snakemake conda is supported and tested, you should be able to install any conda package you want to. Unless it requires MPI, it usually works. When using conda, please make sure environment file is given and launch snakemake with "—use-conda": |
||||
|
||||
|
||||
|
||||
# Snakefile |
||||
|
||||
|
||||
|
||||
rule map_reads: |
||||
|
||||
input: |
||||
|
||||
"data/genome.fa", |
||||
|
||||
"data/samples/{sample}.fastq" |
||||
|
||||
output: |
||||
|
||||
"results/mapped/{sample}.bam" |
||||
|
||||
threads: |
||||
|
||||
2 |
||||
|
||||
conda: |
||||
|
||||
"envs/mapping.yaml" |
||||
|
||||
shell: |
||||
|
||||
"bwa mem {input} | samtools view -b - > {output}" |
||||
|
||||
|
||||
|
||||
# job_script.sh |
||||
|
||||
|
||||
|
||||
crun snakemake --use-conda # for standalone |
||||
|
||||
snakemake.helper -j 10 --use-conda # for running in scheduler mode |
||||
|
||||
|
||||
|
||||
Running other container modules is also possible, please just let me know what you need. If you can install what you need with conda, please use conda first since you can do it yourself. |
||||
|
||||
|
||||
|
||||
You can find my sample job script in /home/jsun/snakemake , please let me know if you have any questions or issues. |
||||
@ -0,0 +1,24 @@ |
||||
Survey of Workflow Tools |
||||
======================== |
||||
|
||||
DAGMan - Built-in workflow tool for HTCondor |
||||
-------------------------------------------- |
||||
Built for high-throughput computing. |
||||
|
||||
|
||||
Nextflow |
||||
-------- |
||||
|
||||
Supposedly very popular! |
||||
|
||||
|
||||
Snakemake |
||||
--------- |
||||
|
||||
Supposedly easy to understand and learn for the first time. |
||||
|
||||
|
||||
Common Workflow Language (CWL) |
||||
------------------------------ |
||||
|
||||
Aiming to be the common standard of expressing workflows. |
||||
Loading…
Reference in new issue