RCS-knowledge-base/python/20210705.run-python-jupyter.md

Question: Running Python Code on Jupyter Notebook

I have Python code that I would like to run on HPC. Can I run it through the Jupyter notebook? If so, how can I?

Yes, but we recommend that you launch the script through the HPC job scheduler, called “SLURM”. Let's call the script you want to run SCRIPT.py (include the full path if the script is not the same directory as where you want to run this). Here is a basic SLURM job script that may work for you:

#!/bin/bash
#SBATCH -p gpu
#SBATCH --gres gpu:1
enable_lmod
module load container_env tensorflow-gpu/2.4.0
crun.tensorflow-gpu python3 SCRIPT.py

Important notes:

• Don't worry why we recommend using the tensorflow container. This container has a installation of Python 3.7 that has fairly complete set of basic libraries used by Python users, such as numpy, scipy, pandas, scikit-learn, and of course, TensorFlow.
• Replace SCRIPT.py with your actual script file name.
• We assume that your workload needs a GPU to run your calculation. If you do not, please remove the #SBATCH -p and #SBATCH --gres lines and replace the words tensorflow-gpu with tensorflow-cpu is everywhere.

Create this script using a text editor, save it to a file (say, JOB.sh). Then you will submit the script from the UNIX shell interface of the cluster by typing:

$ sbatch JOB.sh

(The $ at the beginning of the line represents the shell prompt--do not type that.) If successful, there will be a message printed Submitted batch job NNNNNN on the terminal, where NNNNNN is an integer called job ID or job number. The output file will be slurm-NNNNNN.out.

Here are a few documentations to help understand this process:

• Documentation on Python on ODU HPC: https://wiki.hpc.odu.edu/Software/Python

• General workflow of using SLURM job scheduler: https://wiki.hpc.odu.edu/slurm#general-workflow-of-using-slurm

If you prefer video introduction to SLURM, please take a look at this short video to understand what job scheduler is:

"Working with SLURM"
https://odumedia.mediaspace.kaltura.com/playlist/dedicated/1_8eqsb16m/1_oy7ls1o0

(This is part of the Intro to HPC training videos.)

---

Back to your original question: Yes, you can run the script from a notebook. Once you opened a (blank) notebook, you will enter this statement in a new Python cell:

%load SCRIPT.py

Press <Shift+Enter> once, the code will be loaded into that same cell. Press <Shift+Enter> once more, the code will execute. But please note that this mode of execution can be awkward, i.e. we have to open a Jupyter session, %load the script in order to run it. The maximum time to run the script is under 24 hours (the max limit of Jupyter on HPC). This mode is useful if you plan to interactively test or troubleshoot a script, but less useful when you have a bunch of calculations to do, or calculation that you expect to run for a long time.

If you still have issue or question after reading this response: Can you share the script with us so we may understand the requirements of your script?