# Question: I have Python code that I would like to run on HPC. Can I run it through the Jupyter notebook? If so, how can I?
# Question: Running Python Code on Jupyter Notebook
I have Python code that I would like to run on HPC.
Can I run it through the Jupyter notebook?
If so, how can I?
<!-- was taken from Wirawan's WORK SCRAP NOTES -->
<!-- was taken from Wirawan's WORK SCRAP NOTES -->
Yes, but we recommend that you launch the script through the HPC job scheduler, called “SLURM”. Let’s call the script you want to run “SCRI
Yes, but we recommend that you launch the script through the HPC job scheduler,
PT.py” (include the full path if the script is not the same directory as where you want to run this). Here is a basic SLURM job script that may work for you:
called “SLURM”.
Let's call the script you want to run `SCRIPT.py`
*(include the full path if the script is not the same directory
as where you want to run this)*.
Here is a basic SLURM job script that may work for you:
```
#!/bin/bash
#!/bin/bash
#SBATCH -p gpu
#SBATCH -p gpu
#SBATCH --gres gpu:1
#SBATCH --gres gpu:1
enable_lmod
enable_lmod
module load container_env tensorflow-gpu/2.4.0
module load container_env tensorflow-gpu/2.4.0
crun.tensorflow-gpu python3 SCRIPT.py
crun.tensorflow-gpu python3 SCRIPT.py
Important notes:
```
Replace SCRIPT.py with your actual script file name.
We assume that your workload needs a GPU to run your calculation. If you do not, please remove the #SBATCH lines and replace the words “tensorflow-gpu” with “tensorflow-cpu” is everywhere.
**Important notes:**
Create this script using a text editor, save it to a file (say, name this JOB.sh). Then you will submit the script from the UNIX shell interface of the cluster by typing:
* Don't worry why we recommend using the `tensorflow`
This container has a installation of Python 3.7 that has fairly
complete set of basic libraries used by Python users, such as
`numpy`, `scipy`, `pandas`, `scikit-learn`, and of course,
TensorFlow.
* Replace `SCRIPT.py` with your actual script file name.
* We assume that your workload needs a GPU to run your calculation.
If you do not, please remove the `#SBATCH -p` and `#SBATCH --gres` lines
and replace the words `tensorflow-gpu` with `tensorflow-cpu` is everywhere.
Create this script using a text editor, save it to a file (say, `JOB.sh`).
Then you will submit the script
from the UNIX shell interface of the cluster by typing:
```
$ sbatch JOB.sh
$ sbatch JOB.sh
(The “$” at the beginning of the line represents the shell prompt--do not type that.) If successful, there will be a message printed “Submitted batch job NNNNNN”, where NNNNNN is an integer called job ID or job number. The output file will be “slurm-NNNNNN.out”.
```
(The `$` at the beginning of the line represents
the shell prompt--do not type that.)
If successful, there will be a message printed
`Submitted batch job NNNNNN`
on the terminal,
where `NNNNNN` is an integer called job ID or job number.
The output file will be `slurm-NNNNNN.out`.
Here are a few documentations to help understand this process:
Here are a few documentations to help understand this process:
Documentation on Python on ODU HPC: https://wiki.hpc.odu.edu/Software/Python
General workflow of using SLURM job scheduler: https://wiki.hpc.odu.edu/slurm#general-workflow-of-using-slurm
[Intro to HPC](https://wiki.hpc.odu.edu/Training/HPC-Intro)
training videos.)
-----
-----
Now, back to your original question: Yes, you can run the script from a notebook. Once you opened a (blank) notebook, you will make this statement in a new cell:
*Back to your original question*:
Yes, you can run the script from a notebook.
Once you opened a (blank) notebook, you will enter this statement in a new Python cell:
```
%load SCRIPT.py
%load SCRIPT.py
Press <Shift+Enter> once, the code will be loaded into that same cell. Press <Shift+Enter> once more, the code will execute. But please note that this mode of execution can be awkward, i.e. we have to open a Jupyter session, %load the script in order to run it, and even after that, the maximum time to run the script is under 24 hours. This mode is useful if you plan to interactively test or troubleshoot a script, but less useful when you have a bunch of calculations to do, or calculation that you expect to run for a long time.
```
As a closing, I’d like to ask this: Can you share the script with us so I may understand the requirements of your script?
Press `<Shift+Enter>` once, the code will be loaded into that same cell.
Press `<Shift+Enter>` once more, the code will execute.
But please note that this mode of execution can be awkward, i.e.
we have to open a Jupyter session,
`%load` the script in order to run it.
The maximum time to run the script is under 24 hours (the max limit of Jupyter on HPC).
This mode is useful if you plan to interactively test or troubleshoot a script,
but less useful when you have a bunch of calculations to do,
or calculation that you expect to run for a long time.
*If you still have issue or question after reading this response:*
Can you share the script with us so we may understand the requirements of your script?
Wirawan Purwanto
3 years ago