RCS-knowledge-base/python/20210705.run-python-jupyter.md

# Question: Running Python Code on Jupyter Notebook

<!--
    Published URL: https://forum.hpc.odu.edu/t/running-a-python-code-on-hpc-through-jupyter/85
    Note: The version on this doc is slightly different from what published
    there as of 2022-05-19.
-->

I have Python code that I would like to run on HPC.
Can I run it through the Jupyter notebook?
If so, how can I?

<!-- was taken from Wirawan's WORK SCRAP NOTES -->

Yes, but we recommend that you launch the script through the HPC job scheduler,
called “SLURM”.
Let's call the script you want to run `SCRIPT.py`
*(include the full path if the script is not the same directory
as where you want to run this)*.
Here is a basic SLURM job script that may work for you:

```
#!/bin/bash
#SBATCH -p gpu
#SBATCH --gres gpu:1
enable_lmod
module load container_env tensorflow-gpu/2.4.0
crun.tensorflow-gpu python3 SCRIPT.py
```

**Important notes:**

  * Don't worry why we recommend using the `tensorflow`
    [container](https://wiki.hpc.odu.edu/Containers/Container-Intro).
    This container has a installation of Python 3.7 that has fairly
    complete set of basic libraries used by Python users, such as
    `numpy`, `scipy`, `pandas`, `scikit-learn`, and of course,
    TensorFlow.
  * Replace `SCRIPT.py` with your actual script file name.
  * We assume that your workload needs a GPU to run your calculation.
    If you do not, please remove the `#SBATCH -p` and `#SBATCH --gres` lines
    and replace the words `tensorflow-gpu` with `tensorflow-cpu` is everywhere.

Create this script using a text editor, save it to a file (say, `JOB.sh`).
Then you will submit the script
from the UNIX shell interface of the cluster by typing:

```
$ sbatch JOB.sh
```

(The `$` at the beginning of the line represents
the shell prompt--do not type that.)
If successful, there will be a message printed
`Submitted batch job NNNNNN`
on the terminal,
where `NNNNNN` is an integer called job ID or job number.
The output file will be `slurm-NNNNNN.out`.

Here are a few documentations to help understand this process:

  * Documentation on Python on ODU HPC:
    <https://wiki.hpc.odu.edu/Software/Python>

  * General workflow of using SLURM job scheduler:
    <https://wiki.hpc.odu.edu/slurm#general-workflow-of-using-slurm>

If you prefer video introduction to SLURM, please take a look at this short video to understand what job scheduler is:

**"Working with SLURM"**<br/>
<https://odumedia.mediaspace.kaltura.com/playlist/dedicated/1_8eqsb16m/1_oy7ls1o0>

(This is part of the
[Intro to HPC](https://wiki.hpc.odu.edu/Training/HPC-Intro)
training videos.)


-----

*Back to your original question*:
Yes, you can run the script from a notebook.
Once you opened a (blank) notebook, you will enter this statement in a new Python cell:

```
%load SCRIPT.py
```

Press `<Shift+Enter>` once, the code will be loaded into that same cell.
Press `<Shift+Enter>` once more, the code will execute.
But please note that this mode of execution can be awkward, i.e.
we have to open a Jupyter session,
`%load` the script in order to run it.
The maximum time to run the script is under 24 hours (the max limit of Jupyter on HPC).
This mode is useful if you plan to interactively test or troubleshoot a script,
but less useful when you have a bunch of calculations to do,
or calculation that you expect to run for a long time.

*If you still have issue or question after reading this response:*
Can you share the script with us so we may understand the requirements of your script?