Commit ed9242b3 authored by Kate Nepovinnykh's avatar Kate Nepovinnykh
Browse files

Modified ReadMe, added docker utils

parent 1d94f672
......@@ -6,11 +6,11 @@
Contact your supervisor to get login credentials to access the GPU server.
Access using the secure shell:
* Linux or MacOS: e.g.,
* Linux, MacOS or Windows (Powershell): e.g.,
```
$ ssh lut4753.pc.lut.fi
```
* Windows: see PuTTY (http://www.putty.org/)
* Windows: see PuTTY (http://www.putty.org/) if using old version and ssh is not supported in Powershell
## Available hardware
......@@ -65,12 +65,20 @@ scp -r /home/user/code user_login@server_ip:/media/students/user/
scp -r /home/user/data user_login@server_ip:/media/students/user/
```
## Proposed workflow for new users
Proposed workflow uses docker + conda to ensure complete isolation and independence. This workflow is also oriented o the usage of [Jupyter Lab](https://jupyterlab.readthedocs.io/en/stable/).
For the convenience of users, we have prepared a number of bash scripts to simplify the workflow. [Relevant scripts and documetation](docker_utils/README.md).
## Dependencies and software installation
We are trying to keep the server clean and to avoid version conflicts. Thus,
docker is a preferred general way of handling dependencies (see docker section).
Python users may also consider using virtualenv (see Python section).
### Docker
From wiki:
......
# DOCKER UTILS FOR WORKING WITH CUDA AND PYTHON
## Introduction
Bash files presented in this directory are created with a goal to simplify the workflow with Docker for users with no prior experience. The idea is to imitate Jupyter `lab/notebook` workflow while using the isolated environment of a docker container for CUDA drivers installation. That way, all users are independent from the system-wide install of CUDA and safe from all sudden updates and changes.
All files work with cuda_conda docker image, which includes CUDA 11 with conda installation.
For the ease of use, you can put all bash files in your home directory and immediately start work by calling them after logging in with ssh.
## Initialization
In order to initialize everything for the intended workflow, run `init_container.sh`. The initialization might take approximately 5 minutes. After that, you can use other provided scripts.
What it does:
1. It creates directory `docker` in your user's media volume. That directory will be mounted to the container and will serve as a persistent data storage and allow users to work with large datasets and save their computation results and source files. The location is `/media/$USER/docker` or `/media/students/$USER/docker` depending on the type of your account. It will automatically create `work` directory, which is intended to be the main working directory. Other subdirectories in `docker` will be populated automatically by the initialization procedure.
2. It will create a conda environment in the `/media/$USER/docker/env` (or `/media/students/$USER/docker/env`) directory and make it a default conda environment. That way, all python and conda packages installed during work sessions with the container will be preserved for the future use. Users can freely stop and run new containers without worrying that their packages will be lost, nor do they need to rebuild the whole image after adding new packages.
3. Both Pytorch and TensorFlow are installed in the new environment along with the Jupyter server.
## Proposed workflow for Jupyter
(First, the container should be initialized with `init_container.sh`. Do this once.)
1. Run `jupyter_container.sh`. After the server starts, you can detach from the container using `Ctrl+P, Ctrl+Q` (this hint will be printed by the command) to allow it to run in background.
Script will attempt to run it using port 8888. If it is already in use, it will try to find the free port and it will print the final used port. Otherwise, if you want to specify other port for the Jupyter, you may use `-p | --port` flag, i.e. `-p 8885`. If you need several ports, for example for tensorboard, you may specify using `<host port>:<container port>` notation. For example `jupyter_container.sh -p 8885 -p 6666:6006` will forward container ports 8885 and 6006 to 8885 and 6666 on host. You will be able to access Jupyter using localhost:8885 and tensorboard (if it uses 6006 in the container) using localhost:6006.
2. Go to `localhost:8888` (or other port, if you specified it or if 8888 is already in use) and start working.
3. By default, jupyter will open folder `/media/$USER/docker/work"` (or `/media/students/$USER/docker/work"`). You may also put any necessary data and source files inside this directory. For example, you can create folders `src` and `data` for the source files and data respectively. Inside the contaier you can access this data using path `/$USER/work/...`, where `$USER` is your username.
## General information
By default, all scripts work with a container with a name `$USER_work` (i.e., `ekaterina_work` for user ekaterina, or `fedor_work` for user fedor). If, for some reason (e.g. you want to run several containers simultaneously), you need other name for a container, you can specify it using flag `-n` or `--name`, e.g. `bash_container.sh -n my_container`. Do not forget to stop and remove containers that you do not need (you may use `stop_remove_container.sh`).
Please keep in mind that containers of other people are also visible to you. DO NOT stop or remove containers of other users, unless specifically given permission by the creator of the container.
By default, scripts will try to run a container, but if it is already running, they will just execute the command inside a running container. You can change this behaviour by using `-r | --run` or `-e | --execute` flags.
Help is available by calling any script with `-h | --help` flag.
## Scripts overview
attach_container.sh - attach to a running container. For example, if you have started jupyter server using `jupyter_container.sh`, you might want to attach to it in order to stop jupyter server (and, by extension, the container itself).
`bash_container.sh` - run an interactive bash terminal inside a container.
`execute_in_container.sh` - execute a command inside a container. You have to specify the command, but keep in mind that only the commands from inside the container system are available. For example, if you want to run some python file, you can put it in `/media/$USER/docker/work` (or `/media/students/$USER/docker/work` if you are a student) directory, i.e `/media/$USER/docker/work/test.py` and run it using path `/$USER/src/test.py`. The full command will look like this: `execute_in_container.sh python /$USER/src/test.py` (or `execute_in_container.sh python /ekaterina/src/test.py` if your username is `ekaterina`).
`init_container.sh` - initialize container with a conda environment on a mountable drive. Call this before calling other scripts. Call this only once.
`jupyter_container.sh` - run jupyter lab in a container. Uses default port 8888, you can change it by using `-p` or `--port` flag. After the server starts, you can detach from the container using `Ctrl+P,Q` (this hint will be printed by the command) to allow it to run in background. If you need to reattach to stop it or check outputs, you can use `attach_container.sh`.
`run_background_container.sh` - starts container and detaches from it immediately. This container will run in background until you stop it with `stop_remove_container.sh` (or docker commands). Might be useful if you want to execute bash terminal in it (e.g., using `bash_container.sh`) and run some long process in background using `screen` command. Overall, the use of this script is discouraged.
`stop_remove_container.sh` - stops and removes a running container.
\ No newline at end of file
#!/bin/bash
name="$USER"_work
usage()
{
echo "Help:"
echo "-n, --name - name of a container. Default is" $USER"_work."
}
while [ "$1" != "" ]; do
case $1 in
-n | --name ) shift
name="$1"
;;
-h | --help ) usage
exit
;;
* ) break
esac
shift
done
docker attach $name
#!/bin/bash
dir="`dirname "$(readlink -f "$0")"`"
$dir/execute_in_container.sh -i "$@" /bin/bash
#!/bin/bash
name="$USER"_work
source="/media/$USER/docker"
it_flag=
d_flag=
run=
exec=
ports=()
ports_arg=()
re='^[0-9]+$'
usage()
{
echo "Help:"
echo "-n, --name - name of a container. Default is" $USER"_work."
echo "-i, --interactive - whether to start the command in interactive mode."
echo "-d, --detach - whether to detach shell from the command."
echo "-r, --run - whether to run container if it is not running."
echo "-e, --exec - whether to execute command if the container is already running."
echo "-h, --help - show this message."
echo "-p, --port - choose a port to forward from the container. "
echo " For example, if you specify port as a number, e.g. 8888 (standard Jupyter port), container port 8888 will be forwarded to host port 8888. "
echo " Otherwise, you can specify port as <host port>:<container port> (i.e. 8880:80) if you want to connect it to a different host port."
echo ""
echo "Note: if neither -r nor -e are specified, the default mode is to allow both."
}
if [ ! -d "$source" ]; then
source="/media/students/$USER/docker"
fi
while [ "$1" != "" ]; do
case $1 in
-n | --name ) shift
name="$1"
;;
-p | --port ) shift
ports+=( "$1" )
;;
-i | --interactive ) it_flag="-it"
;;
-d | --detach ) d_flag="-d"
;;
-r | --run ) run=1
;;
-e | --exec ) exec=1
;;
-h | --help ) usage
exit
;;
* ) break
esac
shift
done
if [ "$run" != "1" ] && [ "$exec" != "1" ]; then
run="1"
exec="1"
fi
for port in ${ports[@]}; do
if [[ $port =~ $re ]]; then
for i in {1..10}; do
nc -z 127.0.0.1 $port
if [ $? == 1 ]; then
echo "Port $port is available"
break
else
echo "Port $port is not available"
port=$((port+1))
echo "Trying port $port..."
fi
done
ports_arg+=( "-p" "$port:$port" )
else
ports_arg+=( "-p" $port )
fi
done
CONTAINER_PATH=/$USER/env/work/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
JUPYTER_CONFIG_DIR=/$USER/.jupyter/.config
JUPYTER_DATA_DIR=/$USER/.jupyter
JUPYTER_RUNTIME_DIR=/$USER/.jupyter/runtime
if [ ! "$(docker ps -q -f name=$name)" ]; then
if [ "$run" = "1" ]; then
if [ "$(docker ps -aq -f status=exited -f name=$name)" ]; then
echo "Removing stopped container $name"
docker rm $name
fi
echo "Running container $name with command \"$@\""
docker run $it_flag $d_flag --name $name --gpus all "${ports_arg[@]}" \
--env USER --env PATH=$CONTAINER_PATH \
--env JUPYTER_CONFIG_DIR=$JUPYTER_CONFIG_DIR \
--env JUPYTER_DATA_DIR=$JUPYTER_DATA_DIR \
--env JUPYTER_RUNTIME_DIR=$JUPYTER_RUNTIME_DIR \
--mount type=bind,source=$source,target=/$USER \
cuda_conda:latest "$@"
fi
else
if [ "$exec" = "1" ]; then
echo "Executing \"$@\" in the container $name"
docker exec $it_flag $d_flag $name "$@"
fi
fi
#!/bin/bash
source="/media/$USER"
if [ ! -d "$source" ]; then
source="/media/students/$USER"
fi
mkdir $source/docker
mkdir $source/docker/env
mkdir $source/docker/.jupyter
mkdir $source/docker/work
dir="`dirname "$(readlink -f "$0")"`"
$dir/execute_in_container.sh "$@" /home/init_container_internal.sh
#!/bin/bash
dir="`dirname "$(readlink -f "$0")"`"
jupyter_port="8888"
args=("$@")
usage()
{
echo "Help:"
echo "-n, --name - name of a container. Default is" $USER"_work."
echo "-i, --interactive - whether to start the command in interactive mode."
echo "-d, --detach - whether to detach shell from the command."
echo "-r, --run - whether to run container if it is not running."
echo "-e, --exec - whether to execute command if the container is already running."
echo "-h, --help - show this message."
echo "-p, --port - choose a port to forward from the container. The first port will be used for jupyter lab and should be specified as a number. The default is 8888."
echo " For example, if you specify port as a number, e.g. 8888 (standard Jupyter port), container port 8888 will be forwarded to host port 8888. "
echo " Otherwise, you can specify port as <host port>:<container port> (i.e. 8880:80) if you want to connect it to a different host port."
echo ""
echo "Note: if neither -r nor -e are specified, the default mode is to allow both."
}
for (( i=0; i<$#; i++ )); do
arg=${args[i]}
if [[ $arg == "-p" ]] || [[ $arg == "--port" ]] ; then
jupyter_port=${args[i+1]}
args=( "${args[@]:0:$i}" "${args[@]:$((i + 2))}" )
break
elif [[ $arg == "-h" ]] || [[ $arg == "--help" ]] ; then
usage
exit
fi
done
for i in {1..10}; do
nc -z 127.0.0.1 $jupyter_port
if [ $? == 1 ]; then
echo "Port $jupyter_port is available"
break
else
echo "Port $jupyter_port is not available"
jupyter_port=$((jupyter_port+1))
echo "Trying port $jupyter_port..."
fi
done
args+=( '-p' $jupyter_port )
echo "Use Ctrl+P,Q to detach from container. Use \"docker attach <container name>\" if you need to check Jupyter status or stop it."
$dir/execute_in_container.sh -i "${args[@]}" jupyter lab --notebook-dir=/$USER/work/ --no-browser --ip=0.0.0.0 --allow-root --port=$jupyter_port
#!/bin/bash
name="$USER"_work
dir="`dirname "$(readlink -f "$0")"`"
$dir/execute_in_container.sh -d -i -r "$@" /bin/bash
#!/bin/bash
name="$USER"_work
usage()
{
echo "Help:"
echo "-n, --name - name of a container. Default is" $USER"_work."
}
while [ "$1" != "" ]; do
case $1 in
-n | --name ) shift
name="$1"
;;
-h | --help ) usage
exit
;;
* ) break
esac
shift
done
docker stop $name
docker rm $name
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment