You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+23-28Lines changed: 23 additions & 28 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -13,21 +13,12 @@ The main tool of this package is **nbstat** / **nbwatch** command line utility.
13
13
14
14
<imgsrc="images/nbwatch.gif"width="90%"/>
15
15
16
-
While in the `watch` mode, you can hit buttons to modify the displayed view:
17
-
18
-
*`tab` — swaps views, from `nbwatch` to `devicewatch` and back.
19
-
*`b` — toggles bar representation for some of the resources: in addition to its value, show colored bar.
20
-
*`m` — toggles moving average column for some of the resources: values are averaged across the latest iterations.
21
-
*`s` — toggles table separators.
22
-
23
-
We also add the **devicestat** and **devicewatch** commands that show transposed view with the same information and parameters.
24
-
25
16
For more information, check out the full [user documentation:](nbtools/nbstat/README.md) explanation of different table views, command line options and ready-to-use snippets.
26
17
27
18
28
19
### Troubleshooting: PID namespaces, user permissions and zombie processes
29
20
A [known problem](https://github.com/NVIDIA/nvidia-docker/issues/179) of NVIDIA drivers is that **nvidia-smi** reports PIDs of processes on devices in the global namespace, not in the container namespace, which does not allow to match PIDs of container processes to their device PIDs. There are a few workarounds:
30
-
* pass `--pid=host` flag to `docker run`.
21
+
*[recommended]pass `--pid=host` flag to `docker run`.
31
22
* patch NVIDIA driver to handle PID namespaces correctly.
32
23
*[Linux only] fallback on manually inspecting */proc/PID/* files to identify the host PID for each process inside of the container.
33
24
@@ -42,8 +33,11 @@ In order to inspect certain properties of processes, we rely on having all neces
42
33
### Contribute
43
34
If you are interested to contribute, check out the [developer/contributor page.](nbtools/nbstat/DEV.md) It contains detailed description about inner workings of the library, my design choices and motivation behind them, as well as discussion of complexities along the way.
44
35
36
+
## Library
37
+
Other than `nbstat / nbwatch` monitoring utilities, this library provides a few useful tools for working with notebooks and GPUs.
38
+
45
39
46
-
## **pylint_notebook**
40
+
###**pylint_notebook**
47
41
Shamelessly taken from [pylint page:](https://pylint.pycqa.org/en/latest/)
48
42
49
43
Function that checks for errors in Jupyter Notebooks with Python code, tries to enforce a coding standard and looks for code smells. It can also look for certain type errors, it can recommend suggestions about how particular blocks can be refactored and can offer you details about the code's complexity.
@@ -58,37 +52,38 @@ pylint_notebook(path_to_ipynb, # If not provided, use path to the cu
58
52
59
53
Under the hood, it converts `.ipynb` notebook to `.py` script, creates a custom `.pylintrc` configuration, runs the `pylint` and removes all temporary files. Learn more about its usage in the [tutorial.](tutorials/NBstat.ipynb)
60
54
61
-
## **run_notebook**
55
+
###**exec_notebook**
62
56
Provides a `eval`-like interface for running Jupyter Notebooks programmatically. We use it for running interactive tests, that are easy to work with: in case of any failures, one can jump right into fixing it with an already set-up environment.
63
57
64
58
```python
65
-
from nbtools importrun_notebook
66
-
run_notebook(path_to_ipynb, # Which notebook to run
67
-
out_path_ipynb, # Where to save result
68
-
inputs={'learning_rate': 0.05,}, # Pass variables to notebook
69
-
outputs=['accuracy']) # Extract variables from notebook
59
+
from nbtools importexec_notebook
60
+
exec_notebook(path_to_ipynb, # Which notebook to run
61
+
out_path_ipynb, # Where to save result
62
+
inputs={'learning_rate': 0.05,}, # Pass variables to notebook
63
+
outputs=['accuracy']) # Extract variables from notebook
70
64
```
71
65
72
66
73
-
## **set_gpus**
67
+
###**set_gpus, free_gpus**
74
68
Select free device(s) and set `CUDA_VISIBLE_DEVICES` environment variable so that the current process sees only them.
75
69
76
70
Eliminates an enormous amount of bugs and unexpected behaviors related to GPU usage.
77
71
78
72
```python
79
-
from nbtools import set_gpus
80
-
set_gpus(n=2, # Number of devices to set.
81
-
min_free_memory=0.7, # Minimum amount of free memory on device to consider it free.
82
-
max_processes=3) # Maximum amount of processes on device to consider it free.
73
+
from nbtools import set_gpus, free_gpus
74
+
used_gpus = set_gpus(n=2, # Number of devices to set.
75
+
min_free_memory=0.7,# Minimum amount of free memory on device to consider free.
76
+
max_processes=3) # Maximum amount of processes on device to consider free.
77
+
free_gpus(used_gpus) # Kill all processes on selected GPUs. Useful at teardown.
83
78
```
84
79
85
-
## Other functions
80
+
###Other functions
86
81
```python
87
-
from nbtools import (in_notebook, # Return True if executed inside of Jupyter Notebook
88
-
get_notebook_path, # If executed in Jupyter Notebook, return its absolute path
89
-
get_notebook_name, # If executed in Jupyter Notebook, return its name
90
-
notebook_to_script) # Convert Jupyter Notebook to an executable Python script.
91
-
# Works well with magics and command line executions.
82
+
from nbtools import (in_notebook, # Return True if executed inside of Jupyter Notebook
83
+
get_notebook_path, # If executed in Jupyter Notebook, return its absolute path
84
+
get_notebook_name, # If executed in Jupyter Notebook, return its name
85
+
notebook_to_script) # Convert Jupyter Notebook to an executable Python script.
86
+
# Works well with magics and command line executions.
0 commit comments