Skip to content

Commit f824364

Browse files
shumashMaria Masha Shugrina
and
Maria Masha Shugrina
authored
Added a fully functional and bundled kaolin-dash3d visualizer, composed of a server and a remote web client. (NVIDIAGameWorks#378)
Dash3d allows viewing remote 3D checkpoints written using kaolin Timelspase API from any machine in a web browser by connecting to a webapp served by the remote server. Components of this change: Added kaolin/experimental subdirectory for features less stable and well-tested Bundled kaolin-dash3d as an executable, installed if KAOLIN_INSTALL_EXPERIMENTAL variable is set Wrote a TimelapseParser class that can monitor a timelapse log directory for changes and summarize usd content Wrote a web server supporting web socket API Wrote a javascript web client with jquery and ThreeJS that can receive and visualize 3D data Set up javascript source compilation using uglifyjs Encapsulated thirdparty dependencies using npm Bundled thirdparty dependencies and acknowledged their use (MIT license) Wrote integration tests that perform visual regression on the web client running in the browser and connecting to a server running over a test set of logs Wrote integration tests for binary data serialization Integrated these tests into the CI pipeline Limitations: kaolin-dash3d support is limited to pointclouds and meshes with no color information big-endian clients and/or servers are likely to break the binary interchange format Signed-off-by: Maria Masha Shugrina <[email protected]> Co-authored-by: Maria Masha Shugrina <[email protected]>
1 parent 2158b5e commit f824364

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

73 files changed

+10102
-39
lines changed

.drone.yml

+7-1
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@ environment:
77
PYTORCH_VERSION: "1.6"
88
CUDNN_VERSION: "7"
99
TORCH_CUDA_ARCH_LIST: "5.2 6.0 6.1 7.0 7.5+PTX"
10+
KAOLIN_INSTALL_EXPERIMENTAL: "1"
1011

1112
steps:
1213
- name: submodules
@@ -42,6 +43,11 @@ steps:
4243
- export PATH=/opt/conda/bin:$PATH
4344
- pip install --upgrade pip>=19.3
4445
- conda install -y pytorch==1.6 cudatoolkit=10.1 -c pytorch
46+
- conda install -c conda-forge nodejs
47+
- >
48+
apt-get update && apt-get install -y --no-install-recommends
49+
libgtk2.0-0 libgtk-3-0 libgbm-dev libnotify-dev libgconf-2-4 libnss3 libxss1 libasound2 libxtst6 xauth xvfb
50+
- npm install
4551
- python setup.py develop
4652
- pip install -r tools/ci_requirements.txt
47-
- pytest --cov=kaolin/ tests/python && pytest --doctest-modules kaolin/
53+
- pytest --cov=kaolin/ tests/python tests/integration && pytest --doctest-modules kaolin/

.gitignore

+5
Original file line numberDiff line numberDiff line change
@@ -15,9 +15,14 @@ kaolin.egg-info
1515
*.egg
1616
.eggs/
1717
kaolin/cython/**/*.cpp
18+
node_modules/
1819

1920
# byte-compiled python files
2021
*.py[cod]
2122

2223
# pip cache
2324
.cache
25+
26+
# editor specific
27+
*~
28+
.idea

CONTRIBUTING.md

+7-3
Original file line numberDiff line numberDiff line change
@@ -47,7 +47,8 @@ Include a license at the top of new files.
4747
* [Python example](examples_license.py)
4848

4949
#### Code organization
50-
* [kaolin](kaolin/) - The Core of Kaolin library, everything that is not in [csrc](kaolin/csrc) is a Python module.
50+
* [kaolin](kaolin/) - The core Kaolin library, comprised of python modules,
51+
except for code under [csrc](kaolin/csrc) or [experimental](kaolin/experimental).
5152
* [csrc](kaolin/csrc/) - Directory for all the C++ / CUDA implementations of custom ops.
5253
The gpu ops parts will be under the subdirectory [csrc/cuda](kaolin/csrc/cuda)
5354
while the cpu parts will be under the subdirectory [csrc/cpu](kaolin/csrc/cpu).
@@ -57,6 +58,7 @@ Include a license at the top of new files.
5758
* [render](kaolin/render/) - Module of all the differentiable renderers modules and advanced implementations.
5859
* [utils](kaolin/utils/) - Module of all the utility features for debugging and testing.
5960
* [visualize](kaolin/visualize/) - Module of all the visualization modules.
61+
* [experimental](kaolin/experimental/) - Contains less thoroughly tested components for early adoption.
6062
* [examples](examples/) - Examples of Kaolin usage
6163
* [tests](tests/) - Tests for all Kaolin
6264

@@ -70,16 +72,18 @@ We follow [PEP8 Style Guide](https://www.python.org/dev/peps/pep-0008/) with som
7072

7173
It is enforced using [flake8](https://pypi.org/project/flake8/), with [flake8-bugbear](https://pypi.org/project/flake8-bugbear/), [flake8-comprehensions](https://pypi.org/project/flake8-comprehensions/), [flake8-mypy](https://pypi.org/project/flake8-mypy/) and [flake8-pyi](https://pypi.org/project/flake8-pyi/)
7274

73-
to run flake8 execute ``flake8 --config=.flake8 .`` from the [root of kaolin](https://gitlab-master.nvidia.com/Toronto_DL_Lab/kaolin-reformat)
75+
To run flake8 execute ``flake8 --config=.flake8 .`` from the [root of kaolin](https://gitlab-master.nvidia.com/Toronto_DL_Lab/kaolin-reformat).
7476

7577
On top of that we use prefixes (``packed\_``, ``padded\_``) to indicate that a module / op is specific to a layout, an , all ops of the same purpose for different layouts should be in the same file.
7678

7779
[tests/python/kaolin/](tests/python/kaolin) should follows the same directory structure of [kaolin/](kaolin/). E.g. each module kaolin/path/to/mymodule.py should have a corresponding tests/python/kaolin/path/to/test\_mymodule.py.
7880

7981
#### Tests
8082
We are applying [pytest](https://docs.pytest.org/en/latest/) on [tests/python directory](tests/python/), with [pytest-cov](https://pytest-cov.readthedocs.io/en/latest/), which should follows the directory structure of [kaolin](kaolin/).
83+
To run these tests execute ``pytest --cov=kaolin/ tests/`` from the [root of kaolin](https://gitlab-master.nvidia.com/Toronto_DL_Lab/kaolin-reformat).
8184

82-
to run the tests execute ``pytest --cov=kaolin/ tests/`` from the [root of kaolin](https://gitlab-master.nvidia.com/Toronto_DL_Lab/kaolin-reformat)
85+
Additional integration tests are located in [tests/integration](tests/integration).
86+
See that subdirectory for details.
8387

8488
#### Documentation
8589
Contributors are encouraged to verify the generated documentation before each pull request.

MANIFEST.in

+2
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
recursive-include kaolin/experimental/dash3d/static *
2+
recursive-include kaolin/experimental/dash3d/templates *

ci/pytest-linux.sh

+1-1
Original file line numberDiff line numberDiff line change
@@ -16,4 +16,4 @@ docker run \
1616
--runtime=nvidia \
1717
--rm \
1818
"$IMAGE_NAME" \
19-
/bin/bash -c "pytest --cov=kaolin/ tests/python && pytest --doctest-modules kaolin/"
19+
/bin/bash -c "pytest --cov=kaolin/ tests/python tests/integration && pytest --doctest-modules kaolin/"

docs/notes/installation.rst

+5-1
Original file line numberDiff line numberDiff line change
@@ -39,12 +39,16 @@ To install the library. You must first clone the repository:
3939
$ git clone --recursive https://github.com/NVIDIAGameWorks/kaolin
4040
$ cd kaolin
4141
42-
you can then select the tag if you want 0.9.0 or 0.1 specific release, example:
42+
You can then select the tag if you want 0.9.0 or 0.1 specific release, example:
4343

4444
.. code-block:: bash
4545
4646
$ git checkout v0.9.0
4747
48+
To enable installation of experimental features, set
49+
environment variable
50+
``export KAOLIN_INSTALL_EXPERIMENTAL=1``. To install, run:
51+
4852
.. code-block:: bash
4953
5054
$ python setup.py develop

examples/tutorial/visualize_main.py

+9-1
Original file line numberDiff line numberDiff line change
@@ -107,10 +107,14 @@ def __normalize_vertices(vertices):
107107
# Emulate a training update
108108
out_pt_clouds = []
109109
out_vert_list = []
110+
out_voxels = []
110111
for i in range(len(gt_vert_list)):
111112
delta_weight = 1.0 - iteration / (args.iterations - 1)
112113
out_vert_list.append(gt_vert_list[i] * (1.0 + delta_list[i] * delta_weight))
113114
out_pt_clouds.append(input_pt_clouds[i] * (1.0 + delta_pt_list[i] * delta_weight))
115+
vg = kaolin.ops.conversions.trianglemeshes_to_voxelgrids(
116+
out_vert_list[-1].unsqueeze(0), face_list[i], 30)
117+
out_voxels.append(vg.squeeze(0).bool())
114118

115119
# Save model predictions to track training progress over time
116120
timelapse.add_mesh_batch(
@@ -122,10 +126,14 @@ def __normalize_vertices(vertices):
122126
iteration=iteration,
123127
category='output',
124128
pointcloud_list=out_pt_clouds)
129+
timelapse.add_voxelgrid_batch(
130+
iteration=iteration,
131+
category='output',
132+
voxelgrid_list=out_voxels)
125133

126134
logger.info('Emulated training complete!\n'
127135
'You can now view created USD files found here: {}\n\n'
128-
'You will soon be able to visualize these in the Kaolin Research Omniverse App '
136+
'You will soon be able to visualize these in the Kaolin Omniverse App '
129137
'and our web visualizer. Stay tuned!'.format(args.output_dir))
130138

131139
# TODO(mshugrina): update command line once finalized

kaolin/experimental/dash3d/Makefile

+52
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
CORE_JS = $(wildcard src/*.js)
2+
CORE_CSS = $(wildcard src/*.css)
3+
THIRDPARTY_JS = ../../../node_modules/jquery/dist/jquery.min.js \
4+
../../../node_modules/three/build/three.min.js \
5+
../../../node_modules/three/examples/js/controls/OrbitControls.js # TODO: this needs to be minified
6+
THIRDPARTY_CSS = ../../../node_modules/mustard-ui/dist/css/mustard-ui.min.css
7+
JSMIN=uglifyjs
8+
AWK=awk
9+
10+
all: css js
11+
12+
js: jscore jsthirdparty
13+
14+
css: csscore cssthirdparty
15+
16+
info: info_jscore info_jsthirdparty info_csscore info_cssthirdparty
17+
18+
info_jscore: $(CORE_JS)
19+
echo "core js files: " $(CORE_JS)
20+
21+
info_jsthirdparty: $(THIRDPARTY_JS)
22+
echo "thirdparty js files: " $(THIRDPARTY_JS)
23+
24+
info_csscore: $(CORE_CSS)
25+
echo "core css files: " $(CORE_CSS)
26+
27+
info_cssthirdparty: $(THIRDPARTY_CSS)
28+
echo "thirdparty css files: " $(THIRDPARTY_CSS)
29+
30+
jscore: static/core-min.js
31+
32+
static/core-min.js: $(CORE_JS)
33+
cat src/copyright.js > static/core-min.js
34+
$(JSMIN) $^ -m >> static/core-min.js
35+
36+
jsthirdparty: static/thirdparty.js
37+
38+
static/thirdparty.js: $(THIRDPARTY_JS)
39+
$(AWK) 1 $^ > static/thirdparty.js
40+
41+
csscore: static/style.css
42+
43+
static/style.css: $(CORE_CSS)
44+
cat $^ > static/style.css
45+
46+
cssthirdparty: static/thirdparty.css
47+
48+
static/thirdparty.css: $(THIRDPARTY_CSS)
49+
cat $^ > static/thirdparty.css
50+
51+
clean:
52+
rm -f static/style.css static/core-min.js static/thirdparty.js static/thirdparty.css

kaolin/experimental/dash3d/README.md

+81
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,81 @@
1+
2+
# Kaolin Dash3D (beta)
3+
4+
WebGL dashboard that allows viewing checkpoints written
5+
with kaolin's `Timelapse` from any remote machine in a web browser.
6+
Kaolin Dash3D is a **lightweight alternative** to the Training Visualizer
7+
within the
8+
[Kaolin Omniverse App](https://docs.omniverse.nvidia.com/app_kaolin/app_kaolin/user_manual.html),
9+
which is a desktop application with wider visualization support
10+
and superior rendering quality.
11+
12+
Currently Dash3D can only display geometric data of the following types:
13+
14+
* 3d meshes (no texture or vertex color support)
15+
* point clouds (no color)
16+
17+
Voxelgrid, as well as texture and color support is planned.
18+
19+
20+
## Using
21+
22+
Simply run `kaolin-dash3d` on the machine that has your training checkpoints:
23+
24+
```
25+
$ kaolin-dash3d --logdir=$MY_EXPERIMENTS/test0/checkpts --port=8080
26+
```
27+
28+
Then, view the 3D checkpoints in a browser window locally by going to
29+
[localhost:8080](http://localhost:8080/), or connect to the server remotely.
30+
(Tip: use ssh port forwarding on your client machine.)
31+
32+
## Bundled Third-party code
33+
34+
We would like to acknowledge the following third-party code bundled with
35+
this application:
36+
* [jQuery](https://jquery.com/download/), released under [MIT License](https://jquery.org/license/)
37+
* [ThreeJS](https://github.com/mrdoob/three.js/), released under [MIT License](https://github.com/mrdoob/three.js/blob/dev/LICENSE)
38+
* [Mustard UI](https://kylelogue.github.io/mustard-ui/index.html), released under [MIT License](https://github.com/kylelogue/mustard-ui/blob/master/LICENSE)
39+
40+
## Developing
41+
42+
#### Dependencies
43+
44+
Third party dependencies for the javascript frontend as well as for
45+
testing are managed by `npm`. To fetch dependencies, first
46+
install `nodejs`, which is available through `conda`, and run `npm install`
47+
from the root of kaolin (this will install both production and development
48+
dependencies).
49+
50+
To add new dependencies, run this from the root of kaolin:
51+
```
52+
npm install <PKG> --save-dev
53+
```
54+
This will also update the `package.json` file in the root of kaolin.
55+
If the new dependencies must be served to the client, omit
56+
`--save-dev` from the command, and add the appropriate source to
57+
the `Makefile` in this directory.
58+
59+
#### Compiling client side code
60+
61+
Compiled source and thirdparty javascript is bundled with the app in the
62+
`static` subdirectory, while source javascript and CSS code is
63+
located under `src/`. After edits, it
64+
must be compiled into an optimized version to be served:
65+
```
66+
$ cd kaolin/visualize/dash3d
67+
$ make clean
68+
$ make
69+
```
70+
Note that this also compiles thirdparty dependencies which are
71+
assumed to reside in `node-modules` in the root of `kaolin`, a
72+
directory created automatically by `npm`. This is not ideal,
73+
but for now we avoid setting up tools like WebPack,
74+
given our small set of dependencies.
75+
76+
#### Integration testing
77+
78+
Integration tests are located in `kaolin/tests/integration`. Currently
79+
the set of implemented tests is limited and excluded from continuous
80+
integration due to their complexity. Please refer to the README
81+
in the test directory.

kaolin/experimental/dash3d/__init__.py

Whitespace-only changes.
+6
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
#!/usr/bin/env python3
2+
3+
from kaolin.experimental.dash3d.run import run_main
4+
5+
if __name__ == "__main__":
6+
run_main()

kaolin/experimental/dash3d/run.py

+86
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,86 @@
1+
# Copyright (c) 2019-2021, NVIDIA CORPORATION. All rights reserved.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
from __future__ import print_function
15+
16+
import argparse
17+
import logging
18+
import os
19+
import sys
20+
21+
from flask import Flask, render_template
22+
from tornado.wsgi import WSGIContainer
23+
from tornado.web import Application, FallbackHandler
24+
from tornado.ioloop import IOLoop
25+
26+
from kaolin.experimental.dash3d.util import StreamingGeometryHelper, GeometryWebSocketHandler
27+
28+
logger = logging.getLogger(__name__)
29+
30+
31+
def create_server(logdir):
32+
""" Create the server, including websocket handler through tornado, and flask http server.
33+
34+
Args:
35+
logdir (str): directory where Timelapse data is written.
36+
"""
37+
# Helper for streaming geometry from the logdir
38+
helper = StreamingGeometryHelper(logdir)
39+
40+
# Flask for HTTP
41+
_base_dir = os.path.dirname(__file__)
42+
_template_dir = os.path.join(_base_dir, 'templates')
43+
_static_dir = os.path.join(_base_dir, 'static')
44+
app = Flask('kaolin_dash3d',
45+
template_folder=_template_dir,
46+
static_url_path='/static',
47+
static_folder=_static_dir)
48+
49+
@app.route('/')
50+
def index():
51+
helper.parser.check_for_updates()
52+
return render_template('home.html', logdir=helper.logdir,
53+
nmeshes=helper.parser.num_mesh_categories(),
54+
npointclouds=helper.parser.num_pointcloud_categories())
55+
56+
# Tornado server to handle websockets
57+
container = WSGIContainer(app)
58+
server = Application([
59+
(r'/websocket/', GeometryWebSocketHandler, dict(helper=helper)),
60+
(r'.*', FallbackHandler, dict(fallback=container))
61+
])
62+
return server
63+
64+
65+
def run_main():
66+
aparser = argparse.ArgumentParser(
67+
description='NVIDIA Kaolin Tensorboard 3d visualizer for USD files generated during training.')
68+
aparser.add_argument('--logdir', action='store', type=str, required=True,
69+
help='The vis folder generated by the Timelapse module.')
70+
aparser.add_argument('--log_level', action='store', type=int, default=logging.INFO,
71+
help='Logging level, DEBUG: 10, INFO: 20, WARN: 30, ERROR: 40.')
72+
aparser.add_argument('--port', action='store', default=8080)
73+
args = aparser.parse_args()
74+
75+
logging.basicConfig(level=args.log_level,
76+
format='%(asctime)s|%(levelname)8s|%(name)15s| %(message)s',
77+
handlers=[logging.StreamHandler(sys.stdout)])
78+
79+
print(f'Dash3D server starting. Go to: http://localhost:{args.port}')
80+
server = create_server(args.logdir)
81+
server.listen(args.port)
82+
IOLoop.instance().start()
83+
84+
85+
if __name__ == "__main__":
86+
run_main()

0 commit comments

Comments
 (0)