You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
* Water detection (using the [Water Observations from Space](https://www.ga.gov.au/scientific-topics/community-safety/flood/wofs) algorithm)
27
27
28
-
## Installation
28
+
## Guide
29
29
-------
30
30
31
-
First follow the instructions in the [Docker Installation Guide](docs/docker_install.md) if you do not have Docker installed yet.
31
+
First follow the instructions in the [Environment Setup Guide](https://ceos-odc.readthedocs.io/en/latest/modules/install_docs/environment_setup.html) if you have not yet.
32
32
33
-
Follow the instructions in the
34
-
[Open Data Cube Database Installation Guide](docs/odc_db_setup.md) to setup the Open Data Cube (ODC) database.
33
+
Follow the instructions in the [Open Data Cube Database Installation Guide](https://ceos-odc.readthedocs.io/en/latest/modules/install_docs/database_install.html) to setup the Open Data Cube (ODC) database.
35
34
36
-
Follow the instructions in the [Open Data Cube UI Installation Guide](docs/ui_install.md) to install the ODC UI. That document also contains troubleshooting information for the UI in the form of an FAQ at the end.
35
+
Follow the instructions in the [Open Data Cube UI Guide](docs/ui_guide.md) to install the ODC UI. That document also contains troubleshooting information for the UI in the form of an FAQ at the end.
37
36
38
37
Follow the instructions in the [Open Data Cube UI Algorithm Addition Guide](docs/adding_new_pages.md) to add new applications to the ODC UI. This guide is only intended for programmers.
39
38
@@ -47,4 +46,4 @@ If you encounter issues with Open Data Cube or this UI that are not documented i
47
46
## More
48
47
-------
49
48
50
-
You may also consider running a Jupyter Notebook server that uses the Open Data Cube. The CEOS ODC Notebooks repository is [here](https://github.com/ceos-seo/data_cube_notebooks).
49
+
You may also consider running a Jupyter Notebook server that uses the Open Data Cube. The CEOS ODC Notebooks repository is [here](https://github.com/ceos-seo/data_cube_notebooks).
Copy file name to clipboardExpand all lines: docs/adding_new_pages.md
+67-68Lines changed: 67 additions & 68 deletions
Original file line number
Diff line number
Diff line change
@@ -1,37 +1,33 @@
1
-
Open Data Cube UI Algorithm Addition Guide
2
-
=================
1
+
# Open Data Cube UI Algorithm Addition Guide
3
2
4
3
This document will guide users through the process of adding new algorithms or analysis cases to the UI, including the general Django files and the Celery workflow. This guide assumes that the user has a working Data Cube installation and has completed the full UI installation guide.
5
4
6
-
Contents
7
-
=================
5
+
-[Introduction](#introduction)
6
+
-[Basic Components](#basic-components)
7
+
-[Base Classes](#base-classes)
8
+
-[Generating Base Files](#generating-base-files)
9
+
-[Basic Applications](#basic-applications)
10
+
-[Complex Applications](#complex-applications)
11
+
-[Template Changes](#template-changes)
8
12
9
-
*[Introduction](#introduction)
10
-
*[Basic Components](#basic_components)
11
-
*[Base Classes](#base_classes)
12
-
*[Generating Base Files](#base_files)
13
-
*[Basic Applications](#band_math_app)
14
-
*[Complex Applications](#complex_app)
15
-
*[Template Changes](#templates)
16
-
<!-- * [Common problems/FAQs](#faqs) -->
13
+
## Introduction
14
+
-----
17
15
18
-
<aname="introduction"></a> Introduction
19
-
=================
20
16
The Data Cube UI is designed to be quickly and easily extended to integrate new algorithms that operate on Data Cube based raster datasets. A new algorithm implementation includes three main parts:
21
17
22
-
* Django views - map URLs to rendered HTML pages
23
-
* Django models - Hold the main task attributes (parameters, results, etc)
24
-
* Celery workflows - Process asynchronous tasks, populating a Django model with updates and results
18
+
- Django views - map URLs to rendered HTML pages
19
+
- Django models - Hold the main task attributes (parameters, results, etc)
20
+
- Celery workflows - Process asynchronous tasks, populating a Django model with updates and results
25
21
26
22
Two manage.py commands are provided to expedite this processing, requiring only simple and defined changes to get an application working.
27
23
28
-
<aname="basic_components"></a> Basic Components
29
-
=================
24
+
## Basic Components
25
+
-----
26
+
30
27
The three main components of a new algorithm implementation were outlined above - the views, models, and celery tasks.
31
28
32
-
Django Views
33
-
--------
34
-
```
29
+
>### Django Views
30
+
```python
35
31
classSubmitNewRequest(SubmitNewRequest):
36
32
"""
37
33
Submit new request REST API Endpoint
@@ -54,9 +50,8 @@ class SubmitNewRequest(SubmitNewRequest):
54
50
55
51
Each view class subclasses the base dc_algorithm class so only minor changes need to be made for a new page. Functions and variables that need to be provided are defined in the base class docstring, and if they are omitted then an exception is raised. HTML content that is rendered in the browser is controlled here - forms, panels, etc.
56
52
57
-
Django models
58
-
------
59
-
```
53
+
>### Django Models
54
+
```python
60
55
classQuery(BaseQuery):
61
56
"""
62
57
@@ -154,11 +149,10 @@ class Query(BaseQuery):
154
149
155
150
Django models contain all of the information and parameters required to perform your operation and create output products. This includes where to save the output products, how to split data for processing, and the Python function that should be used to create the output products. All input and output parameters should be enumerated here.
156
151
157
-
Celery tasks
158
-
------
152
+
>### Celery tasks
159
153
Celery tasks perform all processing for the algorithms. Tasks use information found on the models to separate data chunks into manageable sizes, perform analysis functions, and create output products.
"""Chunk parameter sets into more manageable sizes
@@ -253,23 +247,24 @@ The Celery tasks open models by their id and parse out the required information.
253
247
254
248
The processing pipeline in tasks.py is organized so that each task operates independently of one another - the process is fully asynchronous and non-blocking. The tasks are also arranged so that you work down the page as the task executes, and each task completes a single function.
255
249
256
-
* Parameters are parsed out from the task model
257
-
* Parameters are verified and validated
258
-
* Parameters are chunked into smaller, more manageable pieces for parallelized processing
259
-
* A processing pipeline is created from the parameter chunks and submitted for processing - this involves both geographic and time based chunks
260
-
* Each chunk is processed, with the results being saved to disk. Metadata is collected here and passed forward
261
-
* The chunks are combined both geographically and over time, combining metadata as well
262
-
* The output products are generated and the model is updated
250
+
- Parameters are parsed out from the task model
251
+
- Parameters are verified and validated
252
+
- Parameters are chunked into smaller, more manageable pieces for parallelized processing
253
+
- A processing pipeline is created from the parameter chunks and submitted for processing - this involves both geographic and time based chunks
254
+
- Each chunk is processed, with the results being saved to disk. Metadata is collected here and passed forward
255
+
- The chunks are combined both geographically and over time, combining metadata as well
256
+
- The output products are generated and the model is updated
257
+
258
+
## Base Classes
259
+
-----
263
260
264
-
<aname="base_classes"></a> Base Classes
265
-
=================
266
261
Django models and views allow for standard Python inheritance, allowing us to simply subclass a common dc_algorithm base to quickly create new apps. Take some time to look through the files in `apps/dc_algorithm` - the docstrings explain exactly what each view and model does and what attributes are required.
267
262
268
263
The main portion of the dc_algorithm base app lies within `views.py` and `models/abstract_base_models.py`.
269
264
270
-
Below is the base class for task submission. You'll notice the extensive docstrings outlining all required attributes and parameters. Additionally, there are 'getter' functions that raise a NotImplementedError when a required attribute is not present. Compare this class to the implementation in <aname="basic_components">Basic Components</a> - Only the required attributes are defined, and everything else lies within the base class.
265
+
Below is the base class for task submission. You'll notice the extensive docstrings outlining all required attributes and parameters. Additionally, there are 'getter' functions that raise a NotImplementedError when a required attribute is not present. Compare this class to the implementation in [Basic Components](#basic-components) - Only the required attributes are defined, and everything else lies within the base class.
271
266
272
-
```
267
+
```python
273
268
classSubmitNewRequest(View, ToolClass):
274
269
"""Submit a new request for processing using a task created with form data
275
270
@@ -365,41 +360,43 @@ class SubmitNewRequest(View, ToolClass):
365
360
366
361
Model base classes work in the same way - common attributes are defined, and users are free to add or remove fields in child classes.
367
362
368
-
<aname="base_files"></a> Generating Base Files
369
-
=================
363
+
## Generating Base Files
364
+
-----
365
+
370
366
Two manage.py commands are provided to easily create appliations - one for simple algorithms (band math, any application that should be run on a mosaic) and a more flexible base used for more complex concepts.
371
367
372
-
```
368
+
```bash
373
369
python manage.py start_bandmath_app app_name
374
370
python manage.py start_dc_algorithm_app app_name
375
371
```
376
372
377
373
These commands do a few things:
378
374
379
-
* Copy the associated base files to apps/app_name
380
-
* Rename all of the templated values for app_name in all of the Python files, models, views, forms, tasks, etc.
381
-
* Create a dc_algorithm.models.Application model for your new app
382
-
* Provides instructions on next steps to get your application working
375
+
- Copy the associated base files to apps/app_name
376
+
- Rename all of the templated values for app_name in all of the Python files, models, views, forms, tasks, etc.
377
+
- Create a dc_algorithm.models.Application model for your new app
378
+
- Provides instructions on next steps to get your application working
383
379
384
380
The instructions include to run the migrations and to initialize the database with your new models. These base files contain a few `TODO:` statements within the files, marking where inputs are required. To integrate a new algorithm, generate an app using the command above, grep/search for instance of `TODO:` and follow the instructions there.
385
381
386
382
Apps come in two main classes - Band math-like apps and more complex apps. Band math apps perform compositing over the selected time and geographic area then perform some function on the resulting mosaic. The dc_algorithm app is set up to be more general and will require more input. As a general rule, if an algorithm doesn't have a time series component, you can use the band math app.
387
383
388
384
For example, fractional cover involves creating a mosaic and running a computationally intensive function on the resulting data, so the band math base app was used. Coastal change involves non standard time chunking, animation generation, etc. so the more generalized app was used.
389
385
390
-
<aname="band_math_app"></a> Basic Applications
391
-
=================
386
+
## Basic Applications
387
+
-----
388
+
392
389
For a basic application, there are very few changes required to have a functional app. After generating the base files, run the migrations commands as seen below.
393
390
394
-
```
391
+
```bash
395
392
python manage.py makemigrations
396
393
python manage.py migrate
397
394
```
398
395
399
396
Now you will have all of the base tables generated in the database and are ready to implement your algorithm. If you search for `TODO` in that directory, there will only be one occurrence in `tasks.py` and one in models.
Replace the expression above with your band math-like algorithm. If it is somewhat more complicated, like fractional cover, the snippet will look like the snippet below:
@@ -426,7 +423,7 @@ Replace the expression above with your band math-like algorithm. If it is somewh
426
423
427
424
Now that you've implemented your algorithm, you'll need to handle the output. The base application will produce a single true color mosaic and the result of your band math. To do this, a color scale needs to be provided. The default color scale is a simple red->green scale for 0%->100% - to replace this, create a [GDALDEM compatible color scale](http://www.gdal.org/gdaldem.html#gdaldem_color_relief) and name it the same as your app (app_name) and place it in utils/color_scales.
428
425
429
-
```
426
+
```bash
430
427
-0.40 172 21 14
431
428
-0.30 247 103 50
432
429
-0.20 249 133 20
@@ -442,17 +439,18 @@ nan 0 0 0
442
439
443
440
Now that this is all complete, you can see your working application by:
444
441
445
-
* Restarting Apache2
446
-
* Restarting Celery workers
447
-
* Go to your site and select your application under Tools, choose your area, and submit a task.
442
+
- Restarting Apache2
443
+
- Restarting Celery workers
444
+
- Go to your site and select your application under Tools, choose your area, and submit a task.
445
+
446
+
## Complex Applications
447
+
-----
448
448
449
-
<aname="complex_app"></a> Complex Applications
450
-
=================
451
449
The process for implementing an advanced application is similar to the band math app. Generate the base application using the manage.py command, but don't run the migrations yet. You can start by Grepping/Searching for all instance of `TODO` and filling in the information that you're able to.
452
450
453
451
Determine what additional input parameters are required. Add these input parameters to the form in the app's forms.py. It is at this step that you can also extend the base DataSelectionForm to change parameters as wel, e.g. for coastal change, we override time start and time end to be years rather than dates.
454
452
455
-
```
453
+
```python
456
454
classAdditionalOptionsForm(forms.Form):
457
455
"""
458
456
Django form to be created for selecting information and validating input for:
@@ -485,15 +483,16 @@ In tasks.py, implement your algorithm by filling in all the `TODO` blocks. Add a
485
483
486
484
A few general tips/tricks:
487
485
488
-
* Only a single NetCDF storage unit is passed from task to task, along with a metadata dict and some general chunk details. If you have multiple data products, merge them into a single NetCDF before writing to disk.
489
-
* The main Celery processing pipeline may seem confusing, but due to some weird issues with how Celery interprets/unrolls groups and chords it can't be helped. If you want to add additional steps, do so with the '|' like the combination functions are added.
486
+
- Only a single NetCDF storage unit is passed from task to task, along with a metadata dict and some general chunk details. If you have multiple data products, merge them into a single NetCDF before writing to disk.
487
+
- The main Celery processing pipeline may seem confusing, but due to some weird issues with how Celery interprets/unrolls groups and chords it can't be helped. If you want to add additional steps, do so with the '|' like the combination functions are added.
488
+
489
+
## Template Changes
490
+
-----
490
491
491
-
<aname="templates"></a> Template Changes
492
-
=================
493
492
Template changes are the last step in the app development process. The base apps include only the common features - time start/end, execution times, geographic bounds. Make the following changes to add more data to your template.
494
493
495
-
* In all the templates (there should be four of them), add any additional query parameters (fields you added to Query in models.py)
496
-
* In output_list.html, modify the entries in the Data Guide section so that they are accurate to your output products. Additionally, ensure that the options cover all of the output products in the download_options block
497
-
* In results_list.html, in the task_table_rows block add checkbox inputs for any additional image outputs that should be displayed on the map. Make sure that the function in the functions_block handles this - removing old images, adding new, highlighting.
498
-
* In results_list.html in the meta_table_rows block, add any additional 'full task' metadata that exists on the task model.
499
-
* In results_list.html in the metadata_dl_block, ensure that the zipped fields corresponds with the fields enumerated on your Metadata model.
494
+
- In all the templates (there should be four of them), add any additional query parameters (fields you added to Query in models.py)
495
+
- In output_list.html, modify the entries in the Data Guide section so that they are accurate to your output products. Additionally, ensure that the options cover all of the output products in the download_options block
496
+
- In results_list.html, in the task_table_rows block add checkbox inputs for any additional image outputs that should be displayed on the map. Make sure that the function in the functions_block handles this - removing old images, adding new, highlighting.
497
+
- In results_list.html in the meta_table_rows block, add any additional 'full task' metadata that exists on the task model.
498
+
- In results_list.html in the metadata_dl_block, ensure that the zipped fields corresponds with the fields enumerated on your Metadata model.
0 commit comments