Skip to content

Commit ef50e91

Browse files
committed
Merge branch 'master' into ceos_master
2 parents dbeae9d + 74c6350 commit ef50e91

File tree

8 files changed

+246
-271
lines changed

8 files changed

+246
-271
lines changed

.gitlab-ci.yml

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
# image:
2+
# name: busybox:latest
3+
4+
stages:
5+
- build
6+
7+
build:
8+
stage: build
9+
script:
10+
- echo hi

Makefile

Lines changed: 12 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,8 @@
11
SHELL:=/bin/bash
2-
docker_compose_dev = docker-compose --project-directory build/docker/dev -f build/docker/dev/docker-compose.yml
2+
# Set the project name to the path - making underscore the path separator.
3+
# Remove the leading slash and use lowercase since docker-compose will.
4+
project_name=$(shell PWD_var=$$(pwd); PWD_no_lead_slash=$${PWD_var:1}; echo $${PWD_no_lead_slash//\//_} | awk '{print tolower($$0)}' | cat)
5+
docker_compose_dev = docker-compose --project-directory build/docker/dev -f build/docker/dev/docker-compose.yml -p $(project_name)
36

47
# The `export` here is to allow commands (notably `docker-compose`)
58
# in the Make targets to use them.
@@ -30,6 +33,9 @@ base-build:
3033
base-run:
3134
docker run -it ${UI_BASE_IMG} bash
3235

36+
base-pull:
37+
docker pull ${UI_BASE_IMG}
38+
3339
base-push:
3440
docker push ${UI_BASE_IMG}
3541
## End Base ##
@@ -72,6 +78,11 @@ dev-clear:
7278
$(docker_compose_dev) stop
7379
$(docker_compose_dev) rm -fs
7480

81+
dev-pull-no-rcv:
82+
docker pull ${DEV_OUT_IMG}
83+
84+
dev-pull: base-pull dev-pull-no-rcv
85+
7586
dev-push:
7687
docker push ${DEV_OUT_IMG}
7788
## End Development ##

README.md

Lines changed: 5 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -25,15 +25,14 @@ Currently supported applications include:
2525
* Urbanization (NDBI-NDVI-NDWI false-color composites)
2626
* Water detection (using the [Water Observations from Space](https://www.ga.gov.au/scientific-topics/community-safety/flood/wofs) algorithm)
2727

28-
## Installation
28+
## Guide
2929
-------
3030

31-
First follow the instructions in the [Docker Installation Guide](docs/docker_install.md) if you do not have Docker installed yet.
31+
First follow the instructions in the [Environment Setup Guide](https://ceos-odc.readthedocs.io/en/latest/modules/install_docs/environment_setup.html) if you have not yet.
3232

33-
Follow the instructions in the
34-
[Open Data Cube Database Installation Guide](docs/odc_db_setup.md) to setup the Open Data Cube (ODC) database.
33+
Follow the instructions in the [Open Data Cube Database Installation Guide](https://ceos-odc.readthedocs.io/en/latest/modules/install_docs/database_install.html) to setup the Open Data Cube (ODC) database.
3534

36-
Follow the instructions in the [Open Data Cube UI Installation Guide](docs/ui_install.md) to install the ODC UI. That document also contains troubleshooting information for the UI in the form of an FAQ at the end.
35+
Follow the instructions in the [Open Data Cube UI Guide](docs/ui_guide.md) to install the ODC UI. That document also contains troubleshooting information for the UI in the form of an FAQ at the end.
3736

3837
Follow the instructions in the [Open Data Cube UI Algorithm Addition Guide](docs/adding_new_pages.md) to add new applications to the ODC UI. This guide is only intended for programmers.
3938

@@ -47,4 +46,4 @@ If you encounter issues with Open Data Cube or this UI that are not documented i
4746
## More
4847
-------
4948

50-
You may also consider running a Jupyter Notebook server that uses the Open Data Cube. The CEOS ODC Notebooks repository is [here](https://github.com/ceos-seo/data_cube_notebooks).
49+
You may also consider running a Jupyter Notebook server that uses the Open Data Cube. The CEOS ODC Notebooks repository is [here](https://github.com/ceos-seo/data_cube_notebooks).

docs/adding_new_pages.md

Lines changed: 67 additions & 68 deletions
Original file line numberDiff line numberDiff line change
@@ -1,37 +1,33 @@
1-
Open Data Cube UI Algorithm Addition Guide
2-
=================
1+
# Open Data Cube UI Algorithm Addition Guide
32

43
This document will guide users through the process of adding new algorithms or analysis cases to the UI, including the general Django files and the Celery workflow. This guide assumes that the user has a working Data Cube installation and has completed the full UI installation guide.
54

6-
Contents
7-
=================
5+
- [Introduction](#introduction)
6+
- [Basic Components](#basic-components)
7+
- [Base Classes](#base-classes)
8+
- [Generating Base Files](#generating-base-files)
9+
- [Basic Applications](#basic-applications)
10+
- [Complex Applications](#complex-applications)
11+
- [Template Changes](#template-changes)
812

9-
* [Introduction](#introduction)
10-
* [Basic Components](#basic_components)
11-
* [Base Classes](#base_classes)
12-
* [Generating Base Files](#base_files)
13-
* [Basic Applications](#band_math_app)
14-
* [Complex Applications](#complex_app)
15-
* [Template Changes](#templates)
16-
<!-- * [Common problems/FAQs](#faqs) -->
13+
## Introduction
14+
-----
1715

18-
<a name="introduction"></a> Introduction
19-
=================
2016
The Data Cube UI is designed to be quickly and easily extended to integrate new algorithms that operate on Data Cube based raster datasets. A new algorithm implementation includes three main parts:
2117

22-
* Django views - map URLs to rendered HTML pages
23-
* Django models - Hold the main task attributes (parameters, results, etc)
24-
* Celery workflows - Process asynchronous tasks, populating a Django model with updates and results
18+
- Django views - map URLs to rendered HTML pages
19+
- Django models - Hold the main task attributes (parameters, results, etc)
20+
- Celery workflows - Process asynchronous tasks, populating a Django model with updates and results
2521

2622
Two manage.py commands are provided to expedite this processing, requiring only simple and defined changes to get an application working.
2723

28-
<a name="basic_components"></a> Basic Components
29-
=================
24+
## Basic Components
25+
-----
26+
3027
The three main components of a new algorithm implementation were outlined above - the views, models, and celery tasks.
3128

32-
Django Views
33-
--------
34-
```
29+
>### Django Views
30+
```python
3531
class SubmitNewRequest(SubmitNewRequest):
3632
"""
3733
Submit new request REST API Endpoint
@@ -54,9 +50,8 @@ class SubmitNewRequest(SubmitNewRequest):
5450

5551
Each view class subclasses the base dc_algorithm class so only minor changes need to be made for a new page. Functions and variables that need to be provided are defined in the base class docstring, and if they are omitted then an exception is raised. HTML content that is rendered in the browser is controlled here - forms, panels, etc.
5652

57-
Django models
58-
------
59-
```
53+
>### Django Models
54+
```python
6055
class Query(BaseQuery):
6156
"""
6257
@@ -154,11 +149,10 @@ class Query(BaseQuery):
154149

155150
Django models contain all of the information and parameters required to perform your operation and create output products. This includes where to save the output products, how to split data for processing, and the Python function that should be used to create the output products. All input and output parameters should be enumerated here.
156151

157-
Celery tasks
158-
------
152+
>### Celery tasks
159153
Celery tasks perform all processing for the algorithms. Tasks use information found on the models to separate data chunks into manageable sizes, perform analysis functions, and create output products.
160154

161-
```
155+
```python
162156
@task(name="coastal_change.perform_task_chunking")
163157
def perform_task_chunking(parameters, task_id):
164158
"""Chunk parameter sets into more manageable sizes
@@ -253,23 +247,24 @@ The Celery tasks open models by their id and parse out the required information.
253247

254248
The processing pipeline in tasks.py is organized so that each task operates independently of one another - the process is fully asynchronous and non-blocking. The tasks are also arranged so that you work down the page as the task executes, and each task completes a single function.
255249

256-
* Parameters are parsed out from the task model
257-
* Parameters are verified and validated
258-
* Parameters are chunked into smaller, more manageable pieces for parallelized processing
259-
* A processing pipeline is created from the parameter chunks and submitted for processing - this involves both geographic and time based chunks
260-
* Each chunk is processed, with the results being saved to disk. Metadata is collected here and passed forward
261-
* The chunks are combined both geographically and over time, combining metadata as well
262-
* The output products are generated and the model is updated
250+
- Parameters are parsed out from the task model
251+
- Parameters are verified and validated
252+
- Parameters are chunked into smaller, more manageable pieces for parallelized processing
253+
- A processing pipeline is created from the parameter chunks and submitted for processing - this involves both geographic and time based chunks
254+
- Each chunk is processed, with the results being saved to disk. Metadata is collected here and passed forward
255+
- The chunks are combined both geographically and over time, combining metadata as well
256+
- The output products are generated and the model is updated
257+
258+
## Base Classes
259+
-----
263260

264-
<a name="base_classes"></a> Base Classes
265-
=================
266261
Django models and views allow for standard Python inheritance, allowing us to simply subclass a common dc_algorithm base to quickly create new apps. Take some time to look through the files in `apps/dc_algorithm` - the docstrings explain exactly what each view and model does and what attributes are required.
267262

268263
The main portion of the dc_algorithm base app lies within `views.py` and `models/abstract_base_models.py`.
269264

270-
Below is the base class for task submission. You'll notice the extensive docstrings outlining all required attributes and parameters. Additionally, there are 'getter' functions that raise a NotImplementedError when a required attribute is not present. Compare this class to the implementation in <a name="basic_components">Basic Components</a> - Only the required attributes are defined, and everything else lies within the base class.
265+
Below is the base class for task submission. You'll notice the extensive docstrings outlining all required attributes and parameters. Additionally, there are 'getter' functions that raise a NotImplementedError when a required attribute is not present. Compare this class to the implementation in [Basic Components](#basic-components) - Only the required attributes are defined, and everything else lies within the base class.
271266

272-
```
267+
```python
273268
class SubmitNewRequest(View, ToolClass):
274269
"""Submit a new request for processing using a task created with form data
275270
@@ -365,41 +360,43 @@ class SubmitNewRequest(View, ToolClass):
365360

366361
Model base classes work in the same way - common attributes are defined, and users are free to add or remove fields in child classes.
367362

368-
<a name="base_files"></a> Generating Base Files
369-
=================
363+
## Generating Base Files
364+
-----
365+
370366
Two manage.py commands are provided to easily create appliations - one for simple algorithms (band math, any application that should be run on a mosaic) and a more flexible base used for more complex concepts.
371367

372-
```
368+
```bash
373369
python manage.py start_bandmath_app app_name
374370
python manage.py start_dc_algorithm_app app_name
375371
```
376372

377373
These commands do a few things:
378374

379-
* Copy the associated base files to apps/app_name
380-
* Rename all of the templated values for app_name in all of the Python files, models, views, forms, tasks, etc.
381-
* Create a dc_algorithm.models.Application model for your new app
382-
* Provides instructions on next steps to get your application working
375+
- Copy the associated base files to apps/app_name
376+
- Rename all of the templated values for app_name in all of the Python files, models, views, forms, tasks, etc.
377+
- Create a dc_algorithm.models.Application model for your new app
378+
- Provides instructions on next steps to get your application working
383379

384380
The instructions include to run the migrations and to initialize the database with your new models. These base files contain a few `TODO:` statements within the files, marking where inputs are required. To integrate a new algorithm, generate an app using the command above, grep/search for instance of `TODO:` and follow the instructions there.
385381

386382
Apps come in two main classes - Band math-like apps and more complex apps. Band math apps perform compositing over the selected time and geographic area then perform some function on the resulting mosaic. The dc_algorithm app is set up to be more general and will require more input. As a general rule, if an algorithm doesn't have a time series component, you can use the band math app.
387383

388384
For example, fractional cover involves creating a mosaic and running a computationally intensive function on the resulting data, so the band math base app was used. Coastal change involves non standard time chunking, animation generation, etc. so the more generalized app was used.
389385

390-
<a name="band_math_app"></a> Basic Applications
391-
=================
386+
## Basic Applications
387+
-----
388+
392389
For a basic application, there are very few changes required to have a functional app. After generating the base files, run the migrations commands as seen below.
393390

394-
```
391+
```bash
395392
python manage.py makemigrations
396393
python manage.py migrate
397394
```
398395

399396
Now you will have all of the base tables generated in the database and are ready to implement your algorithm. If you search for `TODO` in that directory, there will only be one occurrence in `tasks.py` and one in models.
400397

401-
tasks.py
402-
```
398+
`tasks.py`
399+
```python
403400
def _apply_band_math(dataset):
404401
#TODO: apply your band math here!
405402
return (dataset.nir - dataset.red) / (dataset.nir + dataset.red)
@@ -410,7 +407,7 @@ tasks.py
410407

411408
Replace the expression above with your band math-like algorithm. If it is somewhat more complicated, like fractional cover, the snippet will look like the snippet below:
412409

413-
```
410+
```python
414411
def _apply_band_math(dataset):
415412
clear_mask = create_cfmask_clean_mask(dataset.cf_mask) if 'cf_mask' in dataset else create_bit_mask(
416413
dataset.pixel_qa, [1, 2])
@@ -426,7 +423,7 @@ Replace the expression above with your band math-like algorithm. If it is somewh
426423

427424
Now that you've implemented your algorithm, you'll need to handle the output. The base application will produce a single true color mosaic and the result of your band math. To do this, a color scale needs to be provided. The default color scale is a simple red->green scale for 0%->100% - to replace this, create a [GDALDEM compatible color scale](http://www.gdal.org/gdaldem.html#gdaldem_color_relief) and name it the same as your app (app_name) and place it in utils/color_scales.
428425

429-
```
426+
```bash
430427
-0.40 172 21 14
431428
-0.30 247 103 50
432429
-0.20 249 133 20
@@ -442,17 +439,18 @@ nan 0 0 0
442439

443440
Now that this is all complete, you can see your working application by:
444441

445-
* Restarting Apache2
446-
* Restarting Celery workers
447-
* Go to your site and select your application under Tools, choose your area, and submit a task.
442+
- Restarting Apache2
443+
- Restarting Celery workers
444+
- Go to your site and select your application under Tools, choose your area, and submit a task.
445+
446+
## Complex Applications
447+
-----
448448

449-
<a name="complex_app"></a> Complex Applications
450-
=================
451449
The process for implementing an advanced application is similar to the band math app. Generate the base application using the manage.py command, but don't run the migrations yet. You can start by Grepping/Searching for all instance of `TODO` and filling in the information that you're able to.
452450

453451
Determine what additional input parameters are required. Add these input parameters to the form in the app's forms.py. It is at this step that you can also extend the base DataSelectionForm to change parameters as wel, e.g. for coastal change, we override time start and time end to be years rather than dates.
454452

455-
```
453+
```python
456454
class AdditionalOptionsForm(forms.Form):
457455
"""
458456
Django form to be created for selecting information and validating input for:
@@ -485,15 +483,16 @@ In tasks.py, implement your algorithm by filling in all the `TODO` blocks. Add a
485483

486484
A few general tips/tricks:
487485

488-
* Only a single NetCDF storage unit is passed from task to task, along with a metadata dict and some general chunk details. If you have multiple data products, merge them into a single NetCDF before writing to disk.
489-
* The main Celery processing pipeline may seem confusing, but due to some weird issues with how Celery interprets/unrolls groups and chords it can't be helped. If you want to add additional steps, do so with the '|' like the combination functions are added.
486+
- Only a single NetCDF storage unit is passed from task to task, along with a metadata dict and some general chunk details. If you have multiple data products, merge them into a single NetCDF before writing to disk.
487+
- The main Celery processing pipeline may seem confusing, but due to some weird issues with how Celery interprets/unrolls groups and chords it can't be helped. If you want to add additional steps, do so with the '|' like the combination functions are added.
488+
489+
## Template Changes
490+
-----
490491

491-
<a name="templates"></a> Template Changes
492-
=================
493492
Template changes are the last step in the app development process. The base apps include only the common features - time start/end, execution times, geographic bounds. Make the following changes to add more data to your template.
494493

495-
* In all the templates (there should be four of them), add any additional query parameters (fields you added to Query in models.py)
496-
* In output_list.html, modify the entries in the Data Guide section so that they are accurate to your output products. Additionally, ensure that the options cover all of the output products in the download_options block
497-
* In results_list.html, in the task_table_rows block add checkbox inputs for any additional image outputs that should be displayed on the map. Make sure that the function in the functions_block handles this - removing old images, adding new, highlighting.
498-
* In results_list.html in the meta_table_rows block, add any additional 'full task' metadata that exists on the task model.
499-
* In results_list.html in the metadata_dl_block, ensure that the zipped fields corresponds with the fields enumerated on your Metadata model.
494+
- In all the templates (there should be four of them), add any additional query parameters (fields you added to Query in models.py)
495+
- In output_list.html, modify the entries in the Data Guide section so that they are accurate to your output products. Additionally, ensure that the options cover all of the output products in the download_options block
496+
- In results_list.html, in the task_table_rows block add checkbox inputs for any additional image outputs that should be displayed on the map. Make sure that the function in the functions_block handles this - removing old images, adding new, highlighting.
497+
- In results_list.html in the meta_table_rows block, add any additional 'full task' metadata that exists on the task model.
498+
- In results_list.html in the metadata_dl_block, ensure that the zipped fields corresponds with the fields enumerated on your Metadata model.

docs/docker_install.md

Lines changed: 0 additions & 22 deletions
This file was deleted.

0 commit comments

Comments
 (0)