You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The vanilla architecture as in the following figure is implemented. Convolutions are run along spatial dimensions of input tensor, which is supposed to have `[M, H, W, D]` shape, where M is the mini-batch size, and H, W and D are the height, width and number of bands (i.e. depth) of the input image tensor. The 2d convolutions perform a `VALID` convolution, therefore the output tensor size is smaller than the input size.
6
+
7
+
![FCN](./figures/fcn-architecture.png"FCN")
8
+
9
+
An example training script is provided. To run it execute the `configs/fcn_example.json` configuration:
10
+
```
11
+
python -m eoflow.execute configs/fcn_example.json
12
+
```
13
+
14
+
The example configuration can be used as a base to run your own experiments.
15
+
16
+
## Temporal Fully-Convolutional-Network (TFCN)
17
+
18
+
Similarly to the RFCN, the TFCN works with time-series of input shape `[M, T, H, W, D]`. This network performs 3d convolutions along the tempo-spatial dimensions, i.e. the convolutional kernels are 3d `k x k x k`. As default, the temporal dimension is not pooled. For temporal pooling, enough time-frames need to be available in the input tensors. At the bottom of the TFCN and along the skip connections, a 1d convolution along the temporal dimension is performed to linearly combine the temporal features. The resulting tensors are 4d of shape `[M, H, W, D]`. The decoding path is as in FCN.
19
+
20
+
![TFCN](./figures/tfcn-architecture.png"TFCN")
21
+
22
+
An example training script is provided. To run it execute the `configs/tfcn_example.json` configuration:
The example configuration can be used as a base to run your own experiments.
28
+
29
+
## Recurrent Fully-Convolutional-Network (RFCN)
30
+
31
+
A recurrent version of the **FCN** is implemented as in below figure. The input tensor in this case is 5d with shape `[M, T, H, W, D]`, where `T` is the number of temporal acquisitions. As for the FCN, the 2d convolutions operate along the `H` and `W` dimensions. The recurrent layers are applied along the skip connections and the bottom layers to model the temporal relationship between the features extracted by the 2d convolutions. The output of the recurrent layers is a 4d tensor of shape `[M, H, W, D]` (the height, width and depth of the tensors will vary along the network). The decoding path is as in **FCN**. The 2d convolutions perform a `VALID` convolution, therefore the output tensor size is smaller than the input size.
32
+
33
+
![RFCN](./figures/rfcn-architecture.png"RFCN")
34
+
35
+
An example training script is provided. To run it execute the `configs/rfcn_example.json` configuration:
This repository provides a templated structure to generate TensorFlow projects. Using the same base structure for all TF projects should benefit model creation, debugging and experiments reproducibility.
3
+
This repository provides code and examples for creation of Earth Observation (EO) projects using TensorFlow. The code uses TensorFlow 2.0 with Keras as the main model building API.
4
4
5
-
The project contains the package `eoflow` which contains the base abstract classes and implements the common models, tasks and input methods. Custom models tasks and input methods can also be implemented building on top of the provided abstract classes.
5
+
Common model architectures, layers, and input methods for EO tasks are provided in the package `eoflow`. Custom models and input methods can also be implemented building on top of the provided abstract classes. This package aims at seamlessly integrate with [`eo-learn`](https://github.com/sentinel-hub/eo-learn), and favours both creation of models for prototypying as well as production of EO applications.
6
+
7
+
Architectures and examples for land cover and crop classification using time-series derived from satellite images are provided.
8
+
9
+
## Installation
6
10
7
11
The package can be installed by running the following command.
8
12
```
@@ -14,92 +18,53 @@ You can also install the package from source. Clone the repository and run the f
14
18
$ pip install .
15
19
```
16
20
17
-
The project also contains example configurations to run common tasks on implemented models. Examples of implementing and running custom models and input methods are also provided.
18
-
19
-
## Package structure
20
-
21
-
The structure is inspired by the tensorflow [Estimator API](https://www.tensorflow.org/guide/custom_estimators).
22
-
23
-
The subpackages of `eoflow` are as follows:
24
-
*`base`: this directory contains the abstract classes to build models, inputs and tasks. Any useful abstract class should go in this folder.
25
-
*`models`: classes implementing the TF models (e.g. Fully-Convolutional-Network, GANs, seq2seq, ...). These classes inherit and implement the `BaseModel` abstract class.
26
-
*`tasks`: classes handling the actions that can be applied to each TF model. These actions may include training, inference, exporting the model, validation, etc. The tasks inherit the `BaseTask` abstract class. Currently only the training task is implemented in `TrainTask` class.
27
-
*`input`: these classes handle the loading of the input data into a tf Dataset. These classes may be specific to the problem and data at hand, but can also contain common classes for reading certain type of data (EOPatch, np arrays, etc.). Currently only random input data generation is implemented.
28
-
*`utils`: collection of utility functions
29
-
30
-
## Examples and scripts
31
-
32
-
Project also contains other folders:
33
-
*`configs`: folder containing example configurations for different models. Config parameters are stored in .json files. Results of an experiment should be reproducible by re-running the same config file. Config files specify the whole workflow (model, task, data input if required).
34
-
*`examples`: folder containing example implementations of custom models and input functions. Also contains a jupyter notebook example.
35
-
36
-
## Currently implemented projects
37
-
38
-
The projects currently implemented are:
39
-
* Fully-Convolutional-Network (FCN, a.k.a. U-net), vanilla implementation of method described in this [paper](https://arxiv.org/abs/1505.04597). This network expects 2D MSI images as inputs and predicts 2D label maps as output.
40
-
* Recurrent FCN, where a time series is used as input and the temporal dependency between images is modelled by recurrent convolutions. The output of the network is a 2D label map as in previous case.
41
-
* Temporal FCN, where the whole time-series is considered as a 3D MSI volume and convolutions are performed along the temporal dimension as well spatial dimension. The output of the network is a 2D label map as in previous cases.
42
-
43
-
### Fully-Convolutional-Network (FCN)
44
-
45
-
The vanilla architecture as in the following figure is implemented. Convolutions are run along spatial dimensions of input tensor, which is supposed to have `[M, H, W, D]` shape, where M is the mini-batch size, and H, W and D are the height, width and number of bands (i.e. depth) of the input image tensor. The 2d convolutions perform a `VALID` convolution, therefore the output tensor size is smaller than the input size.
46
-
47
-
![FCN](./figures/fcn.png"FCN")
48
-
49
-
An example training script is provided. To run it execute the `configs/fcn_example.json` configuration:
50
-
```
51
-
python -m eoflow.execute configs/fcn_example.json
52
-
```
53
-
54
-
The example configuration can be used as a base to run your own experiments.
55
-
56
-
### Recurrent Fully-Convolutional-Network (RFCN)
21
+
## Getting started
57
22
58
-
A recurrent version of the **FCN** is implemented as in below figure. The input tensor in this case is 5d with shape `[M, T, H, W, D]`, where `T` is the number of temporal acquisitions. As for the FCN, the 2d convolutions operate along the `H`and `W` dimensions. The recurrent layers are applied along the skip connections and the bottom layers to model the temporal relationship between the features extracted by the 2d convolutions. The output of the recurrent layers is a 4d tensor of shape `[M, H, W, D]` (the height, width and depth of the tensors will vary along the network). The decoding path is as in **FCN**. The 2d convolutions perform a `VALID` convolution, therefore the output tensor size is smaller than the input size.
23
+
The `eoflow` package can be used in two ways. For best control over the workflow and faster prototyping, the package can be used programmatically (in code). The [example notebook](examples/notebook.ipynb) should help you get started with that. It demonstrates how to prepare a dataset pipeline, train the model, evaluate the model and make predictions using the trained model.
59
24
60
-
![RFCN](./figures/rfcn.png"RFCN")
25
+
An alternate way of using `eoflow` is by writing configuration `json` files and running them using `eoflow`'s execute script. Configuration files specify and configure the task (training, evaluation, etc.) and contain the configurations of the model and input methods. Example configurations are provided in the `configs` directory. Once a configuration file is created it can be run using the execute command.
61
26
62
-
An example training script is provided. To run it execute the `configs/rfcn_example.json` configuration:
27
+
A simple example can be run using the following command. More advanced configurations are also provided.
The example configuration can be used as a base to run your own experiments.
68
-
69
-
### Temporal Fully-Convolutional-Network (TFCN)
70
-
71
-
Similarly to the RFCN, the TFCN works with time-series of input shape `[M, T, H, W, D]`. This network performs 3d convolutions along the tempo-spatial dimensions, i.e. the convolutional kernels are 3d `k x k x k`. As default, the temporal dimension is not pooled. For temporal pooling, enough time-frames need to be available in the input tensors. At the bottom of the TFCN and along the skip connections, a 1d convolution along the temporal dimension is performed to linearly combine the temporal features. The resulting tensors are 4d of shape `[M, H, W, D]`. The decoding path is as in FCN.
72
-
73
-
![TFCN](./figures/tfcn.png"TFCN")
32
+
This will create an output folder `temp/experiment` containing the tensorboard logs and model checkpoints.
74
33
75
-
An example training script is provided. To run it execute the `configs/tfcn_example.json` configuration:
The example configuration can be used as a base to run your own experiments.
39
+
## Writing custom code
81
40
82
-
## Custom models and input functions
41
+
To get started with writing custom models and input methods for `eoflow` take a look at the example implementations ([`examples` folder](examples/)). Custom classes use schemas to define the configuration parameters in order to work with the execute script and configuration files. Since eoflow builds on top of TF2 and Keras, model building is very similar.
83
42
84
-
In order to create your own model, create a new class that inherits from `BaseModel` or any of its subclasses. The model must specify a schema for its configuration and a function `build_model` that builds the model with the provided input and label tensors. Look at the example implementation for more details.
43
+
## Package structure
85
44
86
-
The package `eoflow` contains a few of the most common input methods. It also provides common dataset building blocks: EOPatch loading, subpatch extraction, data augmentation. You can use these functions to create a custom input method that fits you data and model. To implement a custom input method create a new class that inherits from `BaseInput` or any of its subclasses. The input method must specify a schema for its configuration and a function `get_dataset` that builds a tensorflow Dataset that reads the input data. For further details look at the example implementation.
45
+
The subpackages of `eoflow` are as follows:
46
+
*`base`: this directory contains the abstract classes to build models, inputs and tasks. Any useful abstract class should go in this folder.
47
+
*`models`: classes implementing the TF models (e.g. Fully-Convolutional-Network, GANs, seq2seq, ...). These classes inherit and implement the `BaseModel` abstract class. The module also contains custom losses, metrics and layers.
48
+
*`tasks`: classes handling the configurable actions that can be applied to each TF model, when using the execute script. These actions may include training, inference, exporting the model, validation, etc. The tasks inherit the `BaseTask` abstract class.
49
+
*`input`: building blocks and helper methods for loading the input data (EOPatch, numpy arrays, etc.) into a tensoflow Dataset and applying different transformations (data augmentation, patch extraction)
50
+
*`utils`: collection of utility functions
87
51
88
-
Implementations in files `examples/models.py`and `examples/input.py` are available as guidelines on how the configurable classes for **models** and **input** should be implemented.
52
+
### Examples and scripts
89
53
90
-
A toy example using the example model and input is configured in the`configs/example.json` configuration file.
54
+
Project also contains other folders:
55
+
*`configs`: folder containing example configurations for different models. Config parameters are stored in .json files. Results of an experiment should be reproducible by re-running the same config file. Config files specify the whole workflow (model, task, data input if required).
56
+
*`examples`: folder containing example implementations of custom models and input functions. Also contains a jupyter notebook example.
91
57
92
-
To run the example, run
93
-
```
94
-
$ python -m eoflow.execute configs/example.json
95
-
```
96
-
This will create an output folder `temp/experiment` containing the tensorboard logs and model checkpoints.
58
+
## Implemented architectures
97
59
98
-
To visualise the logs in TensorBoard, run
99
-
```
100
-
$ tensorboard --logdir=temp/experiment
101
-
```
60
+
Segmentation models for land cover semantic segmentation:
61
+
***Fully-Convolutional-Network (FCN, a.k.a. U-net)**, vanilla implementation of method described in this [paper](https://arxiv.org/abs/1505.04597). This network expects 2D MSI images as inputs and predicts 2D label maps as output.
62
+
***Temporal FCN**, where the whole time-series is considered as a 3D MSI volume and convolutions are performed along the temporal dimension as well spatial dimension. The output of the network is a 2D label map as in previous cases. More details can be found in this [paper](https://www.researchgate.net/publication/333262625_Spatio-Temporal_Deep_Learning_An_Application_to_Land_Cover_Classification).
102
63
103
-
## Programatic use
64
+
Classification models for crop classification using time-series:
65
+
***TCN**: Implementation of the TCN network taken from the [keras-TCN implementation by Philippe Remy](https://github.com/philipperemy/keras-tcn).
66
+
***TempCNN**: Implementation of the TempCNN network taken from the [temporalCNN implementation of Charlotte Pelletier](https://github.com/charlotte-pel/temporalCNN).
67
+
***Recurrent NN**: Implementation of (bidirectional) Recurrent Neural Networks for the classification of time-series. Implementation allows to use either `SimpleRNN`, `GRU` or `LSTM` layers as building blocks of the architecture.
68
+
***TransformerEncoder**: Implementation of a time-series classification architecture based on [self-attention](https://arxiv.org/abs/1706.03762) layers. This implementation follows [this PyTorch implementation of Marc Russwurm](https://github.com/MarcCoru/crop-type-mapping).
104
69
105
-
Notebook `examples/notebook.ipnyb` shows how `eoflow` can be also used in code. The notebook shows model building, defining the input datasets, training, evaluation and prediction.
70
+
Descriptions and examples of semantic segmentation architectures are available [here](MODELS.md).
0 commit comments