Please provide some suggestions.

[daveida]

My data for this project is CFB003_ortho.tif (Figure 1) from the link https://radar.kit.edu/radar/en/dataset/AwwREVscwqlcTVSw.FORTRESS. When I used scripts/inference.py in the project to perform predictions, two files were generated: intermediate_output_3.tif (Figure 2) and CFB003.sqlite (Figure 3). My parameter settings are as follows:
args = get_parser().parse_args("-i ./TreeCrownDelineation-master/data/dataset/orthomosaic/CFB003_ortho.tif "
"-o ./TreeCrownDelineation-master/data/output/output_3/CFB003.sqlite "
"-m ./TreeCrownDelineation-master/model/tcd-20cm-RGBI-v1.0/Unet-resnet18_epochs=209_lr=0.0001_width=224_bs=32_divby=255_custom_color_augs_k=0_jitted.pt "
"-w 1024 "
"--ndvi "
"--min-dist 200 "
"--div 255 "
"--rescale-ndvi "
"--a "
"--sigmoid "
"--save-prediction./TreeCrownDelineation-master/data/output/intermediate_output "
"-s 0.05 "
"-b 0.05 "
"-l 0.4 "
"--sigma 2".split())
Please review this prediction and see if it makes sense. Please provide some suggestions.

[maxfreu]

The problem is that the network was trained on 20 cm images, while the images you feed in have a resolution of 1.4 cm. Consequently, you have to downsample them. The script provides an option for upsampling, which can also be used for downsampling. Try --upsample 0.07 and report back.

[daveida]

Hi, I'm glad to hear from you. Following your suggestion, I added "--upsample 0.07" to the parameter settings. While keeping everything else the same, an error occurred while running the script.
The error is as follows:
The following operation failed in the TorchScript interpreter.
Traceback of TorchScript, serialized code (most recent call last):
File "code/torch/treecrowndelineation/model/tcd_model.py", line 12, in forward
dist_model = self.dist_model
seg_model = self.seg_model
_0 = (seg_model).forward(img, )
~~~~~~~~~~~~~~~~~~ <--- HERE
_1 = [_0, (dist_model).forward(_0, img, )]
return torch.cat(_1, 1)
File "code/torch/treecrowndelineation/model/segmentation_model.py", line 11, in forward
img: Tensor) -> Tensor:
model = self.model
return (model).forward(img, )
~~~~~~~~~~~~~~ <--- HERE
File "code/torch/segmentation_models_pytorch/unet/model.py", line 15, in forward
encoder = self.encoder
_0, _1, _2, _3, _4, = (encoder).forward(img, )
_5 = (decoder).forward(_0, _1, _2, _3, _4, )
~~~~~~~~~~~~~~~~ <--- HERE
return (segmentation_head).forward(_5, )
File "code/torch/segmentation_models_pytorch/unet/decoder.py", line 26, in forward
center = self.center
_5 = (center).forward()
_6 = (_0).forward(argument_1, argument_2, )
~~~~~~~~~~~ <--- HERE
_7 = (_2).forward((_1).forward(_6, argument_3, ), argument_4, )
_8 = (_4).forward((_3).forward(_7, argument_5, ), )
File "code/torch/segmentation_models_pytorch/unet/decoder.py", line 47, in forward
attention1 = self.attention1
x = torch.upsample_nearest2d(argument_1, None, [2., 2.])
input = torch.cat([x, argument_2], 1)
~~~~~~~~~ <--- HERE
_4 = (attention1).forward()
_5 = (conv2).forward((conv1).forward(input, ), )

Traceback of TorchScript, original code (most recent call last):
/home/max/.conda/envs/dr2/lib/python3.8/site-packages/segmentation_models_pytorch/unet/decoder.py(38): forward
/home/max/.conda/envs/dr2/lib/python3.8/site-packages/torch/nn/modules/module.py(1090): _slow_forward
/home/max/.conda/envs/dr2/lib/python3.8/site-packages/torch/nn/modules/module.py(1102): _call_impl
/home/max/.conda/envs/dr2/lib/python3.8/site-packages/segmentation_models_pytorch/unet/decoder.py(119): forward
/home/max/.conda/envs/dr2/lib/python3.8/site-packages/torch/nn/modules/module.py(1090): _slow_forward
/home/max/.conda/envs/dr2/lib/python3.8/site-packages/torch/nn/modules/module.py(1102): _call_impl
/home/max/.conda/envs/dr2/lib/python3.8/site-packages/segmentation_models_pytorch/base/model.py(16): forward
/home/max/.conda/envs/dr2/lib/python3.8/site-packages/torch/nn/modules/module.py(1090): _slow_forward
/home/max/.conda/envs/dr2/lib/python3.8/site-packages/torch/nn/modules/module.py(1102): _call_impl
/home/max/.conda/envs/dr2/lib/python3.8/site-packages/treecrowndelineation-0.1.0-py3.8.egg/treecrowndelineation/model/segmentation_model.py(51): forward
/home/max/.conda/envs/dr2/lib/python3.8/site-packages/torch/nn/modules/module.py(1090): _slow_forward
/home/max/.conda/envs/dr2/lib/python3.8/site-packages/torch/nn/modules/module.py(1102): _call_impl
/home/max/.conda/envs/dr2/lib/python3.8/site-packages/treecrowndelineation-0.1.0-py3.8.egg/treecrowndelineation/model/tcd_model.py(38): forward
/home/max/.conda/envs/dr2/lib/python3.8/site-packages/torch/nn/modules/module.py(1090): _slow_forward
/home/max/.conda/envs/dr2/lib/python3.8/site-packages/torch/nn/modules/module.py(1102): _call_impl
/home/max/.conda/envs/dr2/lib/python3.8/site-packages/torch/jit/_trace.py(958): trace_module
/home/max/.conda/envs/dr2/lib/python3.8/site-packages/torch/jit/_trace.py(741): trace
/home/max/.conda/envs/dr2/lib/python3.8/site-packages/pytorch_lightning/core/lightning.py(1778): to_torchscript
/home/max/.conda/envs/dr2/lib/python3.8/site-packages/torch/autograd/grad_mode.py(28): decorate_context
training_BKG_k_fold.py(160):
RuntimeError: Sizes of tensors must match except in dimension 1. Expected size 4 but got size 3 for tensor number 1 in the list.
File "/media/dell/c598f9db-1320-48e9-bc13-78eb2afb0931/dell/table/lkz/TreeCrownDelineation-master/scripts/inference_text.py", line 302, in
result = utils.predict_on_array_cf(model,
RuntimeError: The following operation failed in the TorchScript interpreter.
Traceback of TorchScript, serialized code (most recent call last):
File "code/torch/treecrowndelineation/model/tcd_model.py", line 12, in forward
dist_model = self.dist_model
seg_model = self.seg_model
_0 = (seg_model).forward(img, )
~~~~~~~~~~~~~~~~~~ <--- HERE
_1 = [_0, (dist_model).forward(_0, img, )]
return torch.cat(_1, 1)
File "code/torch/treecrowndelineation/model/segmentation_model.py", line 11, in forward
img: Tensor) -> Tensor:
model = self.model
return (model).forward(img, )
~~~~~~~~~~~~~~ <--- HERE
File "code/torch/segmentation_models_pytorch/unet/model.py", line 15, in forward
encoder = self.encoder
_0, _1, _2, _3, _4, = (encoder).forward(img, )
_5 = (decoder).forward(_0, _1, _2, _3, _4, )
~~~~~~~~~~~~~~~~ <--- HERE
return (segmentation_head).forward(_5, )
File "code/torch/segmentation_models_pytorch/unet/decoder.py", line 26, in forward
center = self.center
_5 = (center).forward()
_6 = (_0).forward(argument_1, argument_2, )
~~~~~~~~~~~ <--- HERE
_7 = (_2).forward((_1).forward(_6, argument_3, ), argument_4, )
_8 = (_4).forward((_3).forward(_7, argument_5, ), )
File "code/torch/segmentation_models_pytorch/unet/decoder.py", line 47, in forward
attention1 = self.attention1
x = torch.upsample_nearest2d(argument_1, None, [2., 2.])
input = torch.cat([x, argument_2], 1)
~~~~~~~~~ <--- HERE
_4 = (attention1).forward()
_5 = (conv2).forward((conv1).forward(input, ), )

Traceback of TorchScript, original code (most recent call last):
/home/max/.conda/envs/dr2/lib/python3.8/site-packages/segmentation_models_pytorch/unet/decoder.py(38): forward
/home/max/.conda/envs/dr2/lib/python3.8/site-packages/torch/nn/modules/module.py(1090): _slow_forward
/home/max/.conda/envs/dr2/lib/python3.8/site-packages/torch/nn/modules/module.py(1102): _call_impl
/home/max/.conda/envs/dr2/lib/python3.8/site-packages/segmentation_models_pytorch/unet/decoder.py(119): forward
/home/max/.conda/envs/dr2/lib/python3.8/site-packages/torch/nn/modules/module.py(1090): _slow_forward
/home/max/.conda/envs/dr2/lib/python3.8/site-packages/torch/nn/modules/module.py(1102): _call_impl
/home/max/.conda/envs/dr2/lib/python3.8/site-packages/segmentation_models_pytorch/base/model.py(16): forward
/home/max/.conda/envs/dr2/lib/python3.8/site-packages/torch/nn/modules/module.py(1090): _slow_forward
/home/max/.conda/envs/dr2/lib/python3.8/site-packages/torch/nn/modules/module.py(1102): _call_impl
/home/max/.conda/envs/dr2/lib/python3.8/site-packages/treecrowndelineation-0.1.0-py3.8.egg/treecrowndelineation/model/segmentation_model.py(51): forward
/home/max/.conda/envs/dr2/lib/python3.8/site-packages/torch/nn/modules/module.py(1090): _slow_forward
/home/max/.conda/envs/dr2/lib/python3.8/site-packages/torch/nn/modules/module.py(1102): _call_impl
/home/max/.conda/envs/dr2/lib/python3.8/site-packages/treecrowndelineation-0.1.0-py3.8.egg/treecrowndelineation/model/tcd_model.py(38): forward
/home/max/.conda/envs/dr2/lib/python3.8/site-packages/torch/nn/modules/module.py(1090): _slow_forward
/home/max/.conda/envs/dr2/lib/python3.8/site-packages/torch/nn/modules/module.py(1102): _call_impl
/home/max/.conda/envs/dr2/lib/python3.8/site-packages/torch/jit/_trace.py(958): trace_module
/home/max/.conda/envs/dr2/lib/python3.8/site-packages/torch/jit/_trace.py(741): trace
/home/max/.conda/envs/dr2/lib/python3.8/site-packages/pytorch_lightning/core/lightning.py(1778): to_torchscript
/home/max/.conda/envs/dr2/lib/python3.8/site-packages/torch/autograd/grad_mode.py(28): decorate_context
training_BKG_k_fold.py(160):
RuntimeError: Sizes of tensors must match except in dimension 1. Expected size 4 but got size 3 for tensor number 1 in the list.

Hope to receive your reply soon！

[maxfreu]

Try the most simple thing: Resample the image to 20cm with another program and run the inference on that.

[daveida]

Thank you for your answer. Figure 1 below shows the prediction result at 20cm resolution.
But I encountered problems during training again. Figure 2 shows the prediction results after 210 epochs of training. Below is part of my training code:
...
rasters = "./data/tile_rgb_rsp"
masks = "./data/masks"
outlines = "./data/outlines"
dist = "./data/dist_trafo"
logdir = "./data/log"
model_save_path = "./data/model"
experiment_name = "test_003"
arch = "Unet-resnet18"
width = 256
batchsize = 100
in_channels = 3
devices = 1
accelerator = "auto"
max_epochs = 210
lr = 3E-4
training_split = 0.8
...
train_augmentation = A.Compose([A.RandomCrop(width, width,p=1.0), A.RandomRotate90(),
A.VerticalFlip()
])
#val_augmentation = A.RandomCrop(width, width, always_apply=True)
val_augmentation = A.RandomCrop(width, width,p=1.0)
data = InMemoryDataModule(rasters,
(masks, outlines, dist),
width=width,
batchsize=batchsize,
training_split=training_split,
train_augmentation=train_augmentation,
val_augmentation=val_augmentation,
#concatenate_ndvi=True,

red=0,

nir=3,

dilate_second_target_band=2,
#rescale_ndvi=True
)
model = TreeCrownDelineationModel(in_channels=in_channels, lr=lr)
trainer = Trainer(devices=devices,
accelerator=accelerator,
logger=logger,
callbacks=callbacks,

checkpoint_callback=False, # set this to avoid logging into the working directory

max_epochs=max_epochs,
log_every_n_steps=20,
precision=32)
trainer.fit(model, data)

The parameters used for prediction are as follows:

Train model

args = get_parser().parse_args(
"-i ./data/tile_rgb_rsp/tile_rgb_rsp_1.tif "
"-o ./data/output/output_3/tile_rgb_rsp_1.sqlite "
"-m ./data/model/test_003/Unet-resnet18_epochs=210_lr=0.0003_width=256_bs=100_jitted.pt

"-w 512"
#"--ndvi"
"--min-dist 20"
"--div 255"
#"--rescale-ndvi"
"--a"
"--sigmoid"
"--save-prediction ./data/output/intermediate_1"
"-s 0.05"
"-b 0.05"
"-l 0.4"
"--sigma 2".split())

I used this code to convert all datasets to only RGB bands with a resolution of 5cm, which also corresponds to the shp, mask, outline, and dist_trafo files.
Why do the prediction results appear like a chessboard? How can I fix this?
I hope to hear from you soon.

[maxfreu]

What's the resolution of your input images at 5cm? Can you post screenshots of the training input (mask, outlines, dist) for one image? Can you post a screenshot of your loss curves?

Also, I recommend that you format code in your posts using triple backticks ``` before and after. That will make things more readable.

```python
<code here>
```

[daveida]

I resampled the original dataset to 5cm resolution for training. The following pictures are the mask, outline, and dist_trafo matching of tile_rgb_rsp_1.tif

If you need more information, please let me know.

[daveida]

I found that if I comment out "--div 255" in the prediction code parameter and then predict again, I can get the correct result.

   args = get_parser().parse_args(
"-i ./data/tile_rgb_rsp/tile_rgb_rsp_3.tif "
"-o ./data/output/output_3/tile_rgb_rsp_3.sqlite "
"-m ./data/model/test_003/Unet-resnet18_epochs=210_lr=0.0003_width=256_bs=100_jitted.pt "
                                   "-w 512 "
                                   #"--ndvi "
                                   "--min-dist 20 "
                                    #    "--div 255 "
                                   #"--rescale-ndvi "
                                   "--a "
                                   "--sigmoid "
                                   "--save-prediction ./data/output/intermediate_1 "
                                #    "-s 0.05 " 
                                #    "-b 0.05 "
                                #    "-l 0.4 "
                                   "--sigma 3".split())

This is my prediction result after 210 rounds of training

[maxfreu]

Ah nice! Not perfect, but ok. You could play around with the minimum distance and threshold parameters a bit and see how that changes the results.

I found that if I comment out "--div 255" in the prediction code parameter and then predict again, I can get the correct result.

Yes, it depends whether you trained on the data range 0-255 or 0-1. If that does not match what you input during inference you have no chance.

[daveida]

Hello, I'm encountering some issues, such as missed tree crown segmentation, unevenly circled tree crowns, and incomplete circles.
Figure 1 shows the predicted tif file, but I don't understand its purpose. The circled image in Figure 1 looks quite good, but when converted to the SQLite file in Figure 2, the result isn't as good. Why is this?