[WIP] Added support for temporal segmentation data in encoder decoder factory #355
Conversation
|
Hi, @rijuld Maybe you would prefer to convert your PR to a draft and add the prefix [WIP] (work in progress) while work on it. |
rijuld
changed the title
@Joao-L-S-Almeida Thanks, Done. |
|
Thank you for your contribution. |
|
|
||
| def forward(self, x): | ||
| # x is a list of tensors, each corresponding to a different timestamp | ||
| features = [self.encoder(t) for t in x] |
There was a problem hiding this comment.
I like your approach, thanks for starting this draft!
Other models in terratorch process data in the format [B, C, T, H, W]. That is also the format how data is provided from the generic data modules. It might be good to follow this this pattern und iterate over dim=2 instead of expecting a list.
| self.pooling = torch.mean | ||
| elif pooling == "max": | ||
| self.pooling = torch.max | ||
| else: |
There was a problem hiding this comment.
It would be good to have a method concat which merges the embeddings of all timestamps along the embedding dim. E.g. for testing how much accuracy we loose if the time stamps are averaged before the decoder.
There was a problem hiding this comment.
This requires fix time stamps defined by the user so that the decoder gets the correct out_channels.
| @@ -136,6 +164,10 @@ def build_model( | |||
| decoder, channel_list, decoder_kwargs, head_kwargs, num_classes=num_classes | |||
| ) | |||
|
|
|||
| # Add temporal wrapper if enabled | |||
| if use_temporal: | |||
| backbone = TemporalWrapper(backbone, pooling=temporal_pooling) | |||
There was a problem hiding this comment.
I would apply the wrapper when building the backbone. I.e. backbone_use_temporal is passed as use_temporal to _get_backbone. The only important thing is that you save the backbone.out_channels in your Wrapper as self.out_channels as well (For concat you have to modify it as well).
|
@rijuld Thanks for working on this feature, that will be very helpful! |
No description provided.