sample data for finetuning tutorial

[dpeng28]

Hi Nicheformer Team,

I am wondering if it's possible to provide a sample/subsetted anndata for us to follow the 'downstream_fine_tune.ipynb' notebook so we know how to format our training data? I am running into an issue when running

trainer.fit(
    model=fine_tune_model,
    train_dataloaders=train_loader,
    val_dataloaders=val_loader
)

during the sanity checking stage of dataloader, and it would be great if we can get a template on how we should format our custom anndata.

Thanks!

[AlejandroTL]

Hello,

which issue do you run into? Can you please post the error trace? I used just a h5ad I had with cell type labels, nothing special.

Thanks!

[dpeng28]

Hi,

I have been following this tutorial jupyter notebook to finetune the regression task of predicting neighborhood proportions. Here are the errors

| Name        | Type        | Params | Mode 
----------------------------------------------------
0 | backbone    | Nicheformer | 49.3 M | eval 
1 | linear_head | Linear      | 16.9 K | train
2 | softmax     | Softmax     | 0      | train
3 | cls_loss    | MSELoss     | 0      | train
----------------------------------------------------
16.9 K    Trainable params
49.3 M    Non-trainable params
49.3 M    Total params
197.276   Total estimated model params size (MB)
3         Modules in train mode
142       Modules in eval mode
SLURM auto-requeueing enabled. Setting signal handlers.
[/homes/pengdx4/miniconda3/envs/nicheformer_env/lib/python3.9/site-packages/torch/cuda/__init__.py:734](http://localhost:8917/lab/tree/Projects/skin_xenium_analysis/scripts/miniconda3/envs/nicheformer_env/lib/python3.9/site-packages/torch/cuda/__init__.py#line=733): UserWarning: Can't initialize NVML
  warnings.warn("Can't initialize NVML")
Training the model...
Sanity Checking: |          | 0/? [00:00<?, ?it/s]
[/homes/pengdx4/miniconda3/envs/nicheformer_env/lib/python3.9/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:476](http://localhost:8917/lab/tree/Projects/skin_xenium_analysis/scripts/miniconda3/envs/nicheformer_env/lib/python3.9/site-packages/pytorch_lightning/trainer/connectors/data_connector.py#line=475): Your `val_dataloader`'s sampler has shuffling enabled, it is strongly recommended that you turn shuffling off for val[/test](http://localhost:8917/test) dataloaders.
Sanity Checking DataLoader 0:   0%|          | 0/2 [00:00<?, ?it/s]
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Cell In[14], line 3
      1 # Train the model
      2 print("Training the model...")
----> 3 trainer.fit(
      4     model=fine_tune_model,
      5     train_dataloaders=train_loader,
      6     val_dataloaders=train_loader
      7 )

File [~/miniconda3/envs/nicheformer_env/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py:561](http://localhost:8917/lab/tree/Projects/skin_xenium_analysis/scripts/~/miniconda3/envs/nicheformer_env/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py#line=560), in Trainer.fit(self, model, train_dataloaders, val_dataloaders, datamodule, ckpt_path)
    559 self.training = True
    560 self.should_stop = False
--> 561 call._call_and_handle_interrupt(
    562     self, self._fit_impl, model, train_dataloaders, val_dataloaders, datamodule, ckpt_path
    563 )

File [~/miniconda3/envs/nicheformer_env/lib/python3.9/site-packages/pytorch_lightning/trainer/call.py:48](http://localhost:8917/lab/tree/Projects/skin_xenium_analysis/scripts/~/miniconda3/envs/nicheformer_env/lib/python3.9/site-packages/pytorch_lightning/trainer/call.py#line=47), in _call_and_handle_interrupt(trainer, trainer_fn, *args, **kwargs)
     46     if trainer.strategy.launcher is not None:
     47         return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
---> 48     return trainer_fn(*args, **kwargs)
     50 except _TunerExitException:
     51     _call_teardown_hook(trainer)

File [~/miniconda3/envs/nicheformer_env/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py:599](http://localhost:8917/lab/tree/Projects/skin_xenium_analysis/scripts/~/miniconda3/envs/nicheformer_env/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py#line=598), in Trainer._fit_impl(self, model, train_dataloaders, val_dataloaders, datamodule, ckpt_path)
    592     download_model_from_registry(ckpt_path, self)
    593 ckpt_path = self._checkpoint_connector._select_ckpt_path(
    594     self.state.fn,
    595     ckpt_path,
    596     model_provided=True,
    597     model_connected=self.lightning_module is not None,
    598 )
--> 599 self._run(model, ckpt_path=ckpt_path)
    601 assert self.state.stopped
    602 self.training = False

File [~/miniconda3/envs/nicheformer_env/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py:1012](http://localhost:8917/lab/tree/Projects/skin_xenium_analysis/scripts/~/miniconda3/envs/nicheformer_env/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py#line=1011), in Trainer._run(self, model, ckpt_path)
   1007 self._signal_connector.register_signal_handlers()
   1009 # ----------------------------
   1010 # RUN THE TRAINER
   1011 # ----------------------------
-> 1012 results = self._run_stage()
   1014 # ----------------------------
   1015 # POST-Training CLEAN UP
   1016 # ----------------------------
   1017 log.debug(f"{self.__class__.__name__}: trainer tearing down")

File [~/miniconda3/envs/nicheformer_env/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py:1054](http://localhost:8917/lab/tree/Projects/skin_xenium_analysis/scripts/~/miniconda3/envs/nicheformer_env/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py#line=1053), in Trainer._run_stage(self)
   1052 if self.training:
   1053     with isolate_rng():
-> 1054         self._run_sanity_check()
   1055     with torch.autograd.set_detect_anomaly(self._detect_anomaly):
   1056         self.fit_loop.run()

File [~/miniconda3/envs/nicheformer_env/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py:1083](http://localhost:8917/lab/tree/Projects/skin_xenium_analysis/scripts/~/miniconda3/envs/nicheformer_env/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py#line=1082), in Trainer._run_sanity_check(self)
   1080 call._call_callback_hooks(self, "on_sanity_check_start")
   1082 # run eval step
-> 1083 val_loop.run()
   1085 call._call_callback_hooks(self, "on_sanity_check_end")
   1087 # reset logger connector

File [~/miniconda3/envs/nicheformer_env/lib/python3.9/site-packages/pytorch_lightning/loops/utilities.py:179](http://localhost:8917/lab/tree/Projects/skin_xenium_analysis/scripts/~/miniconda3/envs/nicheformer_env/lib/python3.9/site-packages/pytorch_lightning/loops/utilities.py#line=178), in _no_grad_context.<locals>._decorator(self, *args, **kwargs)
    177     context_manager = torch.no_grad
    178 with context_manager():
--> 179     return loop_run(self, *args, **kwargs)

File [~/miniconda3/envs/nicheformer_env/lib/python3.9/site-packages/pytorch_lightning/loops/evaluation_loop.py:145](http://localhost:8917/lab/tree/Projects/skin_xenium_analysis/scripts/~/miniconda3/envs/nicheformer_env/lib/python3.9/site-packages/pytorch_lightning/loops/evaluation_loop.py#line=144), in _EvaluationLoop.run(self)
    143     self.batch_progress.is_last_batch = data_fetcher.done
    144     # run step hooks
--> 145     self._evaluation_step(batch, batch_idx, dataloader_idx, dataloader_iter)
    146 except StopIteration:
    147     # this needs to wrap the `*_step` call too (not just `next`) for `dataloader_iter` support
    148     break

File [~/miniconda3/envs/nicheformer_env/lib/python3.9/site-packages/pytorch_lightning/loops/evaluation_loop.py:437](http://localhost:8917/lab/tree/Projects/skin_xenium_analysis/scripts/~/miniconda3/envs/nicheformer_env/lib/python3.9/site-packages/pytorch_lightning/loops/evaluation_loop.py#line=436), in _EvaluationLoop._evaluation_step(self, batch, batch_idx, dataloader_idx, dataloader_iter)
    431 hook_name = "test_step" if trainer.testing else "validation_step"
    432 step_args = (
    433     self._build_step_args_from_hook_kwargs(hook_kwargs, hook_name)
    434     if not using_dataloader_iter
    435     else (dataloader_iter,)
    436 )
--> 437 output = call._call_strategy_hook(trainer, hook_name, *step_args)
    439 self.batch_progress.increment_processed()
    441 if using_dataloader_iter:
    442     # update the hook kwargs now that the step method might have consumed the iterator

File [~/miniconda3/envs/nicheformer_env/lib/python3.9/site-packages/pytorch_lightning/trainer/call.py:328](http://localhost:8917/lab/tree/Projects/skin_xenium_analysis/scripts/~/miniconda3/envs/nicheformer_env/lib/python3.9/site-packages/pytorch_lightning/trainer/call.py#line=327), in _call_strategy_hook(trainer, hook_name, *args, **kwargs)
    325     return None
    327 with trainer.profiler.profile(f"[Strategy]{trainer.strategy.__class__.__name__}.{hook_name}"):
--> 328     output = fn(*args, **kwargs)
    330 # restore current_fx when nested context
    331 pl_module._current_fx_name = prev_fx_name

File [~/miniconda3/envs/nicheformer_env/lib/python3.9/site-packages/pytorch_lightning/strategies/strategy.py:412](http://localhost:8917/lab/tree/Projects/skin_xenium_analysis/scripts/~/miniconda3/envs/nicheformer_env/lib/python3.9/site-packages/pytorch_lightning/strategies/strategy.py#line=411), in Strategy.validation_step(self, *args, **kwargs)
    410 if self.model != self.lightning_module:
    411     return self._forward_redirection(self.model, self.lightning_module, "validation_step", *args, **kwargs)
--> 412 return self.lightning_module.validation_step(*args, **kwargs)

File [~/Github/nicheformer/src/nicheformer/models/_nicheformer_fine_tune.py:226](http://localhost:8917/lab/tree/Projects/skin_xenium_analysis/scripts/~/Github/nicheformer/src/nicheformer/models/_nicheformer_fine_tune.py#line=225), in NicheformerFineTune.validation_step(self, batch, batch_idx)
    223 masked_indices = batch['masked_indices']
    224 attention_mask = batch['attention_mask']
--> 226 predictions = self.forward(masked_indices, attention_mask)
    227 cls_prediction = predictions['cls_prediction']
    228 label = batch['label']

File [~/Github/nicheformer/src/nicheformer/models/_nicheformer_fine_tune.py:121](http://localhost:8917/lab/tree/Projects/skin_xenium_analysis/scripts/~/Github/nicheformer/src/nicheformer/models/_nicheformer_fine_tune.py#line=120), in NicheformerFineTune.forward(self, x, attention_mask)
    119 """Forward pass through the model."""
    120 # Get embeddings
--> 121 token_embedding = self.backbone.embeddings(x)
    123 # Add positional embeddings
    124 if self.backbone.hparams.learnable_pe:

File [~/miniconda3/envs/nicheformer_env/lib/python3.9/site-packages/torch/nn/modules/module.py:1739](http://localhost:8917/lab/tree/Projects/skin_xenium_analysis/scripts/~/miniconda3/envs/nicheformer_env/lib/python3.9/site-packages/torch/nn/modules/module.py#line=1738), in Module._wrapped_call_impl(self, *args, **kwargs)
   1737     return self._compiled_call_impl(*args, **kwargs)  # type: ignore[misc]
   1738 else:
-> 1739     return self._call_impl(*args, **kwargs)

File [~/miniconda3/envs/nicheformer_env/lib/python3.9/site-packages/torch/nn/modules/module.py:1750](http://localhost:8917/lab/tree/Projects/skin_xenium_analysis/scripts/~/miniconda3/envs/nicheformer_env/lib/python3.9/site-packages/torch/nn/modules/module.py#line=1749), in Module._call_impl(self, *args, **kwargs)
   1745 # If we don't have any hooks, we want to skip the rest of the logic in
   1746 # this function, and just call forward.
   1747 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   1748         or _global_backward_pre_hooks or _global_backward_hooks
   1749         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1750     return forward_call(*args, **kwargs)
   1752 result = None
   1753 called_always_called_hooks = set()

File [~/miniconda3/envs/nicheformer_env/lib/python3.9/site-packages/torch/nn/modules/sparse.py:190](http://localhost:8917/lab/tree/Projects/skin_xenium_analysis/scripts/~/miniconda3/envs/nicheformer_env/lib/python3.9/site-packages/torch/nn/modules/sparse.py#line=189), in Embedding.forward(self, input)
    189 def forward(self, input: Tensor) -> Tensor:
--> 190     return F.embedding(
    191         input,
    192         self.weight,
    193         self.padding_idx,
    194         self.max_norm,
    195         self.norm_type,
    196         self.scale_grad_by_freq,
    197         self.sparse,
    198     )

File [~/miniconda3/envs/nicheformer_env/lib/python3.9/site-packages/torch/nn/functional.py:2551](http://localhost:8917/lab/tree/Projects/skin_xenium_analysis/scripts/~/miniconda3/envs/nicheformer_env/lib/python3.9/site-packages/torch/nn/functional.py#line=2550), in embedding(input, weight, padding_idx, max_norm, norm_type, scale_grad_by_freq, sparse)
   2545     # Note [embedding_renorm set_grad_enabled]
   2546     # XXX: equivalent to
   2547     # with torch.no_grad():
   2548     #   torch.embedding_renorm_
   2549     # remove once script supports set_grad_enabled
   2550     _no_grad_embedding_renorm_(weight, input, max_norm, norm_type)
-> 2551 return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)

RuntimeError: Expected tensor for argument #1 'indices' to have one of the following scalar types: Long, Int; but got torch.DoubleTensor instead (while checking arguments for embedding)