Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
methods/catalog/probe/model.py
Outdated
| import pandas as pd | ||
|
|
||
| from ...api import RecourseMethod | ||
| from methods.catalog.roar.library import wachter_rip_recourse | ||
| from methods.processing import ( |
There was a problem hiding this comment.
The Probe model imports wachter_rip_recourse from methods.catalog.roar.library, but that module only exposes roar_recourse. The implementation of wachter_rip_recourse is added in this commit under methods.catalog.probe.library. As written, importing Probe raises ImportError: cannot import name 'wachter_rip_recourse' and the model cannot be instantiated. The import should target the new Probe library instead.
Useful? React with 👍 / 👎.
methods/catalog/probe/model.py
Outdated
| encoded_feature_names = self._mlmodel.encoder.get_feature_names( | ||
| self._mlmodel.data.categoricals | ||
| ) |
There was a problem hiding this comment.
categoricals attribute on dataset
When building categorical feature indices, the code calls self._mlmodel.encoder.get_feature_names(self._mlmodel.data.categoricals). The dataset objects elsewhere in the code expose the list of categorical features via the categorical attribute (singular). Because categoricals does not exist, calling get_counterfactuals will raise an AttributeError before any recourse is generated. Use the existing categorical attribute instead.
Useful? React with 👍 / 👎.
Initially had the PROBE model commits inside this branch. They have now been moved to their own branch, and this branch now only contains the RBR model commits.
(WIP) since the method is implemented, the main thing left is making the data processing and experiment process the same as the original paper. They fo use the same datasets but process thems slightly different (like using different features). Their model (mlp) is also different. So running with the dataset and model as they are (the way we have them) will definitly not get the same results. Will have to think of how to best combine with their format. One option is not using our model and data catalogs to load the data, and simply port over their model and data creation/processing code.
Initially had the PROBE model commits inside this branch. They have now been moved to their own branch, and this branch now only contains the RBR model commits.
(WIP) since the method is implemented, the main thing left is making the data processing and experiment process the same as the original paper. They fo use the same datasets but process thems slightly different (like using different features). Their model (mlp) is also different. So running with the dataset and model as they are (the way we have them) will definitly not get the same results. Will have to think of how to best combine with their format. One option is not using our model and data catalogs to load the data, and simply port over their model and data creation/processing code.
Ran throught the code to simply get the mothod to simply run. Need more work to confirm correctness of results. WIP commit with debug prints and small fixes.
Getting the reproduce for this method was a bit tricky. The main challenges seem to be just making sure that the dataset and models are processed and trained correctly. I have tried my best to build the model just like they have and also processing the dataset the same as them. Although the results are not identical, they can be classified atleast a level 1 on the reproduction scale and definitly can be improved in the near future. The Method does infact work well in finding robust recourse and some metrics seem to be inline with the results of the paper.
I worked to resolve several issues, mainly stemming from the predict function I implemented in the rbr_loss script. The method results in the reproduce file now fairly closely align with those from the original authors code. The method does seem to heavily rely on the effectiveness of the trained model for its recourse finding. That means that if the model is very poor, then this method may not be able to actually get recourse. Overall, in terms of reproduction, I believe this can be marked between a 1-2.
2 participants
No description provided.