Calculate Embeddings for all entries

[lea-33]

Now the script / notebook to calculate the embeddings for all Zenodo Records in our Training Material exists.

We now have to run it for all Records to get the Embeddings.

@haesleinhuepf I have a few questions before getting started:

• I am not sure if it works for you to change one of my Huggingface Repositories / Datasets. I think I would have to make you a collaborator first, for that I need your username!
• right now, the code / functions make use of the free Github Models. For you to run the Code with another API Key I have to make some minor changes. Maybe you could tell me which API Key (openai?) you want to use and I could change the code to work for that!

Thanks! :)

[haesleinhuepf]

• I think I would have to make you a collaborator first, for that I need your username!

I think so too. My username on HF is haesleinhuepf

Later, we could also transfer the dataset to the ScaDS.AI organisation. I think I founded one at some point...

• For you to run the Code with another API Key I have to make some minor changes.

Which changes? Could this be an if-else statement? E.g. if USE_OPENAI: ... elif USE_GH_MODELS: ... else: warn() ? Similar to here:
https://github.com/haesleinhuepf/human-eval-bia/blob/main/demo/create_samples.ipynb

[lea-33]

Hi @haesleinhuepf,
the new version of the notebook is now available and should work with the openai API as well!

I was not able to execute the cell the last time due to limitations of the GH models for today, just in case you wondered why there is a error as last output!