Add black formatter as pre-commit step

[maxjakob]

We want to ensure consistent formatting of all notebooks and code. Hence we check on each commit and in CI that everything was formatted using black.

I added a make target to apply the pre-commit hooks to all files so that users can fix formatting issues locally.

[gitnotebooks]

Found 12 changed notebooks. Review the changes at https://gitnotebooks.com/elastic/elasticsearch-labs/pull/189

[maxjakob]

I was really happy that we have nbtest to make sure the notebooks are still doing the same as before 💚

[maxjakob]

Commented on notebook notebooks/document-chunking/with-index-pipelines.ipynb Cell 13 Line 8

# Setup the pipeline
CHUNK_SIZE = 400

client.ingest.put_pipeline(
    id="chunk_text_to_passages",
    processors=[
        {
            "script": {

@miguelgrinberg check it out, this is how GitNotebooks works. Let me know what you think. It's enabled for now but we can also remove it again if we decide against it

Respond and view the context here.

[maxjakob]

@miguelgrinberg We are running out of disk space in CI because of all the pip installs in the notebooks (not because of data in ES as I previously thought). This makes the virtual env directory get larger with every nbtest run. As a consequence, Elasticsearch stops accepting data and notebooks that run later start failing.

I also noticed that there is a bunch of unnecessary software on the GitHub runner taking up space, like the Android SDK (8GB) and other stuff. Somebody made an action that can remove these extras. This frees up the 8GB in ~1 minutes and resolves the issue.

But the maintainers admit that "This action is a hack, really", so I'm interested in your opinion on whether to keep it. The only alternative I see right now is to wipe the pip cache in between nbtest runs. And I would guess that will increase the runtime by more than the new action so my preference would probably be to keep it as in this PR.

[joemcelroy]

keep to remove those extras, freeing up the 8gb space :)

[miguelgrinberg]

I vote against. Not a good idea to incorporate actions from unknown sources into our CI from a security point of view. We can manually delete anything directly in our own buld logic if necessary.

Would it be the same to add these directly into our build to avoid running 3rd party code?

            sudo rm -rf /usr/share/dotnet
            sudo rm -rf /usr/local/lib/android
            sudo rm -rf /opt/ghc
            sudo rm -rf /opt/hostedtoolcache/CodeQL

[maxjakob]

That's fine for me. I guess the only advantage the action has is that if the directories move we don't have to adapt. It's unlikely and we would find out soon enough if we run out of disk space again.

[miguelgrinberg]

My guess is that if the directories move then the action would break in the same way our own deletions would, and then it would take some time for the author of the action to fix things up, so there would be down time either way. And doing it ourselves removes the risk of a future update to the 3rd party action doing bad things.