We need a solution for caching in Nodes

[scorinaldi]

Sometimes you'll want to cache information after a single node run , like in the case of very large ML models. Consider @Roulbac 's PR here: #41

This PR introduces a library for managing HuggingFace models, but comes with the wrinkle where now, in order to develop new HF nodes , contributors have to work in two places (/nodes repo and /python repo)

How should we think about caching? Also, how can we consolidate all Node creation work into a single place going forward?

cc @jackparmer and @smahmed776

[Roulbac]

How about we move flojoy.hflib into nodes.hflib?

[jackparmer]

How about we move flojoy.hflib into nodes.hflib?

makes sense to me?

what do u think @scorinaldi @Ben-Epstein ?

[Roulbac]

How about we move flojoy.hflib into nodes.hflib?

makes sense to me?

what do u think @scorinaldi @Ben-Epstein ?

Make sense to me too, I'll open draft PRs in the meantime

[Roulbac]

I gave this some more thought in light of our recent conversation about isolating nodes in their own respective virtual environments (with something like the @run_in_venv(pip_dependencies=[...]) decorator used here for example), and it won't be practical to have these nodes share a common utility library to manage re-usable models when their dependencies are isolated.

Moreover, we can use the huggingface_hub library that only handles caching of files from the hub, and does it well.

This code for example downloads a file and caches it locally on disk such that it does not need to re-download it the next time it is invoked.

[Ben-Epstein]

How about we move flojoy.hflib into nodes.hflib?

makes sense to me?

what do u think @scorinaldi @Ben-Epstein ?

I might be missing some context here, but i generally agree that it's difficult at first to be developing in both the nodes repo and the python repo to add new functionality.

It would seem to me as though if you wanted to implement a general caching framework within flojoy it should be generic, and implemented in the python repo, which can then be pulled into various nodes downstream (hopefully the caching mechanism would be fairly stable and wouldn't require frequent changes).

If you need specialized caching logic for HF, then I agree that hflib should be moved into the nodes repo.

When we talk about caching in flojoy, what are the goals for the user? Does a node stay alive between consecutive executions? ie will it's filesystem continue to exist? If so, it would be fairly easy as a starting point to write to the local disk whenever a user has a @cache decorator for example. You can even just leverage pythons cachetools.cached module to extend the regular python caching to look at disk instead of hold things in memory.

[scorinaldi]

Agree with @Ben-Epstein that this issue calls for a generalized approach for all of FJ to caching. I think folks should sit down and actually meet about this and produce a doc with recommendations. As a short-term solution I'm fine with a very specific solution that only covers HuggingFace models as @Roulbac notes, but I think we leave this issue open until we have the beginnings of something much more general in place.

[jackparmer]

@scorinaldi can you please corral technical stakeholders together and setup a Zoom for next week? I don't need to be part of this meeting.

…

On Fri, Jun 30, 2023 at 10:13 AM Sharif Corinaldi ***@***.***> wrote: Agree with @Ben-Epstein <https://github.com/Ben-Epstein> that this issue calls for a generalized approach for all of FJ to caching. I think folks should sit down and actually meet about this and produce a doc with recommendations. As a short-term solution I'm fine with a very specific solution that only covers HuggingFace models as @Roulbac <https://github.com/Roulbac> notes, but I think we leave this issue open until we have the beginnings of something much more general in place. — Reply to this email directly, view it on GitHub <#42 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAOHQ2SNJ3WDRMZ5AHAKCDLXN3NHTANCNFSM6AAAAAAZGSOXIE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[scorinaldi]

For sure! Will reach out to folks over slack