chore: restructure project into sub domains and move llm crates

[iambenkay]

What's new?

Create a parent folder for all LLM related features. This is the first step in organizing the project to accommodate all the different domains that will be introduced (embed, vector, graph, etc).

I will mention this PR after it gets merged in the relevant issues so that the bounty hunters can take note of this.

/claim #29
closes #29

[vigoo]

I really like the simplification of the scripts in the cargo makefile, and that it's a mininial change.

The wit directory in the root contains the golem:llm package and there can be only one root package. So the wit directory definitely need to be moved into llm/wit and every other "domain" will have its own wit package.

I'm not sure what kind of support cargo-make has for hierarchies of makefiles, but I like the idea in the issue about structuring them that way. But only with the addition that there must be a root makefile with commands that clean / update / build etc. everything together (while also having the ability to do so for only llm, for example).

If you could make these improvements, I'd happy to merge.

[iambenkay]

Alright I will work on these improvements and mention you when it's ready!

[iambenkay]

@vigoo you can check this out now! I have setup a hierarchy for the Makefile.

[kmatasfp]

The wit directory in the root contains the golem:llm package and there can be only one root package. So the wit directory definitely need to be moved into llm/wit and every other "domain" will have its own wit package.

Yeah this is the issue I ran into while working on #30. Temporarily ended up creating a new repo for my work to get around it and unblock myself. So cannot wait for this PR to get merged

[iambenkay]

Alright so I spent some more time designing the scripts. I wanted them to be really reusable and require minimal changes from people introducing new crates. Instead of hard coding make tasks in the root makefile I have chosen to allow these sub tasks to be run dynamically.

cargo make build

Will build all crates across all domains.

cargo make build llm (or any other incoming domain)

Will build all crates in the llm domain only.

To run for individual crates then the approach to use is to call the nested Makefile directly (this assumes that the file in the nested directory is named Makefile.toml):

cargo make --cwd llm build-openai

If desired I can spend some more time and implement something like:

cargo make build llm/openai

But at this point it feels like overkill that doesn't provide much value.

[vigoo]

Looks good. Let's go with this if the CI passes, we can refine it as more code is added if needed!

[iambenkay]

Let me update the script. 

[iambenkay]

@vigoo approve this workflow. It should work now. I changed to using the duckscript runner.
Originally I couldn't find a way to execute binaries in the duckscript docs (I searched everywhere), but now I decided to try using exec and it worked. Means the script can maintain it's cross platform nature.