New adapters and local development

[ronaldmannak]

Question on the new adapters: how are downstream apps expected to develop against a local mlx-swift-lm checkout when all four adapter packages (DePasqualeOrg/swift-huggingface-mlx, DePasqualeOrg/swift-hf-api-mlx, DePasqualeOrg/swift-tokenizers-mlx, DePasqualeOrg/swift-transformers-mlx) pin mlx-swift-lm to an exact commit?
That makes local development and updating hard for projects that track main or use a local path dependency. Is the intended workflow to fork/vendor the adapters, or is the plan to retire the adapters with v3? As it stands, this feels overly complicated for downstream consumers.
Having multiple adapter options is already unfortunate, and the current exact-pin approach makes that worse. Especially since using the old HF packages directly is no longer an option. Right now the practical answer seems to be “fork/vendor the adapters or write the bridge yourself,” which undermines much of the value of publishing them as separate packages. If there’s a better alternative or if this will be addressed in v3, please let me know.

[davidkoski]

That is an interesting point. When we tag mlx-swift-lm, it could use up to next major, but using main for development is a good point.

I think there are two methods right now:

• use the macros in mlx-swift-lm to adapt plain HuggingFace, see mlx-swift-examples prep for mlx-swift-lm 3.x release mlx-swift-examples#468

• you can point to a local checkout of the package (which is probably what you want with main anyway), see https://developer.apple.com/documentation/xcode/editing-a-package-dependency-as-a-local-package You can also drag and drop the directory into Xcode and hook it up that way. This is how I normally do dev/test so that I can edit all the pieces at once.

[ronaldmannak]

Thanks for the quick response!

• use the macros in mlx-swift-lm to adapt plain HuggingFace, see mlx-swift-examples prep for mlx-swift-lm 3.x release mlx-swift-examples#468

I missed that part.There are built-in macros that don't require the use of adapters and allow you to use the old HF packages direcly?

• you can point to a local checkout of the package (which is probably what you want with main anyway), see https://developer.apple.com/documentation/xcode/editing-a-package-dependency-as-a-local-package You can also drag and drop the directory into Xcode and hook it up that way. This is how I normally do dev/test so that I can edit all the pieces at once.

Correct me if I'm wrong, but I believe that swift packages work differently than an xcproj what you're referring to. I'm using a swift package and the dependency graph doesn't resolve when I add an adapter because of the conflicting mlx-lm-swift references. The only way to make this work is by downloading / forking the adapters and editing the location of their mlx-swift-lm dependency.

If the macros provide a local bridge without needing the external adapter packages, that seems like the simplest solution. I’ll take a closer look tomorrow morning and report back. Thanks!

[ronaldmannak]

Oops, I might have posted a comment meant for this thread in a different discussion. If you see a weird comment by me somewhere else, it was meant to be posted here ;) Here's a short version of the missing comment:

The issue might be the pinning of a specific mlx-swift-lm commit in the adapters.

The adapters shouldn't really pin a commit or version of mlx-swift-lm since new models and updates are too frequent. Forcing developers to wait for an external dependency to be updated makes this repo brittle and vulnerable.

Alternative solutions might be to (somehow) remove the mlx-swift-lm dependencies in the adapters, or to add the adapters as separate targets in mlx-swift-lm. Not sure if either solution is possible or easy, but it is worth to explore because the current architecture is very unfortunate

[DePasqualeOrg]

The adapters will track mlx-swift-lm version 3 when that is released. At the moment they need to pin a commit hash.

Having multiple adapter options is already unfortunate

@ronaldmannak, do you find it unfortunate to have the option to load models more than 2x faster and tokenize between 2 and 13x faster? That is what the adapter options allow. With Swift Tokenizers, users can now even choose to use the Rust backend for tokenization, JSON parsing, and Jinja template compilation, which offers even better performance and forward compatibility.

Previously Swift users had only one option, which still lags far behind the Python ecosystem in performance. Now MLX Swift LM allows anyone to use any tokenizer and downloader solution they want – or even roll their own. Please choose your words carefully.

Alternative solutions might be to (somehow) remove the mlx-swift-lm dependencies in the adapters, or to add the adapters as separate targets in mlx-swift-lm. Not sure if either solution is possible or easy, but it is worth to explore because the current architecture is very unfortunate

Neither is possible. We explored all the options in #118.

The current architecture gives people both maximum freedom (use any tokenizer or downloader package with an adapter, or just conform to the protocols by copying ~100 lines of code) as well as convenience options for the most common use cases (use the macro for Hugging Face, or use the adapter packages). Again, I think you should reconsider whether this is "very unfortunate" or actually the best possible outcome.

When MLX Swift LM version 3 is released, it should stay updated for anyone using the adapters. If there are any issues with using them, please let me know and I will update them accordingly.

[DePasqualeOrg]

To make it easier for users to understand the integration options, I've added some explanations to the readme in #201.

[davidkoski]

Correct me if I'm wrong, but I believe that swift packages work differently than an xcproj what you're referring to. I'm using a swift package and the dependency graph doesn't resolve when I add an adapter because of the conflicting mlx-lm-swift references. The only way to make this work is by downloading / forking the adapters and editing the location of their mlx-swift-lm dependency.

No, I am talking about swiftpm packages! This might be useful documentation as it makes developing multiple repos pretty easy.

Let me demonstrate -- there are a different things going on here:

You can see I pointed mlx-swift-lm to a branch (since I can make branches in the main repo, this is doable). In the lower section it shows mlx-swift-lm as checked out on that branch.

But notice the mlx-swift and mlx-swift-lm in the upper section. They both have symlink icons. The mlx-swift-lm one is "broken" because I have mlx-swift-lm open in another window (you don't want to do this). I can edit any file in mlx-swift just like it was part of the project. No need to commit and push or anything.

I got this by dragging the mlx-swift and mlx-swift-lm directory onto the mlx-swift-examples item at the top. Then chose to reference files in place:

Try it out -- if you have a local checkout you can point it at whatever you want.

[davidkoski]

OK, I finally found a reference to this elusive feature:

• https://developer.apple.com/videos/play/wwdc2019/410/?time=1467

[ronaldmannak]

OK, I finally found a reference to this elusive feature:

Yeah no, I knew what you meant. I’m familiar with that feature, it’s how you used to add local packages to an .xcodeproj before Xcode introduced the Add Local… button in the Package Dependencies tab.

My point was that this does not work when the project itself is a package rather than an .xcodeproj, which is the blocker for me. If you drag a local package onto a package, Xcode suggests “upgrading” it to a workspace instead. Of course, you can still add a local path dependency directly in the package manifest. However, you still need to remove the commit pin in the adapters.

But more importantly, this is a bigger issue than just developing mlx-swift-lm locally. Even people using the GitHub-hosted version are now affected. By pinning the adapter packages to a specific commit (or eventually a version) of mlx-swift-lm, the adapter maintainer has effectively become the gatekeeper for access to new mlx-swift-lm releases.

In practice, new versions of mlx-swift-lm will not be usable until the adapter pins are updated. Progress is therefore tied to whether and when the adapter maintainer updates all four adapters. Even worse, developers can no longer choose their own commit or version without forking the adapters. The best-case outcome is simply a delay. Based on the timing of their comments, I suspect the maintainer is not in the Pacific time zone, so updates may be delayed further if PRs are typically merged during Pacific daytime. Losing the freedom to choose a commit or version is, in my view, extremely unfortunate.

I assume this could be avoided if the adapters pointed to the main branch instead of a fixed commit or version, but that would require the maintainers to give up their new gatekeeping role.

Let me check the macros now. If those work, it might actually a good idea to add that to the documentation too, so developers understand the choices.

[davidkoski]

My point was that this does not work when the project itself is a package rather than an .xcodeproj, which is the blocker for me. If you drag a local package onto a package, Xcode suggests “upgrading” it to a workspace instead. Of course, you can still add a local path dependency directly in the package manifest. However, you still need to remove the commit pin in the adapters.

No, it does work with packages! Unless I am misunderstanding what you are asking!

You can use the drag & drop technique if the enclosing project is an xcodeproj OR a Package.swift.

[DePasqualeOrg]

@ronaldmannak, the adapters are an optional convenience. No one is forced to use them. Anyone can fork them or just copy the ~100 lines of code for the protocol conformance. This is already clear in the readme. Your characterization of "gatekeeping" is not accurate.

I would prefer not to have to host the Hugging Face adapters at all, and can transfer them to ml-explore or Hugging Face at any time.

The adapter packages should point to a major version of mlx-swift-lm in order to avoid problems with package resolution. The commit hash pin is a temporary solution in order to avoid these problems until the major version release.

My update to the readme outlines all the options for integration, including macros. Maybe @davidkoski can add a section with usage examples for the macros.

[ronaldmannak]

The adapter packages should point to a major version of mlx-swift-lm in order to avoid problems with package resolution

This choice has broad implications

[DePasqualeOrg]

@ronaldmannak, so far you've only raised histrionic objections to an optional convenience that no one is forced to use. Please stop this exaggerated posturing. If you don't want to use these optional conveniences, just copy ~100 lines of code.

[ronaldmannak]

You are totally right @DePasqualeOrg and I appreciate you taking time out of your busy schedule to not only reply but also give me a free psychological evaluation.

[davidkoski]

@ronaldmannak, so far you've only raised histrionic objections to an optional convenience that no one is forced to use. Please stop this exaggerated posturing. If you don't want to use these optional conveniences, just copy ~100 lines of code.

@DePasqualeOrg Please no ad hominem attacks (or any attacks at all). I think the questions are legit. This is a major API break (hence the major API bump). People are still not used to it. It will improve, but it is still all new.

Be supportive or please do not post at all -- I want this to be a welcoming space for all. You contributions are valued, so I hope that you do continue to post, but be constructive. If people have problems or questions, assume that there is a good reason for it.

• https://github.com/ml-explore/mlx-swift-lm/blob/main/CODE_OF_CONDUCT.md

[DePasqualeOrg]

@davidkoski, I've been falsely accused of being a "gatekeeper". I am objecting to the exaggerated, false, and vaguely ominous ("This choice has broad implications") claims being made against me without any argument of substance.

So far, as best I can tell, @ronaldmannak's main objection is that there might be a gap of a few hours between a major version release and an update to the optional integration packages. Do you think this is worth an extended discussion, given that:

• The integration packages are entirely optional conveniences
• Anyone can fork them, copy the code, or use the macros

Note that the integration packages need to pin either a major version or a commit hash to be deemed "stable", and otherwise packages won't resolve and projects won't build.

If there is some other problem here, please clarify and be specific, and otherwise let's please move on to more important things.