Can't run on iOS with tauri - error: failed to run custom build command for ort-sys v2.0.0-rc.9

[do-me]

Describe the bug
EmbedAnything runs on MacOS with tauri but doesn't on iOS. Instead it throws an error:
ort-sys v2.0.0-rc.9

To Reproduce
See this example repo with all instructions and errors:
https://github.com/do-me/tauri_test_embed_anything

Desktop (please complete the following information):
Works on MacOS Sequoia.

Smartphone (please complete the following information):
Does not work on iOS iPhone 15 Pro, 16.1 through xcode.

Seems like it might be related to a candle bug. See also this conversation on tauri channel: https://discord.com/channels/616186924390023171/1329833396758908928/1329833396758908928

As well as this issue for candle directly: huggingface/candle#1841

Related: tauri-apps/tauri#12370

[sonam-pankaj95]

Hi, It's an issue in candle at the moment, you can try the onnx models meanwhile.

Let us know which models are you trying?

[do-me]

You can find the logic here: https://github.com/do-me/tauri_test_embed_anything/blob/main/src-tauri/src/lib.rs

I was testing with sentence-transformers/all-MiniLM-L6-v2 (because it's small) but would eventually use BAAI/bge-m3.
Thanks, will try the onnx version and report back here. Did you ever try to run on iOS yourself by chance?

[sonam-pankaj95]

Thank you for reporting, @akshayballal95 has integrated it with tauri, so I will let him speak :)

[do-me]

Wow you folks are fast, thanks! 🚀
Looking forward to your update!

[akshayballal95]

Hey. Unfortunately I haven't been able to test with iOS. But if the issue is related to ort, one thing I can do is to move ort to a feature flag. That way we can isolate the bug to either onnx or candle. Let me do this and get back here. You can test it out then and let me know how it goes.

[akshayballal95]

I have now moved ort behind a feature flag... so by default only candle models will load. You can try to build the branch: make-features

[do-me]

Thanks a lot. So for instance when running this code, by default it would now use pure candle instead of ort right?
Will give it a spin the next days and report back.

use embed_anything::{
    config::TextEmbedConfig,
    embed_query,
    embeddings::embed::Embedder,
};
use std::sync::Arc;

#[tauri::command]
async fn generate_embedding(text: String) -> Result<String, String> {
    let embedder = Arc::new(Embedder::from_pretrained_hf(
        "bert",
        "sentence-transformers/all-MiniLM-L6-v2",
        None,
    )
    .map_err(|e| e.to_string())?);
   
    let config = TextEmbedConfig::default().with_batch_size(1);

    let embeddings = embed_query(vec![text], &embedder, Some(&config))
        .await
        .map_err(|e| e.to_string())?;

    let embedding_string = format!("{:?}", embeddings);
        Ok(embedding_string)
}

[akshayballal95]

Exactly! Just make sure that you specify the dependency in your Cargo.toml file as follows:

embed_anything = {git = "https://github.com/StarlightSearch/EmbedAnything.git", branch = "make-features", default-features=false}

If this works out for you, I can merge it to main and make a release

Also on a side note, with my experience in integrating embed_anything in tauri, make sure that you move the embedding methods to a separate thread by doing tauri::async_runtime::spawn(...) to avoid the UI from getting locked up.

[do-me]

@akshayballal95 bingo! No more compilation error, awesome! 🥳
I had to fix the gemm-f16 problem (since mobile devices apparently do not support it) with this workflow. Need to fix a few other things in my frontend setup (considering cargo tauri dev / ios dev setup) to actually confirm it's working properly. Will report back soon.

[do-me]

Update, I guess I'm 99% there. My app runs properly on MacOS but on iOS - for whatever reason - it's missing the token file:

[hf_hub] Token file not found "/private/var/mobile/Containers/Data/Application/12345....12345/.cache/huggingface/token"

Do you think it's feasible to allow for passing tokens directly in this function?

pub fn from_pretrained_hf(
model: &str,
model_id: &str,
revision: Option<&str>,
) -> Result<Self, Error>

https://docs.rs/embed_anything/latest/embed_anything/embeddings/embed/enum.TextEmbedder.html#method.from_pretrained_hf

Trying to simply dump the token file to that path meanwhile.

[akshayballal95]

That's weird because even my machine doesn't have a token file in a similar path. The token is usually required to access gated models. Are you trying to access a gated model?

Anyway, I can allow passing a token string to the function. However, I am not sure if it will still need a token file to be present.

[do-me]

No not at all. On MacOS it works perfectly fine as well, that's the weird thing.

  let embedder = Arc::new(
      Embedder::from_pretrained_hf(
          "bert",
          "sentence-transformers/all-MiniLM-L6-v2",
          None,
      )
      .map_err(|e| e.to_string())?,
  );

Related: huggingface/hf-hub#54

I guess I got this specific problem solved by just creating the token file before accessing the HF hub. However, running into a new problem now:

Error: 1/0 error Operation not permitted (os error 1)

Not quite sure where this problem stems from. I have a feeling that it's some nasty permission/security bug that has something to do with tauri/xcode but is not necessarily related to embedanything. Here is the test repo: https://github.com/do-me/tauri-embedanything-ios

Created an issue here: tauri-apps/tauri#12497

[akshayballal95]

I have allowed passing a token. To avoid passing too many arguments to the function for scalability in the future, I have moved the Embedder creation to a builder pattern.

You can now use it as follows:

let model = Arc::new(EmbedderBuilder::new()
        .model_architecture("jina")
        .model_id(Some("jinaai/jina-embeddings-v2-small-en"))
        .revision(None)
        .token(None)
        .from_pretrained_hf()
        .unwrap());

I hope this is more convenient. The changes are in the same make-features branch. Let me know how it goes.

[akshayballal95]

Sadly, I won't be able to try your repo out as I don't have an Apple device. But thanks for sharing. It will be good to have this here in case people face similar issues in the future.

[do-me]

Thank you very much for your immediate reaction, that's awesome! Really keeps up my motivation this way 😄

From what I read online Android devices behave similarly considering the initial ort and gemm-f16 bugs so there might be a good chance that the error I encounter now on iOS is actually mobile specific.

Anyway I have a strong suspicion that this is not related to embedanything so even though I didn't solve my main problem yet, I'd close this issue for now.

Will come back here if I succeed eventually. Else, if you or anyone else is interested, I'll update the test repo above.

[do-me]

Dug a bit more and the problem is that on iOS I cannot use AppData dir but only AppCache dir. Is there already a way to define the path/dir where to download/cache the model files etc. to?
Like, e.g. here https://github.com/do-me/tauri-embedanything-ios/blob/main/src-tauri/src/lib.rs or with your code example, could I add the path somewhere?

let model = Arc::new(EmbedderBuilder::new()
        .model_architecture("jina")
        .model_id(Some("jinaai/jina-embeddings-v2-small-en"))
        .revision(None)
        .token(None)
        .from_pretrained_hf()
        .unwrap());

[akshayballal95]

Currently, there is not option to add path. But what you can do is set the following environment variable to where you want it to download

HF_HOME

https://huggingface.co/docs/huggingface_hub/en/package_reference/environment_variables

Let me know if this solution works for the short term. I will have to figure out how to allow a path to be set through the builder.