-
Notifications
You must be signed in to change notification settings - Fork 35
Description
It looks like nodes containing embedded images like:
![[image.png]]
are successfully ignored as tokens when you run the cannoli and have vision disabled. This is normal. No image content goes to the tokenizer.
But then, the image's content gets sent to the tokenizer (even when vision is disabled) when you give the image embedding a custom width (such as 123 pixels wide) like so:
![[image.png|123]]
I was wondering why I kept getting told I had hundreds of thousands of tokens getting sent out. This was the culprit. It was resolved when I replaced all custom-width image embeddings (![[image.png|123]]) in my document with regular image embeddings (![[image.png]] ). Apparently this is a regex issue.
No real emergency here, just an odd behaviour that tripped me up. Easy to fix on my end.