PGVector gotcha

[humbertovnavarro]

By default postgres uses ASCII encoding which causes document uploads to fail if they contain unicode characters.

[mihirahlawat]

I spent a lot of time and will try to save you some.

The issue: If PDFs have ligatures in them, PyPDF does not recognize them and gives you an empty character (in unicode) which throws error in a PGVector Database.

The solution: I used PyMuPDF for loading pdf documents. It is able to recognize ligatures seamlessly.