issue adding pdf's to knowledge base

[agileandy]

I have a bunch of pdf files in ~/ebooks/agile and when I try to add them to a knowledge base it come up with loads of errors

leet kb add-local -p ~/ebooks/agile/ -k agile

These are valid pdf files, but they are giving errors.

Exception ignored in: <finalize object at 0x42ea66f40; dead>
Traceback (most recent call last):
  File "/Users/<user>/.pyenv/versions/3.11.0/lib/python3.11/weakref.py", line 590, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/<user>/.pyenv/versions/3.11.0/envs/leettools/lib/python3.11/site-packages/pypdfium2/internal/bases.py", line 43, in _close_template
    assert (parent is None) or not parent._tree_closed()
AssertionError:
Exception ignored in: <finalize object at 0x42e910820; dead>
Traceback (most recent call last):
  File "/Users/<user>/.pyenv/versions/3.11.0/lib/python3.11/weakref.py", line 590, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/<user>/.pyenv/versions/3.11.0/envs/leettools/lib/python3.11/site-packages/pypdfium2/internal/bases.py", line 43, in _close_template
    assert (parent is None) or not parent._tree_closed()
AssertionError:
Exception ignored in: <finalize object at 0x42ea66f00; dead>
Traceback (most recent call last):
  File "/Users/<user>/.pyenv/versions/3.11.0/lib/python3.11/weakref.py", line 590, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Is this a program error, is it not meant to ingest entire directories, or am I doing something wrong?

[pengfeng]

I just tested on a fresh check out and it worked OK for me:

(leettools) [~/work/github/leettools (main)]$ leet kb add-local -e .env.ollama -p ~/tmp/test -k test_add_local -l info
...
org	kb	docsource_id	docsink_id	document_uuid	URI
org-default	test_add_local_01	66f1775d-4cb2-475c-8ef8-40d1de7f124a	b7ff1316-97e7-487e-8ac3-51fcae9609ea	4a4b1466-24f0-42c3-913e-8a24b36a74f4	file:///Users/feng/tmp/test/What%20Is%20Leettools.md
org-default	test_add_local_01	66f1775d-4cb2-475c-8ef8-40d1de7f124a	fb9d2ab7-b48b-4246-803d-3c6f6fe0624e	936644fd-504f-4b96-adb9-3d49d0b5e7f4	file:///Users/feng/tmp/test/Results.md
org-default	test_add_local_01	66f1775d-4cb2-475c-8ef8-40d1de7f124a	e9baea7a-1579-45da-b0c3-e18fe2a49e3a	cb922839-db7c-449f-b7e0-9a83c31b18a0	file:///Users/feng/tmp/test/2402.01680v2.pdf

What is your operating system and python version? Also can you run with -l debug to see more detailed logs?

[agileandy]

I'm running macOS 15.2.
Python is 3.11.0 though I've got pyenv setup so can change it.

I notice you are specifying a -e option. If I omit this I'd assumed it uses my local env.ollama?

I'll put the -l option and report back.

[agileandy]

I've run with the -l option and also reduced the folder size to just 5 pdfs... it seems to inject the docs ok, but does give and error when it's finished.

/Users/<user>/.pyenv/versions/3.11.0/lib/python3.11/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '

However, then I put back all my pdfs (312 of them - some 50-60mb) it starts to error again. Seems like some process is just getting overloaded, or not 100% thread safe.

the debug.log is too big too upload.