Review of the Python Library documentation

[caiocmello]

This is a space for a joint effort of the Datalab editorial board to review the documentation of the Impresso Python Library and suggest improvements.

Impresso Python Library: https://impresso.github.io/impresso-py/

Please, add comments and suggestions to this issue so we can collect ideas to be discussed in the next meeting.

The meeting to discuss the outcome of this initiative will be on the 7th of October 2025 (10.30am)

[caiocmello]

Home/Create a session

Replace impresso = connect() with client = connect()

PS: change this in the whole library

[caovy-univers]

Resources

Content items

impresso.content_items.get("NZZ-1794-08-09-a-i0002")

Avoid using content uid in example/documentation as they are not guaranteed to be stable across releases (e.g., after data corrections or new ingests).

[EmanuelaBoros]

• What is the difference between Create a session and connect? Aren't both doing:

from impresso import connect
client = connect() 

• Is public_api_url argument really necessary?
• In what cases would one use persisted_token argument? It is still unclear (for me) how useful is it from the docs.

• Now, after connecting, where does one go? What do Result Object and Resources mean? I cannot propose a new naming, since it is still unclear on what they are.

[EmanuelaBoros]

Search

---

Trying the basic notebook:

result = impresso.search.find(
    term="European Union",
    order_by="date",
)
for article in result.pydantic.data[:3]:
    print(article.transcript)

None
None
None

• Why both search and find?
  • Feels repetitive: “search.find” sounds like a double verb. Is find the only function in search? impresso.find(term="Titanic")? Or since we have impresso.entities.find, maybe impresso.articles.find(term="Titanic")?
• What is pydantic?
  • For non-programmers or users unfamiliar with backend tooling, the word "pydantic" may be entirely meaningless. While the term might be explained in notebook examples, it is not currently documented in the official API documentation.
  • If a user wants to view article transcripts or titles or whatever, they are unlikely to think, “Let me access .pydantic.” Renaming .pydantic to something more intuitive? .records, .data, etc

Re-edit in the afternoon: found pydantic!!!! DataContainer

---

How to use search.find from official API documentation

result = impresso.search.find(term="European Union", order_by="date", limit=10)
result.pydantic.data[0]

ContentItem(uid='luxwort-1948-11-25-a-i0033', copyrightStatus='in_cpy', type='ar', sourceMedium='print', title='Europäischer Föderalistenkongreß in Rom', transcript=None, 
locationEntities=[], personEntities=[NamedEntity(uid='2-50-Pius_XII.', count=1.0)], organisationEntities=[], 
newsAgenciesEntities=[], topics=[TopicMention(uid='tm-de-all-v2.0_tp25_de', relevance=0.202), TopicMention(uid='tm-de-all-v2.0_tp52_de', relevance=0.16), TopicMention(uid='tm-de-all-v2.0_tp86_de', 
relevance=0.157), TopicMention(uid='tm-de-all-v2.0_tp14_de', relevance=0.112), TopicMention(uid='tm-de-all-v2.0_tp77_de', relevance=0.112), TopicMention(uid='tm-de-all-v2.0_tp95_de', relevance=0.096), 
TopicMention(uid='tm-de-all-v2.0_tp24_de', relevance=0.057)], transcriptLength=733.0, totalPages=1.0, languageCode='de', isOnFrontPage=False, publicationDate=datetime.datetime(1948, 11, 25, 0, 0, 
tzinfo=TzInfo(UTC)), issueUid='luxwort-1948-11-25-a', countryCode='LU', providerCode='BNL', 
mediaUid='luxwort', mediaType='newspaper')

• When using order_by="date" in search.find(...), users reasonably expect the returned items to expose a .date field — but instead they must access article.publicationDate.
• The field name publicationDate is a bit verbose and also does not follow the snake_case PEP 8 (publication_date).
• Would a user-friendly alias .date to the ContentItem model that returns the same value as publicationDate be nicer? This could align with the order_by="date". Or order_by="publication_date"`?

---

result = impresso.search.find(term="democracy", newspaper_id="NZZ", limit=5)
for article in result.pydantic.data:
    print(article.title)

None
None
None
None
None

OK, no titles.

---

result = impresso.search.find(entity_id="Q937")  # Albert Einstein
for article in result.pydantic.data:
    print(article)

• What is entity_id?

Ah..! One needs to impresso.entities.find(term="Albert Einstein"), take the internal uid and:

result = impresso.search.find(entity_id="2-50-Albert_Einstein")
for article in result.pydantic.data:
    print(article)

• Wouldn't searching by Q937 be more intuitive?

---

result = impresso.search.find(topic_id="tm-de-all-v2.0_tp25_de", limit=5)
for article in result.pydantic.data:
    print(article.title)

Öffentliche Wirtschaft kritisiert EG-Industriepolitik
Die Sonderstellung
nicht notwendigerweise eine Entmutigung ...
Lockvogelangebote
Geboten

• topic_id values like tm-de-all-v2.0_tp25_de are ... opaque.
• Results show article titles, but not why they belong to the topic.
• There's no helper method to explore or describe topics.
• article.topic_description? # e.g., "industry, economy, EG policy"
• article.topics[0].label? # e.g., "Industry and European Economy"
• Or, the last resort, provide topic lookup utilities?impresso.topics.describe("tm-de-all-v2.0_tp25_de")?

---

[EmanuelaBoros]

Search (continued)

---

result = impresso.search.find(term="European Union", language=["fr", "de"], limit=10)
result

result = impresso.search.find(term="European Union", \
         language={"hello?" : "Is it me you're looking for?"}, limit=10)
result

• Suprinsingly, language accepts any arbitrary value.
• Before the user tries all, maybe a ValueError: Invalid language parameter. Expected one or more of ['fr', 'de', 'en'], but got: {'hello?': "Is it me you're looking for?"} could help?

---

[EmanuelaBoros]

Search (continued)

---

Trying the famous date range.

from impresso.structures import DateRange
from datetime import date

date_range = DateRange(date(1900, 1, 1), date(1910, 12, 31))

result = impresso.search.find(
    term="constitution",
    date_range=date_range,
    limit=10
)

len(result.pydantic.data)

• Not sure if anyone (comfortable with programming or not) wants to do date_range = DateRange(date(1900, 1, 1), date(1910, 12, 31))
• Something simpler and intuitive, like: date_range = ["01-01-1900", "31-12-1910"], could possibly save someone's humour for the day.
• Also, not sure if someone would have looked for DateRange in impresso.structures. It feels more like a helper or filter than a structure.

---

[EmanuelaBoros]

Search (continued)

How many items are there?

---

result = impresso.search.find(term="constitution")
len(result.pydantic.data)

100

This seems to be the limit (not mentioned in official API documentation), so one cannot know the real total items that contain "constitution", unless .. pagination.

result = impresso.search.find(term="constitution", offset=100000)
len(result.pydantic.data)

100

So there are more than 100,000.

result = impresso.search.find(term="constitution", offset=586900)
len(result.pydantic.data)

70

So, in total, 586,970!
Understandable that the API doesn't expose total_results for performance reasons, but it could be nice to have at least an estimate of the total number of hits (some internal binary search?).

---

[EmanuelaBoros]

Search & a final feeling

While browsing the official API documentation, I found myself... having a few questions:

• What is a SearchResource?
  Why is it a resource? What is a resource? Is it a REST concept? A class? A magical portal?

• What is a MediaSource?
  Is this like a newspaper? A publishing house? A metadata tag?
  More importantly: where can I see a list of them?

• What exactly is a ContentItem?
  Is that an article? A paragraph? A page? A paragraph of a page of an article? Asking for a friend.

• What is CollectionsResource?
  The doc says: "Work with collections."
  What counts as a collection? Can I list them? Can I search within them?

• What is ToolsResource?
  It sounds like a toolbox... but of what? And why does it contain ner, nel, and ner_nel?

• What’s a NerContainer?
  Is it a list of entities? A metadata box? An actual container object (like in Docker)?
  How is it different from “Named Entity Recognition”? Why is NER in ToolsResource instead of its own named thing?

---

So.., the docs could really benefit from:

• If it stays like this, a glossary of key concepts, with some simple definitions and links, to explain the famous ContentItem, MediaSource, and Collection.
• A small "Conceptual Overview" section for non-developers or developers (also tired developers 😅) that maps out how all these pieces connect.

Is it .. like this?:

Search  
└── ContentItem  
  ├── has **MediaSource**  
  ├── belongs to **Collection**  
  └──  has **Entities** (via **ToolsResource**)  

---

[caiocmello]

@EmanuelaBoros thanks so much! This is really helpful. I've started designing this 'conceptual overview' for non-developers today. It is just a draft. I will share it here (next comment) so we can talk about it in the next meeting