Xapian issue when title is too "long"

[benoit74]

openzim/mwoffliner#2318 exhibit an issue around Xapian.

Digging deeper, I narrowed down the problem:

• it is linked to item title
• when item title is 122 times or less the ي character, everything is fine
• when item title is 123 times or more the ي character (or any other 2 bytes UTF-8 character), we get the Xapian error
• when item title is 82 times or more the ࠄ character (or any other 3 bytes UTF-8 character), we get the Xapian error

I reproduce the issue both with python-libzim and node-libzim.

Here is minimalist Python code snippet reproducing the error with a 2 bytes character (it will fail when i is 123)

from zimscraperlib.zim import Creator
from pathlib import Path
from zimscraperlib.zim import metadata

creator = Creator(Path("tests.zim"), "index.html").config_metadata(
    std_metadata=metadata.DEFAULT_DEV_ZIM_METADATA
)

# start creator early to detect any problem early as well
creator.start()
creator.set_mainpath("index")

creator.add_item_for("index", "Main Page", content="any", is_front=True )

for i in range(256):
    print(i)
    path = f"path{i}"
    title = "ي" * i
    creator.add_item_for(path, title, content="any", is_front=True )

creator.finish()

I will implement an interim fix in mwoffliner, but we probably need to either fix this issue or document this limitation (if not already done, I might have missed it).

[benoit74]

Nota: exact limit seems to be 245 bytes ; 246 bytes is KO

[kelson42]

@veloman-yunkan Do you confirm? Is that a bug or a limitation of Xapian justifying a issue upstream?

[veloman-yunkan]

245 is the upper limit on the length of a single term for the glass (default) backend of Xapian. An exception is thrown if that limit is exceeded. Titles can be much longer as long as they look like normal text (i.e. contain whitespace and/or punctuation that breaks the text into words/terms).

While experimenting with various abnormal cases in the Suggestion.titleEdgeCases unit-test I observed a few crashes before I found the reason behind the issue reported in this ticket. I was able to reproduce one of those bugs in xapian. The following code results in an error message corrupted size vs. prev_size printed on stderr and a subsequent crash with SIGABRT:

TEST(Suggestion, titleEdgeCases) {
  TempZimArchive tza("testZim");
  zim::writer::Creator creator;
  creator.configIndexing(true, "en");
  creator.startZimCreation(tza.getPath());

  creator.addItem(makeHtmlItem("About", "About"));
  creator.addItem(makeHtmlItem("Trout", "trout"));
  creator.addItem(makeHtmlItem("Without", ""));
  creator.addItem(makeHtmlItem("Stout", "About Rex Stout"));
  creator.addItem(makeHtmlItem("Hangout", "Without a trout"));

  // If the leading space is removed from the title the crash disappears
  auto longTitle = " This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but is provided AS IS, WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, and NON-INFRINGEMENT.  See the USA";

  try {
  creator.addItem(makeHtmlItem("LongTitle", longTitle));
  } catch ( const std::exception& e ) {
    std::cerr << e.what() << std::endl;
  } catch ( const Xapian::Error& e ) {
    std::cerr << e.get_msg() << std::endl;
  }

  creator.addMetadata("Title", "Test zim");
  creator.finishZimCreation();
}