langdetect.lang_detect_exception.LangDetectException: No features in text.

[Maryia-M]

Hi, I tried to collect the dataset following the instruction.
I got the error after running the command python scripts/get_articles_nytimes.py:
File "scripts/get_articles_nytimes.py", line 220, in retrieve_article data['language'] = detect(text) File "/home/maryia/lib/python3.6/site-packages/langdetect/detector_factory.py", line 130, in detect return detector.detect() File "/home/maryia/lib/python3.6/site-packages/langdetect/detector.py", line 135, in detect probabilities = self.get_probabilities() File "/home/maryia/lib/python3.6/site-packages/langdetect/detector.py", line 142, in get_probabilities self._detect_block() File "/home/maryia/lib/python3.6/site-packages/langdetect/detector.py", line 149, in _detect_block raise LangDetectException(ErrorCode.CantDetectError, 'No features in text.') langdetect.lang_detect_exception.LangDetectException: No features in text.

I tried to fix it using try-catch this way:

    try:
        data['language'] = detect(text)
    except :
        data['language'] = 'undefined'

After that python scripts/get_articles_nytimes.py finished successfully, but the folder "transform-and-tell/data/nytimes/images" is empty, so the next command from instruction did nothing and finished much faster than after 6h (there is a comment in instruction that it takes 6h).

I don't know why I didn't collect the images (because of incorrect fix of langdetect error or not). Could you help me please?

[alasdairtran]

I think the HTML structure of the New York Times might've changed since last year so I will need to update the way beautifulsoup extracts images (mostly likely it will be this function that needs updating). This might take me some time (or feel free to try to update it if you can figure the new HTML structure out).

In the meantime, you can also send me an email (see README) and I'll provide a link to my MongoDB dump.