ELIT installation problem

[chan-hub]

Can't install ELIT properly - ran into 'illegal instruction(core dump)' upon a test command.

Here's the result of installation script:

chan@cscope37: pip3 install elit --pre
Collecting elit
Using cached https://files.pythonhosted.org/packages/13/2b/a83c0d2578fa2426ae8f58c4dcf387688e6434752b2cd3cca223c5a9a140/elit-0.1.35-py3-
none-any.whl
Collecting mxnet>=1.4.1 (from elit)
Using cached https://files.pythonhosted.org/packages/81/f5/d79b5b40735086ff1100c680703e0f3efc830fa455e268e9e96f3c857e93/mxnet-1.6.0-py2.
py3-none-any.whl
Collecting numpy==1.14.5 (from elit)
Using cached https://files.pythonhosted.org/packages/68/1e/116ad560de97694e2d0c1843a7a0075cc9f49e922454d32f49a80eb6f1f2/numpy-1.14.5-cp3
6-cp36m-manylinux1_x86_64.whl
Collecting gluonnlp==0.6.0 (from elit)
Collecting marisa-trie==0.7.5 (from elit)
Collecting tqdm==4.26.0 (from elit)
Using cached https://files.pythonhosted.org/packages/79/43/19c9fee28110cd47f73e6bc596394337fe9f3e5825b4de402bbf30b3beb5/tqdm-4.26.0-py2.
py3-none-any.whl
Collecting argparse==1.4.0 (from elit)
Using cached https://files.pythonhosted.org/packages/f2/94/3af39d34be01a24a6e65433d19e107099374224905f1e0cc6bbe1fd22a2f/argparse-1.4.0-p
y2.py3-none-any.whl
Collecting gensim==3.6.0 (from elit)
Using cached https://files.pythonhosted.org/packages/27/a4/d10c0acc8528d838cda5eede0ee9c784caa598dbf40bd0911ff8d067a7eb/gensim-3.6.0-cp3
6-cp36m-manylinux1_x86_64.whl
Collecting graphviz<0.9.0,>=0.8.1 (from mxnet>=1.4.1->elit)
Using cached https://files.pythonhosted.org/packages/53/39/4ab213673844e0c004bed8a0781a0721a3f6bb23eb8854ee75c236428892/graphviz-0.8.4-p
y2.py3-none-any.whl
Collecting requests<3,>=2.20.0 (from mxnet>=1.4.1->elit)
Using cached https://files.pythonhosted.org/packages/2d/61/08076519c80041bc0ffa1a8af0cbd3bf3e2b62af10435d269a9d0f40564d/requests-2.27.1-
py2.py3-none-any.whl
Collecting scipy>=0.18.1 (from gensim==3.6.0->elit)
Using cached https://files.pythonhosted.org/packages/c8/89/63171228d5ced148f5ced50305c89e8576ffc695a90b58fe5bb602b910c2/scipy-1.5.4-cp36
-cp36m-manylinux1_x86_64.whl
Collecting smart-open>=1.2.1 (from gensim==3.6.0->elit)
Using cached https://files.pythonhosted.org/packages/09/db/fab79b619923e26cecc5fb460c80f71f99666fe19182d5bb600ec4d6ff10/smart_open-6.0.0
-py3-none-any.whl
Collecting six>=1.5.0 (from gensim==3.6.0->elit)
Using cached https://files.pythonhosted.org/packages/d9/5a/e7c31adbe875f2abbb91bd84cf2dc52d792b5a01506781dbcf25c91daf11/six-1.16.0-py2.p
y3-none-any.whl
Collecting urllib3<1.27,>=1.21.1 (from requests<3,>=2.20.0->mxnet>=1.4.1->elit)
Using cached https://files.pythonhosted.org/packages/ec/03/062e6444ce4baf1eac17a6a0ebfe36bb1ad05e1df0e20b110de59c278498/urllib3-1.26.9-p
y2.py3-none-any.whl
Collecting certifi>=2017.4.17 (from requests<3,>=2.20.0->mxnet>=1.4.1->elit)
Using cached https://files.pythonhosted.org/packages/11/dd/e015f3780f42dd9af62cf0107b44ea1298926627ecd70c17b0e484e95bcd/certifi-2022.5.1
8.1-py3-none-any.whl
Collecting charset-normalizer~=2.0.0; python_version >= "3" (from requests<3,>=2.20.0->mxnet>=1.4.1->elit)
Using cached https://files.pythonhosted.org/packages/06/b3/24afc8868eba069a7f03650ac750a778862dc34941a4bebeb58706715726/charset_normaliz
er-2.0.12-py3-none-any.whl
Collecting idna<4,>=2.5; python_version >= "3" (from requests<3,>=2.20.0->mxnet>=1.4.1->elit)
Using cached https://files.pythonhosted.org/packages/04/a2/d918dcd22354d8958fe113e1a3630137e0fc8b44859ade3063982eacd2a4/idna-3.3-py3-non
e-any.whl
Installing collected packages: graphviz, urllib3, certifi, charset-normalizer, idna, requests, numpy, mxnet, gluonnlp, marisa-trie, tqdm,
argparse, scipy, smart-open, six, gensim, elit
Successfully installed argparse-1.4.0 certifi-2022.5.18.1 charset-normalizer-2.0.12 elit-0.1.35 gensim-3.6.0 gluonnlp-0.6.0 graphviz-0.8.4 idna-3.3 marisa-trie-0.7.5 mxnet-1.6.0 numpy-1.14.5 requests-2.27.1 scipy-1.5.4 six-1.16.0 smart-open-6.0.0 tqdm-4.26.0 urllib3-1.26.9

And here's the test command ran into core dump:
chan@cscope37:~/ELIT/elit-main$ elit dep_parser parse
Illegal instruction (core dumped)

Omitting the 'dep_parser' arg, the command showed help contents:
chan@cscope37:~/ELIT/elit-main$ elit parse
Unrecognized command
usage: elit []

    commands:
        token_tagger: use token tagger
        pos_tagger: run pos tagger
        ner_tagger: run ner tagger
        dep_parser: run syntactic dependency parser
        sdp_parser: run semantic dependency parser
        install: Install model
        download: download pre-trained model or word embedding

positional arguments:
command command to run

optional arguments:
-h, --help show this help message and exit

[hankcs]

Hi, seems that you have an legacy elit installed. The latest version doesn't print commands you gave. What you called is not what you installed just now. To confirm, run

$ elit --help

which should show ELIT-2.0.0-alpha.8 but your system probably would show 1.x.

You can either uninstall the previous version or install elit in a clean new conda env or a virtual env or use colab instead. See https://colab.research.google.com/drive/1vEatxoXQbfLC4Hpy1lLG_DFOIfguPcUE?usp=sharing#scrollTo=yaHSawqjHObr

[chan-hub]

Hi,
This is the very first ELIT installation - there's no old version of ELIT around.
And the command 'elit --help' doesn't show any version info, but the standard help info:

chan@cscope37:~/ELIT/elit-main/docs$ elit --help
usage: elit []
commands:
token_tagger: use token tagger
pos_tagger: run pos tagger
ner_tagger: run ner tagger
dep_parser: run syntactic dependency parser
sdp_parser: run semantic dependency parser
install: Install model
download: download pre-trained model or word embedding
positional arguments:
command command to run
optional arguments:
-h, --help show this help message and exit

The issue may be related to a broken python runtime - originally, this ubuntu 18.04 server comes w/ python 2.7.17 (as default python) and python 3.6. Since ELIT requires python >= 3.7, python 3.8 was installed.

chan@cscope37:~/ELIT/elit-main$ python3.8 -V
Python 3.8.5

Default python is still 2.7 however:
chan@cscope37:~/ELIT/elit-main$ python -V
Python 2.7.17

[hankcs]

Well, then the pip on your system is of Python2. ELIT2.x only supports Python3 so pip downloaded the legacy version. To install 2.x, you need to run pip3 install instead. I'd always suggest to use a conda env which gives you the access to MKL/CUDA accerlation.

[chan-hub]

The installation log shows that pip3 was used instead of pip, but the end result is 'elit-0.1.35'. The pip3 version is 3.6, however, that might be the reason grabbing older version ? Tried re-installing w/ pip3.8 but then ran into http connection problem. Will try later to see how it goes. Thanks for quick respond.

chan@cscope37: pip3 install elit --pre
Collecting elit
Using cached https://files.pythonhosted.org/packages/13/2b/a83c0d2578fa2426ae8f58c4dcf387688e6434752b2cd3cca223c5a9a140/elit-0.1.35-py3-none-any.whl
Collecting mxnet>=1.4.1 (from elit)
Using cached https://files.pythonhosted.org/packages/81/f5/d79b5b40735086ff1100c680703e0f3efc830fa455e268e9e96f3c857e93/mxnet-1.6.0-py2.py3-none-any.whl
...
Installing collected packages: graphviz, urllib3, certifi, charset-normalizer, idna, requests, numpy, mxnet, gluonnlp, marisa-trie, tqdm, argparse, scipy, smart-open, six, gensim, elit
Successfully installed argparse-1.4.0 certifi-2022.5.18.1 charset-normalizer-2.0.12 elit-0.1.35 gensim-3.6.0 gluonnlp-0.6.0 graphviz-0.8.4 idna-3.3 marisa-trie-0.7.5 mxnet-1.6.0 numpy-1.14.5 requests-2.27.1 scipy-1.5.4 six-1.16.0 smart-open-6.0.0 tqdm-4.26.0 urllib3-1.26.9

[chan-hub]

After installing python3.7 w/ its pip, still ran into an error - looks like python exception:

chan@cscope6:~/project/ELIT/elit-main$ pip install elit --pre
Defaulting to user installation because normal site-packages is not writeable
Collecting elit
Downloading elit-2.0.0a8-py3-none-any.whl (485 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 485.8/485.8 kB 2.0 MB/s eta 0:00:00
Collecting toposort==1.5
Downloading toposort-1.5-py2.py3-none-any.whl (7.6 kB)
Collecting unofficial-stog==0.0.21
Downloading unofficial_stog-0.0.21.tar.gz (220 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 220.9/220.9 kB 1.8 MB/s eta 0:00:00
Preparing metadata (setup.py) ... done
Collecting penman==1.2.1
Downloading Penman-1.2.1-py3-none-any.whl (43 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 43.3/43.3 kB 625.8 kB/s eta 0:00:00
Collecting uvicorn==0.13.1
Downloading uvicorn-0.13.1-py3-none-any.whl (45 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 45.5/45.5 kB 711.5 kB/s eta 0:00:00
Collecting fastapi==0.65.2
Downloading fastapi-0.65.2-py3-none-any.whl (51 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 51.2/51.2 kB 866.7 kB/s eta 0:00:00
Collecting transformers==4.6.1
Downloading transformers-4.6.1-py3-none-any.whl (2.2 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.2/2.2 MB 2.4 MB/s eta 0:00:00
Collecting torch<1.9.0,>=1.6.0
Downloading torch-1.8.1-cp37-cp37m-manylinux1_x86_64.whl (804.1 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 804.1/804.1 MB 484.4 kB/s eta 0:00:00
Collecting termcolor==1.1.0
Downloading termcolor-1.1.0.tar.gz (3.9 kB)
Preparing metadata (setup.py) ... done
Collecting sentencepiece==0.1.95
Downloading sentencepiece-0.1.95-cp37-cp37m-manylinux2014_x86_64.whl (1.2 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 2.5 MB/s eta 0:00:00
Collecting pynvml==8.0.4
Downloading pynvml-8.0.4-py3-none-any.whl (36 kB)
Collecting scipy==1.5.4
Downloading scipy-1.5.4-cp37-cp37m-manylinux1_x86_64.whl (25.9 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 25.9/25.9 MB 3.7 MB/s eta 0:00:00
Collecting phrasetree==0.0.4
Downloading phrasetree-0.0.4.tar.gz (43 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 44.0/44.0 kB 755.8 kB/s eta 0:00:00
Preparing metadata (setup.py) ... done
Collecting editdistance==0.5.3
Downloading editdistance-0.5.3-cp37-cp37m-manylinux1_x86_64.whl (179 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 179.2/179.2 kB 778.1 kB/s eta 0:00:00
Collecting alnlp==1.0.0rc27
Downloading alnlp-1.0.0rc27.tar.gz (130 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 130.8/130.8 kB 1.7 MB/s eta 0:00:00
Preparing metadata (setup.py) ... done
Collecting pydantic!=1.7,!=1.7.1,!=1.7.2,!=1.7.3,!=1.8,!=1.8.1,<2.0.0,>=1.6.2
Downloading pydantic-1.9.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (11.1 MB)
━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.2/11.1 MB 672.6 kB/s eta 0:00:12
ERROR: Exception:
Traceback (most recent call last):
File "/home/chan/.local/lib/python3.7/site-packages/pip/_vendor/urllib3/response.py", line 435, in _error_catcher
yield
File "/home/chan/.local/lib/python3.7/site-packages/pip/_vendor/urllib3/response.py", line 516, in read
data = self._fp.read(amt) if not fp_closed else b""
File "/home/chan/.local/lib/python3.7/site-packages/pip/_vendor/cachecontrol/filewrapper.py", line 90, in read
data = self.__fp.read(amt)
File "/usr/lib/python3.7/http/client.py", line 461, in read
n = self.readinto(b)
File "/usr/lib/python3.7/http/client.py", line 505, in readinto
n = self.fp.readinto(b)
File "/usr/lib/python3.7/socket.py", line 589, in readinto
return self._sock.recv_into(b)
File "/usr/lib/python3.7/ssl.py", line 1071, in recv_into
return self.read(nbytes, buffer)
File "/usr/lib/python3.7/ssl.py", line 929, in read
return self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/chan/.local/lib/python3.7/site-packages/pip/_internal/cli/base_command.py", line 167, in exc_logging_wrapper
status = run_func(*args)
File "/home/chan/.local/lib/python3.7/site-packages/pip/_internal/cli/req_command.py", line 205, in wrapper
return func(self, options, args)
File "/home/chan/.local/lib/python3.7/site-packages/pip/_internal/commands/install.py", line 342, in run
reqs, check_supported_wheels=not options.target_dir
File "/home/chan/.local/lib/python3.7/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 95, in resolve
collected.requirements, max_rounds=try_to_avoid_resolution_too_deep
File "/home/chan/.local/lib/python3.7/site-packages/pip/_vendor/resolvelib/resolvers.py", line 481, in resolve
state = resolution.resolve(requirements, max_rounds=max_rounds)
File "/home/chan/.local/lib/python3.7/site-packages/pip/_vendor/resolvelib/resolvers.py", line 373, in resolve
failure_causes = self._attempt_to_pin_criterion(name)
File "/home/chan/.local/lib/python3.7/site-packages/pip/_vendor/resolvelib/resolvers.py", line 213, in _attempt_to_pin_criterion
criteria = self._get_updated_criteria(candidate)
File "/home/chan/.local/lib/python3.7/site-packages/pip/_vendor/resolvelib/resolvers.py", line 204, in _get_updated_criteria
self._add_to_criteria(criteria, requirement, parent=candidate)
File "/home/chan/.local/lib/python3.7/site-packages/pip/_vendor/resolvelib/resolvers.py", line 172, in _add_to_criteria
if not criterion.candidates:
File "/home/chan/.local/lib/python3.7/site-packages/pip/_vendor/resolvelib/structs.py", line 151, in bool
return bool(self._sequence)
File "/home/chan/.local/lib/python3.7/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 155, in bool
return any(self)
File "/home/chan/.local/lib/python3.7/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 143, in
return (c for c in iterator if id(c) not in self._incompatible_ids)
File "/home/chan/.local/lib/python3.7/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 47, in _iter_built
candidate = func()
File "/home/chan/.local/lib/python3.7/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 220, in _make_candidate_from_link
version=version,
File "/home/chan/.local/lib/python3.7/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 297, in init
version=version,
File "/home/chan/.local/lib/python3.7/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 161, in init
self.dist = self._prepare()
File "/home/chan/.local/lib/python3.7/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 230, in _prepare
dist = self._prepare_distribution()
File "/home/chan/.local/lib/python3.7/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 302, in _prepare_distribution
return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True)
File "/home/chan/.local/lib/python3.7/site-packages/pip/_internal/operations/prepare.py", line 428, in prepare_linked_requirement
return self._prepare_linked_requirement(req, parallel_builds)
File "/home/chan/.local/lib/python3.7/site-packages/pip/_internal/operations/prepare.py", line 479, in _prepare_linked_requirement
hashes,
File "/home/chan/.local/lib/python3.7/site-packages/pip/_internal/operations/prepare.py", line 159, in unpack_url
hashes=hashes,
File "/home/chan/.local/lib/python3.7/site-packages/pip/_internal/operations/prepare.py", line 96, in get_http_url
from_path, content_type = download(link, temp_dir.path)
File "/home/chan/.local/lib/python3.7/site-packages/pip/_internal/network/download.py", line 146, in call
for chunk in chunks:
File "/home/chan/.local/lib/python3.7/site-packages/pip/_internal/cli/progress_bars.py", line 53, in _rich_progress_bar
for chunk in iterable:
File "/home/chan/.local/lib/python3.7/site-packages/pip/_internal/network/utils.py", line 87, in response_chunks
decode_content=False,
File "/home/chan/.local/lib/python3.7/site-packages/pip/_vendor/urllib3/response.py", line 573, in stream
data = self.read(amt=amt, decode_content=decode_content)
File "/home/chan/.local/lib/python3.7/site-packages/pip/_vendor/urllib3/response.py", line 538, in read
raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
File "/usr/lib/python3.7/contextlib.py", line 130, in exit
self.gen.throw(type, value, traceback)
File "/home/chan/.local/lib/python3.7/site-packages/pip/_vendor/urllib3/response.py", line 440, in _error_catcher
raise ReadTimeoutError(self._pool, None, "Read timed out.")
pip._vendor.urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Read timed out.

Could you help on this ?

Thanks !

[hankcs]

Hi, the new error is due to your network failed to reach files.pythonhosted.org, which is a public site not maintained by us. I'd recommend you to either retry, or to use a pip mirror/VPN, depending on your geo location.

[chan-hub]

Hi,
Yes, the server LAN gateway wasn't stable - caused downloading errors.
Migrating to another server w/ solid WAN connection solved the network problem - successfully installed & tested:
Downloading: 43%|██████████████████████████████████████▌ | 587k/1.36M [00:00<00:00, 1.96MB/s]Downloading: 72%|███████████████████████████████████████████████████████████████▊ | 972k/1.36M [00:00<00:00, 2.62MB/s]Downloading: 100%|████████████████████████████████████████████████████████████████████████████████████████| 1.36M/1.36M [00:00<00:00, 2.36MB/s]
Building model

Emory NLP is a research lab in Atlanta.
[W NNPACK.cpp:80] Could not initialize NNPACK! Reason: Unsupported hardware.
{
"lem": [
["emory", "nlp", "be", "a", "research", "lab", "in", "atlanta", "."]
],
"pos": [
["NNP", "NNP", "VBZ", "DT", "NN", "NN", "IN", "NNP", "."]
],
"ner": [
[["ORG", 0, 2, "Emory NLP"], ["GPE", 7, 8, "Atlanta"]]
],
"srl": [
[[["ARG1", 0, 2, "Emory NLP"], ["PRED", 2, 3, "is"], ["ARG2", 3, 8, "a research lab in Atlanta"]]]
],

Two more questions:

1. Can the '[W NNPACK.cpp:80] Could not initialize NNPACK! Reason: Unsupported hardware.' msg be ignored ?
   Look like a C preprocessor complains about a macro line - the server is Ubuntu 18.04 running on Intel CPU

2. How the ELIT parsing result be represented in node list form, where each node captures the form, lemma, PoS,
   dependency label, etc.

Thanks so much for your help !

[hankcs]

1. This is raised by pytorch. Maybe your hardware is just too old. You can build from source to get rid of it: https://discuss.pytorch.org/t/bug-w-nnpack-cpp-80-could-not-initialize-nnpack-reason-unsupported-hardware/107518
2. Format per each annotation can be found in links in the README.md.