SafeConfigParser is replaced by ConfigParser

[anetchaev]

pydoop/pydoop/__init__.py

Line 147 in 5e62a05

parser = configparser.SafeConfigParser()

using Python 3.12.3:

SafeConfigParser is replaced by ConfigParser

[ilveroluca]

Hi @anetchaev. Thanks for the tips in this and #374. We would be happy to merge a PR with the changes, should you be interested in preparing one.

[anetchaev]

hello,
I can deliver that fix. How to run tests? I am struggling with that.

I use Python 3.12.3 and hadoop 3.4.0 .

executing "python setup.py build" fails with java builds:

src/it/crs4/pydoop/mapreduce/pipes/TaskLog.java:172: error: cannot find symbol
IOUtils.cleanup(LOG, fis);
...

executing "python setup.py build_ext -I/usr/include/ntirpc" is ok

fixing "flake8 -v" requires small fixes in *.py files of examples

after that both are ok:

• python .travis/check_script_template.py -v
• docker build -t crs4/pydoop-docs -f Dockerfile.docs .

BUT running script "./.travis/start_container" gives error:

~/Projects/pydoop/.travis ~/Projects/pydoop
~/Projects/pydoop ~/Projects/pydoop/.travis ~/Projects/pydoop
[+] Building 1.0s (3/3) FINISHED docker:default
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 376B 0.0s
=> ERROR [internal] load metadata for docker.io/crs4/pydoop-base:3.4.0-3.12.3 0.9s
=> [auth] crs4/pydoop-base:pull token for registry-1.docker.io 0.0s

[internal] load metadata for docker.io/crs4/pydoop-base:3.4.0-3.12.3:

---

Dockerfile:4

2 | ARG python_version=3.12.3
3 |
4 | >>> FROM crs4/pydoop-base:${hadoop_version}-${python_version}
5 |
6 | COPY . /build/pydoop

ERROR: failed to solve: crs4/pydoop-base:3.4.0-3.12.3: failed to resolve source metadata for docker.io/crs4/pydoop-base:3.4.0-3.12.3: docker.io/crs4/pydoop-base:3.4.0-3.12.3: not found

Any idea how to fix that?

[ilveroluca]

Hi. The problems seem to be due to the fact that you're trying to use newer versions of Hadoop and Python than those that were available when we stopped maintaining Pydoop. We're no longer maintaining Pydoop, so we're not keeping the code up-to-date. The best bet for a quick fix should be to use the latest docker images available, which should be for Hadoop 3.2.0 and Python 3.7.

To use a newer versions you'd have to build the code for the more recent versions of Hadoop and/or Python, but it's likely that you'd run into issues due to breaking changes introduced by Hadoop and/or Python over the past few years.

[anetchaev]

Hello,
Actually, with small adaptation of *.java file by replacing deprecated interface by new interface i was able to build pydoop using:

HADOOP 3.4.0
PY 3.12.3
JAVA 21.0.5

On weekend I will try to run tests with docker if have spare time. Do I need runing hadoop for executing tests?

[ilveroluca]

Excellent news!

To run tests, we used to use Travis CI, when it was free, so you can find the automation in .travis.yml and .travis/*. I don't think you need a working Hadoop instance. If I recall correctly, a self-contained local docker container image is built and used to launch Hadoop within a container. Then the tests are executed in it. Glancing over .travis.yml suggests that you should be able to run something like this:

export HADOOP_VERSION=3.4.0 TRAVIS_PYTHON_VERSION=3.12.3
python .travis/check_script_template.py -v
./.travis/start_container
./.travis/run_checks
docker stop pydoop

For completeness, you should also run the same thing again with the environment variable LOCAL_FS=true.