Fix lint errors and dependency compatibility issues in test workflow by Copilot · Pull Request #14 · fjelltopp/ckanext-harvest

Copilot · 2025-12-08T15:51:40Z

Fix flake8 E721 lint error in test_ckanharvester.py (using isinstance() instead of type())
Fix pyOpenSSL/cryptography dependency compatibility issue
- Removed deprecated urllib3.contrib.pyopenssl usage that was incompatible with cryptography>=42.0.0
- Updated requirements.txt with pyOpenSSL>=23.2.0 and cryptography>=41.0.0
- Removed pyopenssl mocks from tests
Add harvester initdb command to initialize harvest tables
Waiting for CI to verify all tests pass

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

- Fixed E721 flake8 error by using isinstance() instead of type comparison - Removed --user root from container options which was causing permission issues Co-authored-by: ChasNelson1990 <7795189+ChasNelson1990@users.noreply.github.com>

- Restored --user root option to fix checkout permissions - Added harvester initdb command to initialize harvest tables Co-authored-by: ChasNelson1990 <7795189+ChasNelson1990@users.noreply.github.com>

- Fixed dependency compatibility issue between pyOpenSSL and cryptography - Added version constraints to ensure compatible library versions Co-authored-by: ChasNelson1990 <7795189+ChasNelson1990@users.noreply.github.com>

Copilot · 2025-12-08T16:16:17Z

Just as a heads up, I was blocked by some firewall rules while working on your feedback. Expand below for details.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

https://api.github.com/repos/fjelltopp/ckanext-harvest/actions/jobs/57451562339/logs
- Triggering command: /usr/bin/curl curl -s -H Authorization: Bearer REDACTED (http block)

If you need me to access, download, or install something from one of these locations, you can either:

Configure Actions setup steps to set up my environment, which run before the firewall is enabled
Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

ChasNelson1990 · 2025-12-08T16:20:28Z

@copilot the "Setup Extension" step fails with:

Traceback (most recent call last):
  File "/usr/local/bin/ckan", line 8, in <module>
    sys.exit(ckan())
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 781, in main
    with self.make_context(prog_name, args, **extra) as ctx:
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 700, in make_context
    self.parse_args(ctx, args)
  File "/srv/app/src/ckan/ckan/cli/cli.py", line 116, in parse_args
    result = super(ExtendableGroup, self).parse_args(ctx, args)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1212, in parse_args
    rest = Command.parse_args(self, ctx, args)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1048, in parse_args
    value, args = param.handle_parse_result(ctx, opts, args)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1630, in handle_parse_result
    value = invoke_param_callback(self.callback, ctx, self, value)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 123, in invoke_param_callback
    return callback(ctx, param, value)
  File "/srv/app/src/ckan/ckan/cli/cli.py", line 126, in _init_ckan_config
    _add_ctx_object(ctx, value)
  File "/srv/app/src/ckan/ckan/cli/cli.py", line 135, in _add_ctx_object
    ctx.obj = CtxObject(path)
  File "/srv/app/src/ckan/ckan/cli/cli.py", line 57, in __init__
    self.app = make_app(self.config)
  File "/srv/app/src/ckan/ckan/config/middleware/__init__.py", line 56, in make_app
    load_environment(conf)
  File "/srv/app/src/ckan/ckan/config/environment.py", line 123, in load_environment
    p.load_all()
  File "/srv/app/src/ckan/ckan/plugins/core.py", line 165, in load_all
    load(*plugins)
  File "/srv/app/src/ckan/ckan/plugins/core.py", line 179, in load
    service = _get_service(plugin)
  File "/srv/app/src/ckan/ckan/plugins/core.py", line 281, in _get_service
    return plugin.load()(name=plugin_name)
  File "/usr/local/lib/python3.9/site-packages/pkg_resources/__init__.py", line 2778, in load
    return self.resolve()
  File "/usr/local/lib/python3.9/site-packages/pkg_resources/__init__.py", line 2784, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/__w/ckanext-harvest/ckanext-harvest/ckanext/harvest/harvesters/__init__.py", line 1, in <module>
    from ckanext.harvest.harvesters.ckanharvester import CKANHarvester
  File "/__w/ckanext-harvest/ckanext-harvest/ckanext/harvest/harvesters/ckanharvester.py", line 7, in <module>
    from urllib3.contrib import pyopenssl
  File "/usr/local/lib/python3.9/site-packages/urllib3/contrib/pyopenssl.py", line 50, in <module>
    import OpenSSL.SSL
  File "/usr/local/lib/python3.9/site-packages/OpenSSL/__init__.py", line 8, in <module>
    from OpenSSL import crypto, SSL
  File "/usr/local/lib/python3.9/site-packages/OpenSSL/crypto.py", line 1550, in <module>
    class X509StoreFlags(object):
  File "/usr/local/lib/python3.9/site-packages/OpenSSL/crypto.py", line 1568, in X509StoreFlags
    NOTIFY_POLICY = _lib.X509_V_FLAG_NOTIFY_POLICY
AttributeError: module 'lib' has no attribute 'X509_V_FLAG_NOTIFY_POLICY'

- Updated pyOpenSSL from 18.0.0 to >=23.2.0 in requirements.txt - Added cryptography>=41.0.0,<43.0.0 constraint to requirements.txt - Removed redundant version pins from workflow (now in requirements.txt) Co-authored-by: ChasNelson1990 <7795189+ChasNelson1990@users.noreply.github.com>

Copilot · 2025-12-08T16:23:23Z

@copilot the "Setup Extension" step fails with:
Traceback (most recent call last):

...

The issue was caused by requirements.txt pinning pyOpenSSL==18.0.0, which was overriding the upgraded version. Fixed in commit e75db77 by updating requirements.txt to use compatible versions:

pyOpenSSL>=23.2.0
cryptography>=41.0.0,<43.0.0

This should resolve the Setup Extension traceback.

ChasNelson1990 · 2025-12-08T19:00:17Z

@copilot The Setup Extension step fails with:

Traceback (most recent call last):
  File "/usr/local/bin/ckan", line 8, in <module>
    sys.exit(ckan())
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 781, in main
    with self.make_context(prog_name, args, **extra) as ctx:
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 700, in make_context
    self.parse_args(ctx, args)
  File "/srv/app/src/ckan/ckan/cli/cli.py", line 116, in parse_args
    result = super(ExtendableGroup, self).parse_args(ctx, args)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1212, in parse_args
    rest = Command.parse_args(self, ctx, args)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1048, in parse_args
    value, args = param.handle_parse_result(ctx, opts, args)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1630, in handle_parse_result
    value = invoke_param_callback(self.callback, ctx, self, value)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 123, in invoke_param_callback
    return callback(ctx, param, value)
  File "/srv/app/src/ckan/ckan/cli/cli.py", line 126, in _init_ckan_config
    _add_ctx_object(ctx, value)
  File "/srv/app/src/ckan/ckan/cli/cli.py", line 135, in _add_ctx_object
    ctx.obj = CtxObject(path)
  File "/srv/app/src/ckan/ckan/cli/cli.py", line 57, in __init__
    self.app = make_app(self.config)
  File "/srv/app/src/ckan/ckan/config/middleware/__init__.py", line 56, in make_app
    load_environment(conf)
  File "/srv/app/src/ckan/ckan/config/environment.py", line 123, in load_environment
    p.load_all()
  File "/srv/app/src/ckan/ckan/plugins/core.py", line 165, in load_all
    load(*plugins)
  File "/srv/app/src/ckan/ckan/plugins/core.py", line 179, in load
    service = _get_service(plugin)
  File "/srv/app/src/ckan/ckan/plugins/core.py", line 281, in _get_service
    return plugin.load()(name=plugin_name)
  File "/usr/local/lib/python3.9/site-packages/pkg_resources/__init__.py", line 2778, in load
    return self.resolve()
  File "/usr/local/lib/python3.9/site-packages/pkg_resources/__init__.py", line 2784, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/__w/ckanext-harvest/ckanext-harvest/ckanext/harvest/harvesters/__init__.py", line 1, in <module>
    from ckanext.harvest.harvesters.ckanharvester import CKANHarvester
  File "/__w/ckanext-harvest/ckanext-harvest/ckanext/harvest/harvesters/ckanharvester.py", line 7, in <module>
    from urllib3.contrib import pyopenssl
  File "/usr/local/lib/python3.9/site-packages/urllib3/contrib/pyopenssl.py", line 53, in <module>
    from cryptography.hazmat.backends.openssl.x509 import _Certificate
ModuleNotFoundError: No module named 'cryptography.hazmat.backends.openssl.x509'

- Removed urllib3.contrib.pyopenssl import and inject_into_urllib3() call - This fixes compatibility with cryptography>=42.0.0 - Modern Python and urllib3 have built-in SSL/TLS support Co-authored-by: ChasNelson1990 <7795189+ChasNelson1990@users.noreply.github.com>

Copilot · 2025-12-08T19:04:47Z

@copilot The Setup Extension step fails with:
Traceback (most recent call last):

...

The Setup Extension failure was caused by urllib3.contrib.pyopenssl trying to use a deprecated cryptography API that was removed in cryptography>=42.0.0. Fixed in commit c428e1c by removing the deprecated pyOpenSSL injection code (modern Python has built-in SSL/TLS support).

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

https://api.github.com/repos/fjelltopp/ckanext-harvest/actions/jobs/57454473613
- Triggering command: /usr/bin/curl curl -s REDACTED (http block)

If you need me to access, download, or install something from one of these locations, you can either:

Configure Actions setup steps to set up my environment, which run before the firewall is enabled
Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

ChasNelson1990 · 2025-12-08T19:10:31Z

@copilot the tests fail with:

FAILED ckanext/harvest/tests/harvesters/test_ckanharvester.py::TestCkanHarvester::test_get_content_handles_request_exception - ModuleNotFoundError: No module named 'ckanext.harvest.harvesters.ckanharvester.pyopenssl'; 'ckanext.harvest.harvesters.ckanharvester' is not a package
FAILED ckanext/harvest/tests/harvesters/test_ckanharvester.py::TestCkanHarvester::test_get_content_handles_http_error - ModuleNotFoundError: No module named 'ckanext.harvest.harvesters.ckanharvester.pyopenssl'; 'ckanext.harvest.harvesters.ckanharvester' is not a package
FAILED ckanext/harvest/tests/test_action.py::TestHarvestSourceActionUpdate::test_update - ckan.logic.ValidationError: None - {'source_type': ["Unknown harvester type: test. Registered types: ['test-for-action']"]}
FAILED ckanext/harvest/tests/test_blueprint.py::TestBlueprint::test_new_form_is_rendered - AttributeError: 'TestBlueprint' object has no attribute 'extra_environ'
FAILED ckanext/harvest/tests/test_blueprint.py::TestBlueprint::test_edit_form_is_rendered - AttributeError: 'TestBlueprint' object has no attribute 'extra_environ'
FAILED ckanext/harvest/tests/test_blueprint.py::TestBlueprint::test_source_page_rendered - AttributeError: 'TestBlueprint' object has no attribute 'extra_environ'
FAILED ckanext/harvest/tests/test_blueprint.py::TestBlueprint::test_about_page_rendered - AttributeError: 'TestBlueprint' object has no attribute 'extra_environ'
FAILED ckanext/harvest/tests/test_blueprint.py::TestBlueprint::test_job_show_page_rendered - AttributeError: 'TestBlueprint' object has no attribute 'extra_environ'

@patch

- Removed @patch decorators for pyopenssl.inject_into_urllib3 in test_ckanharvester.py - These mocks are no longer needed since we removed the pyopenssl dependency Co-authored-by: ChasNelson1990 <7795189+ChasNelson1990@users.noreply.github.com>

Copilot · 2025-12-08T19:14:08Z

@copilot the tests fail with:

FAILED ckanext/harvest/tests/harvesters/test_ckanharvester.py::TestCkanHarvester::test_get_content_handles_request_exception - ModuleNotFoundError: No module name...

Fixed the pyopenssl-related test failures in commit 57db72f by removing the obsolete @patch decorators for pyopenssl.inject_into_urllib3.

The remaining failures (test_action.py::test_update and test_blueprint.py tests) appear to be pre-existing issues unrelated to the pyopenssl removal:

test_update: Harvester registration issue ("Unknown harvester type: test")
test_blueprint tests: Missing extra_environ attribute (test setup issue)

These were not introduced by my changes.

ChasNelson1990

LGTM for now

Initial plan

6f71610

Copilot AI assigned Copilot and ChasNelson1990 Dec 8, 2025

Copilot AI mentioned this pull request Dec 8, 2025

feat: update Python versions and update GitHub Actions #13

Closed

Copilot started work on behalf of ChasNelson1990 December 8, 2025 15:52 View session

Copilot AI and others added 3 commits December 8, 2025 15:58

fix: restore --user root and add harvester initdb command

bf6199f

- Restored --user root option to fix checkout permissions - Added harvester initdb command to initialize harvest tables Co-authored-by: ChasNelson1990 <7795189+ChasNelson1990@users.noreply.github.com>

fix: add pyOpenSSL and cryptography version constraints

b8a84e2

- Fixed dependency compatibility issue between pyOpenSSL and cryptography - Added version constraints to ensure compatible library versions Co-authored-by: ChasNelson1990 <7795189+ChasNelson1990@users.noreply.github.com>

Copilot AI changed the title ~~[WIP] Fix lint and test failures in PR for Python updates~~ Fix lint errors and dependency compatibility issues in test workflow Dec 8, 2025

Copilot AI requested a review from ChasNelson1990 December 8, 2025 16:16

Copilot finished work on behalf of ChasNelson1990 December 8, 2025 16:16

Copilot started work on behalf of ChasNelson1990 December 8, 2025 16:20 View session

Copilot started work on behalf of ChasNelson1990 December 8, 2025 16:40 View session

Copilot started work on behalf of ChasNelson1990 December 8, 2025 19:00 View session

Copilot finished work on behalf of ChasNelson1990 December 8, 2025 19:08

Copilot started work on behalf of ChasNelson1990 December 8, 2025 19:11 View session

fix: remove pyopenssl mocks from tests

57db72f

- Removed @patch decorators for pyopenssl.inject_into_urllib3 in test_ckanharvester.py - These mocks are no longer needed since we removed the pyopenssl dependency Co-authored-by: ChasNelson1990 <7795189+ChasNelson1990@users.noreply.github.com>

Copilot finished work on behalf of ChasNelson1990 December 8, 2025 19:15

ChasNelson1990 approved these changes Dec 8, 2025

View reviewed changes

ChasNelson1990 marked this pull request as ready for review December 8, 2025 19:18

ChasNelson1990 merged commit 14ce765 into chas/update-python Dec 8, 2025
4 of 10 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix lint errors and dependency compatibility issues in test workflow#14

Fix lint errors and dependency compatibility issues in test workflow#14
ChasNelson1990 merged 7 commits into
chas/update-pythonfrom
copilot/sub-pr-13

Copilot AI commented Dec 8, 2025 •

edited

Loading

Uh oh!

Copilot AI commented Dec 8, 2025

I tried to connect to the following addresses, but was blocked by firewall rules:

Uh oh!

ChasNelson1990 commented Dec 8, 2025

Uh oh!

Copilot AI commented Dec 8, 2025

Uh oh!

ChasNelson1990 commented Dec 8, 2025

Uh oh!

Copilot AI commented Dec 8, 2025 •

edited

Loading

I tried to connect to the following addresses, but was blocked by firewall rules:

Uh oh!

ChasNelson1990 commented Dec 8, 2025

Uh oh!

Copilot AI commented Dec 8, 2025

Uh oh!

ChasNelson1990 left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

Copilot AI commented Dec 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI commented Dec 8, 2025

I tried to connect to the following addresses, but was blocked by firewall rules:

Uh oh!

ChasNelson1990 commented Dec 8, 2025

Uh oh!

Copilot AI commented Dec 8, 2025

Uh oh!

ChasNelson1990 commented Dec 8, 2025

Uh oh!

Copilot AI commented Dec 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

I tried to connect to the following addresses, but was blocked by firewall rules:

Uh oh!

ChasNelson1990 commented Dec 8, 2025

Uh oh!

Copilot AI commented Dec 8, 2025

Uh oh!

ChasNelson1990 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Copilot AI commented Dec 8, 2025 •

edited

Loading

Copilot AI commented Dec 8, 2025 •

edited

Loading