Fix bg tasks #182

samhotep · 2025-04-16T10:43:33Z

Done

Fixed background tasks not running on startup

QA

QA Local

Check that the demo works
Clone the repo, and start a postgres service (if not already running) using

docker run -d -p 5432:5432 -e POSTGRES_PASSWORD=postgres postgres

load env variables, but unset the REDIS_HOST so that tasks are run directly as sub processes

set -a 
source .env
set +a
unset REDIS_HOST

start the service normally

flask --app webapp.app run --debug

You should see logs from the background tasks. For now, disregard any errors related to invalid credentials.

QA Celery

Clone the repo, and start redis and postgres services (if not already running) using

docker run -d -p 6379:6379 redis
docker run -d -p 5432:5432 -e POSTGRES_PASSWORD=postgres postgres

load env variables and start the service normally, using flask

set -a 
source .env
set +a
flask --app webapp.app run --debug

In another terminal, start the celery worker

 celery -A webapp.app.celery_app worker -B  --loglevel=DEBUG

You should see the tasks and their events in the logs, e.g

  . webapp.scheduled_tasks.load_site_trees
  . webapp.scheduled_tasks.update_jira_statuses
. . .
[2025-04-16 15:32:20,992: INFO/MainProcess] Task webapp.scheduled_tasks.load_site_trees[85635314-9c7b-4459-8da3-e8bcb9a63074] received

Fixes

Fixes WD-20758

webteam-app · 2025-04-16T10:43:37Z

Demo

Jenkins

demos.haus

muhammad-ali-pk · 2025-04-21T13:22:26Z

Following the QA steps, I ran into following issues:

Running celery gives the following error:

Shouldn't it use redis if that's running? Does it also require rabbitmq-server to be running? Things are a bit confusing here.
The app is unable to call save_github_file function, and hence, clone the pages.

muhammad-ali-pk · 2025-04-28T12:45:31Z

@samhotep

The celery logs show the following error:

Traceback (most recent call last):
  File "/home/engr-ali/canonical/projects/content-system/cs.canonical.com/.venv/lib/python3.12/site-packages/celery/worker/consumer/consumer.py", line 659, in on_task_received
    strategy = strategies[type_]
               ~~~~~~~~~~^^^^^^^
KeyError: 'webapp.github.async_save_file'
[2025-04-28 17:41:08,557: ERROR/MainProcess] Received unregistered task of type 'webapp.github.async_save_file'.
The message has been ignored and discarded.

Did you remember to import the module containing this task?
Or maybe you're using relative imports?

Please see
https://docs.celeryq.dev/en/latest/internals/protocol.html
for more information.

The full contents of the message body was:
b'[[], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]' (77b)

The full contents of the message headers:
{'lang': 'py', 'task': 'webapp.github.async_save_file', 'id': '945d78b8-ad86-4c79-9a8b-ce67656c6f11', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'group_index': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '945d78b8-ad86-4c79-9a8b-ce67656c6f11', 'parent_id': None, 'argsrepr': '()', 'kwargsrepr': '{}', 'origin': 'gen167101@x570', 'ignore_result': True, 'replaced_task_nesting': 0, 'stamped_headers': None, 'stamps': {}}

The delivery info for this task is:
{'exchange': '', 'routing_key': 'celery'}
Traceback (most recent call last):
  File "/home/engr-ali/canonical/projects/content-system/cs.canonical.com/.venv/lib/python3.12/site-packages/celery/worker/consumer/consumer.py", line 659, in on_task_received
    strategy = strategies[type_]
               ~~~~~~~~~~^^^^^^^
KeyError: 'webapp.github.async_save_file'

I have the REDIS_HOST and REDIS_PORT setup and the redis container running. Still, the flask app shows its using FileCache and not RedisCache

INFO  [webapp] Connecting to FileCache at /home/engr-ali/canonical/projects/content-system/cs.canonical.com/tree-cache

If the redis is not running, the application shuts down instead of using FileCache

((.venv) ) engr-ali@x570:~/canonical/projects/content-system/cs.canonical.com$ flask --app webapp.app run --debug
 * Tip: There are .env files present. Install python-dotenv to use them.
INFO  [webapp] Connecting to FileCache at /home/engr-ali/canonical/projects/content-system/cs.canonical.com/tree-cache
INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
Traceback (most recent call last):
  File "/home/engr-ali/canonical/projects/content-system/cs.canonical.com/.venv/bin/flask", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/engr-ali/canonical/projects/content-system/cs.canonical.com/.venv/lib/python3.12/site-packages/flask/cli.py", line 1129, in main
    cli.main()
  File "/home/engr-ali/canonical/projects/content-system/cs.canonical.com/.venv/lib/python3.12/site-packages/click/core.py", line 1082, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/engr-ali/canonical/projects/content-system/cs.canonical.com/.venv/lib/python3.12/site-packages/click/core.py", line 1697, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/engr-ali/canonical/projects/content-system/cs.canonical.com/.venv/lib/python3.12/site-packages/click/core.py", line 1443, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/engr-ali/canonical/projects/content-system/cs.canonical.com/.venv/lib/python3.12/site-packages/click/core.py", line 788, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/engr-ali/canonical/projects/content-system/cs.canonical.com/.venv/lib/python3.12/site-packages/click/decorators.py", line 92, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/engr-ali/canonical/projects/content-system/cs.canonical.com/.venv/lib/python3.12/site-packages/click/core.py", line 788, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/engr-ali/canonical/projects/content-system/cs.canonical.com/.venv/lib/python3.12/site-packages/flask/cli.py", line 977, in run_command
    raise e from None
  File "/home/engr-ali/canonical/projects/content-system/cs.canonical.com/.venv/lib/python3.12/site-packages/flask/cli.py", line 961, in run_command
    app: WSGIApplication = info.load_app()  # pyright: ignore
                           ^^^^^^^^^^^^^^^
  File "/home/engr-ali/canonical/projects/content-system/cs.canonical.com/.venv/lib/python3.12/site-packages/flask/cli.py", line 349, in load_app
    app = locate_app(import_name, name)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/engr-ali/canonical/projects/content-system/cs.canonical.com/.venv/lib/python3.12/site-packages/flask/cli.py", line 245, in locate_app
    __import__(module_name)
  File "/home/engr-ali/canonical/projects/content-system/cs.canonical.com/webapp/app.py", line 20, in <module>
    init_scheduled_tasks()
  File "/home/engr-ali/canonical/projects/content-system/cs.canonical.com/webapp/scheduled_tasks.py", line 81, in init_scheduled_tasks
    load_site_trees()
  File "/home/engr-ali/canonical/projects/content-system/cs.canonical.com/webapp/tasks.py", line 40, in wrapper
    task.delay(*args, **kwargs)
  File "/home/engr-ali/canonical/projects/content-system/cs.canonical.com/.venv/lib/python3.12/site-packages/celery/app/task.py", line 444, in delay
    return self.apply_async(args, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/engr-ali/canonical/projects/content-system/cs.canonical.com/.venv/lib/python3.12/site-packages/celery/app/task.py", line 594, in apply_async
    return app.send_task(
           ^^^^^^^^^^^^^^
  File "/home/engr-ali/canonical/projects/content-system/cs.canonical.com/.venv/lib/python3.12/site-packages/celery/app/base.py", line 801, in send_task
    amqp.send_task_message(P, name, message, **options)
  File "/home/engr-ali/canonical/projects/content-system/cs.canonical.com/.venv/lib/python3.12/site-packages/celery/app/amqp.py", line 518, in send_task_message
    ret = producer.publish(
          ^^^^^^^^^^^^^^^^^
  File "/home/engr-ali/canonical/projects/content-system/cs.canonical.com/.venv/lib/python3.12/site-packages/kombu/messaging.py", line 190, in publish
    return _publish(
           ^^^^^^^^^
  File "/home/engr-ali/canonical/projects/content-system/cs.canonical.com/.venv/lib/python3.12/site-packages/kombu/connection.py", line 556, in _ensured
    return fun(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/engr-ali/canonical/projects/content-system/cs.canonical.com/.venv/lib/python3.12/site-packages/kombu/messaging.py", line 200, in _publish
    channel = self.channel
              ^^^^^^^^^^^^
  File "/home/engr-ali/canonical/projects/content-system/cs.canonical.com/.venv/lib/python3.12/site-packages/kombu/messaging.py", line 224, in _get_channel
    channel = self._channel = channel()
                              ^^^^^^^^^
  File "/home/engr-ali/canonical/projects/content-system/cs.canonical.com/.venv/lib/python3.12/site-packages/kombu/utils/functional.py", line 34, in __call__
    value = self.__value__ = self.__contract__()
                             ^^^^^^^^^^^^^^^^^^^
  File "/home/engr-ali/canonical/projects/content-system/cs.canonical.com/.venv/lib/python3.12/site-packages/kombu/messaging.py", line 240, in <lambda>
    channel = ChannelPromise(lambda: connection.default_channel)
                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/engr-ali/canonical/projects/content-system/cs.canonical.com/.venv/lib/python3.12/site-packages/kombu/connection.py", line 957, in default_channel
    self._ensure_connection(**conn_opts)
  File "/home/engr-ali/canonical/projects/content-system/cs.canonical.com/.venv/lib/python3.12/site-packages/kombu/connection.py", line 458, in _ensure_connection
    with ctx():
         ^^^^^
  File "/home/engr-ali/.pyenv/versions/3.12.10/lib/python3.12/contextlib.py", line 158, in __exit__
    self.gen.throw(value)
  File "/home/engr-ali/canonical/projects/content-system/cs.canonical.com/.venv/lib/python3.12/site-packages/kombu/connection.py", line 476, in _reraise_as_library_errors
    raise ConnectionError(str(exc)) from exc
kombu.exceptions.OperationalError: Error 111 connecting to localhost:6379. Connection refused.
Closing background tasks...
((.venv) ) engr-ali@x570:~/canonical/projects/content-system/cs.canonical.com$

samhotep · 2025-04-28T12:48:31Z

@muhammad-ali-pk thanks for the review! Fixes incoming :)

samhotep · 2025-05-05T08:26:19Z

@muhammad-ali-pk for the 3 reported issues:

I've made a fix for this, please test it out.
This log is for the database lock, which uses the file cache to store the lock. I will modify it in a future release to either use redis or the file cache.
If REDIS_HOST is set and REDIS itself isn't working, the solution is to remove the REDIS_HOST flag for now. I'll make it such that we can default to local tasks in this case in another PR, as there are several instances where this can happen

muhammad-ali-pk · 2025-05-05T11:57:02Z

@samhotep I receive the events in celery, and the project directories were created in repositories folder. However, no pages were cloned/saved. (I have postgres and redis running)

Upon checking the postgres, neither the projects nor the webpages were created.

samhotep added Review: QA Needed Review: Code Needed labels Apr 16, 2025

petesfrench self-requested a review April 17, 2025 07:58

muhammad-ali-pk added Review: QA -1 and removed Review: QA Needed labels Apr 21, 2025

samhotep added 16 commits May 5, 2025 15:30

fix: separate scheduled tasks

b7811bf

fix: pass additional process args in order

cbcb3eb

chore: use celery tasks if available

f8a1c4e

fix: remove rabbitmq dependency

44247cd

fix: start scheduled tasks on startup

47751d2

chore: check for redis host as proof for celery task

a01a728

fix: pass correct celery object to tasks

de82347

feat: use shared memory lock for migrations

26460cc

run migrations with app context

43a5cde

feat: use filecache lock

b73c4ac

Update CI.yml

c1e13a4

Update CI.yml

ed9645f

fix: crash on filecache object deletion

9514d5a

fix: pass process variables at runtime, and log errors

c0a7b68

feat: pre-register celery tasks

20842a3

feat: return task result, not task

1d4b5e4

samhotep force-pushed the fix-bg-tasks branch from b0007ec to 1d4b5e4 Compare May 5, 2025 14:45

samhotep added 2 commits May 5, 2025 17:59

feat: use app context in scheduled tasks

260669f

feat: use debug mode on ci

cbe1f49

samhotep and others added 18 commits May 5, 2025 18:36

feat: increase curl delay

140b639

feat: use pathlib for directory scans

490a784

retrigger checks

5476b5a

feat: add ci job without redis

350280f

Update CI.yml

5817661

feat: print errors on close

2b5765c

feat: use avaiable cache

bebb6e2

feat: improve error handling in scheduled task

a335144

Update models.py

a782294

Update playwright.yml

dad4725

Update CI.yml

a92ed33

Update CI.yml

0ee0d65

feat: mark root pages with no children as incomplete

aa9a8ac

Update CI.yml

759f439

Update CI.yml

fd47a72

feat: update CI.yaml

f80d302

Update CI.yml

452cafe

Merge branch 'main' into fix-bg-tasks

fb407a0

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix bg tasks #182

Fix bg tasks #182

samhotep commented Apr 16, 2025 •

edited

Loading

webteam-app commented Apr 16, 2025

muhammad-ali-pk commented Apr 21, 2025

muhammad-ali-pk commented Apr 28, 2025

samhotep commented Apr 28, 2025 •

edited

Loading

samhotep commented May 5, 2025

muhammad-ali-pk commented May 5, 2025

Fix bg tasks #182

Are you sure you want to change the base?

Fix bg tasks #182

Conversation

samhotep commented Apr 16, 2025 • edited Loading

Done

QA

QA Local

QA Celery

Fixes

webteam-app commented Apr 16, 2025

Demo

Jenkins

demos.haus

muhammad-ali-pk commented Apr 21, 2025

muhammad-ali-pk commented Apr 28, 2025

samhotep commented Apr 28, 2025 • edited Loading

samhotep commented May 5, 2025

muhammad-ali-pk commented May 5, 2025

samhotep commented Apr 16, 2025 •

edited

Loading

samhotep commented Apr 28, 2025 •

edited

Loading