Merge PR #2871 into 17.0

Signed-off-by thomaspaulb

Author: OCA-git-bot

bus_alt_connection/README.rst

@@ -0,0 +1,190 @@
+==================
+Bus Alt Connection
+==================
+
+.. 
+   !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
+   !! This file is generated by oca-gen-addon-readme !!
+   !! changes will be overwritten.                   !!
+   !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
+   !! source digest: sha256:ab7e1b9d5721f8cb27f93c58ed77e5034bc0099105ac6d5097f1bdc74a4e6973
+   !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
+
+.. |badge1| image:: https://img.shields.io/badge/maturity-Beta-yellow.png
+    :target: https://odoo-community.org/page/development-status
+    :alt: Beta
+.. |badge2| image:: https://img.shields.io/badge/licence-AGPL--3-blue.png
+    :target: http://www.gnu.org/licenses/agpl-3.0-standalone.html
+    :alt: License: AGPL-3
+.. |badge3| image:: https://img.shields.io/badge/github-OCA%2Fserver--tools-lightgray.png?logo=github
+    :target: https://github.com/OCA/server-tools/tree/16.0/bus_alt_connection
+    :alt: OCA/server-tools
+.. |badge4| image:: https://img.shields.io/badge/weblate-Translate%20me-F47D42.png
+    :target: https://translation.odoo-community.org/projects/server-tools-16-0/server-tools-16-0-bus_alt_connection
+    :alt: Translate me on Weblate
+.. |badge5| image:: https://img.shields.io/badge/runboat-Try%20me-875A7B.png
+    :target: https://runboat.odoo-community.org/builds?repo=OCA/server-tools&target_branch=16.0
+    :alt: Try me on Runboat
+
+|badge1| |badge2| |badge3| |badge4| |badge5|
+
+This module makes it possible to use PgBouncer_ as a connection pooler
+for odoo.
+
+.. _PgBouncer: https://pgbouncer.github.io/
+
+Why isn't odoo's connection pooling good enough?
+================================================
+
+Odoo's builtin connection pooling works at process level: each Odoo process
+has its own ConnectionPool_, limited to ``db_maxconn``.
+
+It does the job of re-using open connections available in the pool.
+But it never closes these connections, `unless reaching db_maxconn`_.
+
+.. _ConnectionPool: https://github.com/odoo/odoo/blob/12.0/odoo/sql_db.py#L525
+.. _`unless reaching db_maxconn`: https://github.com/odoo/odoo/blob/12.0/odoo/sql_db.py#L593
+
+In practice, we observe that each odoo worker will end up
+with up to 3 open connection in its pool.
+With 10 http workers, that's up to 30 connection continuously open just
+for one single instance.
+
+Here comes PgBouncer
+====================
+
+PgBouncer will help to limit this number of open connections,
+by sharing a pool of connections at the instance level, between
+all workers. Odoo workers will still have up to 3 open connections,
+but these will be connections to PgBouncer, that on its side will
+close unnecessary connections to pg.
+
+This has proven to help performances on Odoo deployments with
+multiple instances.
+
+It allows you to define how resources should be shared,
+according to your priorities, e.g. :
+
+* key odoo instance on host A can open up to 30 connections
+* while odoo instance on host B, dedicated to reports,
+  can open up to 10 connections only
+
+And most importantly, it helps you to ensure that
+``max_connections`` will never be reached on pg server side.
+
+
+Why is this module needed?
+==========================
+
+When configuring PgBouncer, you can choose between 2 transaction pooling modes:
+
+* `pool_mode = session`
+* `pool_mode = transaction`
+
+If we choose `pool_mode = session`, then one server connection will be tied
+to a given odoo process until its death, which is exactly what we're trying
+to change. Thus, to release the server connection once the transaction is
+complete, we use `pool_mode = transaction`.
+
+This works fine, except for Odoo's longpolling features that relies
+on the `LISTEN/NOTIFY`_ mechanism from pg, which is `not compatible`_ with that
+mode.
+
+.. _`LISTEN/NOTIFY`: https://www.postgresql.org/docs/9.6/static/sql-notify.html
+.. _`not compatible`: https://wiki.postgresql.org/wiki/PgBouncer
+
+
+To be more precise, `NOTIFY` statements are properly transfered by PgBouncer
+in that mode; only the `LISTEN` statement isn't (because it needs to keep the
+server connection open).
+
+So for the unique "listening" connection per instance that requires this
+statement (here_), we need odoo to connect directly to the pg server, bypassing
+PgBouncer.
+
+That's what this module implements, by overriding the relevant method
+of the Dispatcher_.
+
+.. _here: https://github.com/odoo/odoo/blob/12.0/addons/bus/models/bus.py#L166
+.. _Dispatcher: https://github.com/odoo/odoo/blob/12.0/addons/bus/models/bus.py#L105
+
+**Table of contents**
+
+.. contents::
+   :local:
+
+Installation
+============
+
+You don't need to install this module in the database(s) to enable it.
+
+But you need to load it server-wide:
+
+* By starting Odoo with ``--load=web,bus_alt_connection``
+
+* Or by updating its configuration file:
+
+.. code-block:: ini
+
+  [options]
+  (...)
+  server_wide_modules = web,bus_alt_connection
+
+Configuration
+=============
+
+You need to define how to connect directly to the database:
+
+* Either by defining environment variables:
+
+    - ``IMDISPATCHER_DB_HOST=db-01``
+    - ``IMDISPATCHER_DB_PORT=5432``
+
+* Or in Odoo's configuration file:
+
+.. code-block:: ini
+
+  [options]
+  (...)
+  imdispatcher_db_host = db-01
+  imdispatcher_db_port = 5432
+
+Bug Tracker
+===========
+
+Bugs are tracked on `GitHub Issues <https://github.com/OCA/server-tools/issues>`_.
+In case of trouble, please check there if your issue has already been reported.
+If you spotted it first, help us to smash it by providing a detailed and welcomed
+`feedback <https://github.com/OCA/server-tools/issues/new?body=module:%20bus_alt_connection%0Aversion:%2016.0%0A%0A**Steps%20to%20reproduce**%0A-%20...%0A%0A**Current%20behavior**%0A%0A**Expected%20behavior**>`_.
+
+Do not contact contributors directly about support or help with technical issues.
+
+Credits
+=======
+
+Authors
+~~~~~~~
+
+* Trobz
+
+Contributors
+~~~~~~~~~~~~
+
+* Nils Hamerlinck <nils@trobz.com>
+
+Maintainers
+~~~~~~~~~~~
+
+This module is maintained by the OCA.
+
+.. image:: https://odoo-community.org/logo.png
+   :alt: Odoo Community Association
+   :target: https://odoo-community.org
+
+OCA, or the Odoo Community Association, is a nonprofit organization whose
+mission is to support the collaborative development of Odoo features and
+promote its widespread use.
+
+This module is part of the `OCA/server-tools <https://github.com/OCA/server-tools/tree/16.0/bus_alt_connection>`_ project on GitHub.
+
+You are welcome to contribute. To learn how please visit https://odoo-community.org/page/Contribute.

bus_alt_connection/__init__.py

@@ -0,0 +1,3 @@
+# Copyright 2019 Trobz <https://trobz.com>
+# License AGPL-3.0 or later (http://www.gnu.org/licenses/agpl).
+from . import models

bus_alt_connection/__manifest__.py

@@ -0,0 +1,15 @@
+# Copyright 2019 Trobz <https://trobz.com>
+# License AGPL-3.0 or later (http://www.gnu.org/licenses/agpl).
+
+{
+    "name": "Bus Alt Connection",
+    "summary": "Needed when using PgBouncer as a connection pooler",
+    "version": "17.0.1.0.0",
+    "author": "Trobz,Odoo Community Association (OCA)",
+    "website": "https://github.com/OCA/server-tools",
+    "category": "Extra Tools",
+    "license": "AGPL-3",
+    "depends": ["bus"],
+    "installable": True,
+    "auto_install": False,
+}

bus_alt_connection/i18n/bus_alt_connection.pot

@@ -0,0 +1,13 @@
+# Translation of Odoo Server.
+# This file contains the translation of the following modules:
+#
+msgid ""
+msgstr ""
+"Project-Id-Version: Odoo Server 14.0\n"
+"Report-Msgid-Bugs-To: \n"
+"Last-Translator: \n"
+"Language-Team: \n"
+"MIME-Version: 1.0\n"
+"Content-Type: text/plain; charset=UTF-8\n"
+"Content-Transfer-Encoding: \n"
+"Plural-Forms: \n"

bus_alt_connection/models/__init__.py

@@ -0,0 +1,3 @@
+# Copyright 2019 Trobz <https://trobz.com>
+# License AGPL-3.0 or later (http://www.gnu.org/licenses/agpl).
+from . import bus

bus_alt_connection/models/bus.py

@@ -0,0 +1,67 @@
+# Copyright 2019 Trobz <https://trobz.com>
+# License AGPL-3.0 or later (http://www.gnu.org/licenses/agpl).
+
+import json
+import logging
+import os
+import selectors
+
+import psycopg2
+
+import odoo
+from odoo.tools import config
+
+import odoo.addons.bus.models.bus
+from odoo.addons.bus.models.bus import TIMEOUT, hashable, stop_event
+
+_logger = logging.getLogger(__name__)
+
+
+def _connection_info_for(db_name):
+    db_or_uri, connection_info = odoo.sql_db.connection_info_for(db_name)
+
+    for p in ("host", "port"):
+        cfg = os.environ.get("ODOO_IMDISPATCHER_DB_%s" % p.upper()) or config.get(
+            "imdispatcher_db_" + p
+        )
+        if cfg:
+            connection_info[p] = cfg
+    return connection_info
+
+
+class ImDispatch(odoo.addons.bus.models.bus.ImDispatch):
+    def loop(self):
+        """Dispatch postgres notifications to the relevant
+        polling threads/greenlets"""
+        connection_info = _connection_info_for("postgres")
+        _logger.info(
+            "Bus.loop listen imbus on db postgres " "(via %(host)s:%(port)s)",
+            connection_info,
+        )
+        conn = psycopg2.connect(**connection_info)
+        with conn.cursor() as cr, selectors.DefaultSelector() as sel:
+            cr.execute("listen imbus")
+            conn.commit()
+            sel.register(conn, selectors.EVENT_READ)
+            while not stop_event.is_set():
+                if sel.select(TIMEOUT):
+                    conn.poll()
+                    channels = []
+                    while conn.notifies:
+                        channels.extend(json.loads(conn.notifies.pop().payload))
+                    # relay notifications to websockets that have
+                    # subscribed to the corresponding channels.
+                    websockets = set()
+                    for channel in channels:
+                        websockets.update(
+                            self._channels_to_ws.get(hashable(channel), [])
+                        )
+                    for websocket in websockets:
+                        websocket.trigger_notification_dispatching()
+
+
+odoo.addons.bus.models.bus.ImDispatch = ImDispatch
+dispatch = ImDispatch()
+odoo.addons.bus.models.bus.dispatch = dispatch
+odoo.addons.bus.models.ir_websocket.dispatch = dispatch
+odoo.addons.bus.websocket.dispatch = dispatch

bus_alt_connection/pyproject.toml

@@ -0,0 +1,3 @@
+[build-system]
+requires = ["whool"]
+build-backend = "whool.buildapi"

bus_alt_connection/readme/CONFIGURE.md

@@ -0,0 +1,15 @@
+You need to define how to connect directly to the database:
+
+- Either by defining environment variables:
+
+  > - `IMDISPATCHER_DB_HOST=db-01`
+  > - `IMDISPATCHER_DB_PORT=5432`
+
+- Or in Odoo's configuration file:
+
+``` ini
+[options]
+(...)
+imdispatcher_db_host = db-01
+imdispatcher_db_port = 5432
+```

bus_alt_connection/readme/CONFIGURE.rst

@@ -0,0 +1,15 @@
+You need to define how to connect directly to the database:
+
+* Either by defining environment variables:
+
+    - ``IMDISPATCHER_DB_HOST=db-01``
+    - ``IMDISPATCHER_DB_PORT=5432``
+
+* Or in Odoo's configuration file:
+
+.. code-block:: ini
+
+  [options]
+  (...)
+  imdispatcher_db_host = db-01
+  imdispatcher_db_port = 5432

bus_alt_connection/readme/CONTRIBUTORS.md

@@ -0,0 +1 @@
+- Nils Hamerlinck \<<nils@trobz.com>\>