Skip to content

About

No description, website, or topics provided.

Resources

Code of conduct

Contributing

Security policy

Stars

8 stars

Watchers

7 watching

Forks

Repository files navigation

https://travis-ci.org/guillotinaweb/guillotina_elasticsearch.svg?branch=master

Elasticsearch integration for guillotina. Supports Elastic search 7.x, 8.x and 9.x.

pip install guillotina_elasticsearch installs the Elasticsearch 9.x Python client and supports Elasticsearch clusters 7.x, 8.x and 9.x.

guillotina_elasticsearch Python Elasticsearch client ES 6.x ES 7.x ES 8.x ES 9.x
6.x aioelasticsearch 0.5/0.6 Yes * Yes No No
7.x elasticsearch-py 7.x No Yes No No
8.x elasticsearch-py 8.x No Yes Yes No
9.x elasticsearch-py 9.x No Yes Yes Yes
  • Elasticsearch 6.x support in the 6.x package line required pinning aioelasticsearch<0.6.0. Current releases do not support Elasticsearch 6.x clusters.

This package uses Elasticsearch compatibility headers so the 9.x Python client can communicate with supported Elasticsearch 7.x and 8.x clusters.

config.yaml can include elasticsearch section

elasticsearch:
  index_name_prefix: "guillotina-"
  connection_settings:
    hosts:
      - "http://127.0.0.1:9200"
    sniffer_timeout: 0.5
    sniff_on_start: true
  security_query_builder: "guillotina_elasticsearch.queries.build_security_query"

Example custom security_query_builder settings:

async def security_query_builder(container, request):
    return {
        'query': {
            'bool': {
                'filter': {
                    'bool': {
                        'should': [{'match': {'access_roles': "foobar"}}],
                        'minimum_should_match': 1
                    }
                }
            }
        }
    }

Setup your python virtual environment for version >=3.10.

# Linux
pip install -e ".[test]"
ES_TEST_VERSION=7 pytest guillotina_elasticsearch/tests
ES_TEST_VERSION=8 pytest guillotina_elasticsearch/tests
ES_TEST_VERSION=9 pytest guillotina_elasticsearch/tests

By default the tests run an ES fixture with version 9. Use ES_TEST_VERSION to select the Elasticsearch major version. Set it to 6, 7, 8 or 9; the test fixture maps those values to the pinned Docker image used by this branch. ES_TEST_VERSION=6 is available for older release-line checks, but this branch does not support Elasticsearch 6.

POST SITE_URL/@catalog

{}

DELETE SITE_URL/@catalog

{}

guillotina_elasticsearch comes with a es-migrate guillotina command:

./bin/g es-migrate

This command will perform a live migration on the index. It does this by performing the reindex on a new index while the other one is still active.

New index and delete requests are performed on both indexes during live migration.

It is also smart about how to migrate, doing a diff on the mapping and only reindexing the fields that changed.

  • Python 3.10 or newer is required.
  • Guillotina 7.0.0 or newer is required.
  • The Python Elasticsearch client dependency is now 9.x (elasticsearch[async]>=9.0.0,<10.0.0).
  • Elasticsearch clusters 7.x, 8.x and 9.x are supported through the Elasticsearch compatibility headers used by the connection factory.
  • Existing configuration files that still use connection_settings.timeout continue to work. The value is translated to the Elasticsearch Python client 9.x request_timeout setting before the client is created.

In this version, the library elasticsearch 7 has been upgraded to elasticsearch 8. There are some changes that need to be taken into account in the settings of old elasticsearch config files.

  • The hosts field in the guillotina's configuration file, need to include the scheme: http or https
  • The sniffer_timeout in the guillotina's configureation file, can not be None
  • The doc_type has been removed. Specifying types in requests is no longer supported.
  • The include_type_name parameter is removed.

The elasticsearch field of the config.yaml file is directly passed to instantiate AsyncElasticsearch. The class definition is the same of the synchronous one, to know how to configure your ES take a look at: https://elasticsearch-py.readthedocs.io/en/v8.12.0/api/elasticsearch.html#elasticsearch.Elasticsearch

  • ES 6 does not have doc types support
  • aioes deprecated
  • IElasticSearchUtility changes:
    • query: doc_type param no longer used
  • IElasticSearchUtility.conn changes:
    • put_mapping
    • put_settings
    • put_alias
    • get: needs doc_type=DOC_TYPE
    • bulk: needs doc_type=DOC_TYPE
    • conn.transport.get_connection(): ._session -> .session, ._base_url -> .base_url
    • conn.transport.get_connection().[method] -> need to include content-type: application/json

If container es (elasticsearch) fails to start when running tests, you should increase max_map_count. command

# Linux
sudo sysctl -w vm.max_map_count=262144

All mappings are flattened when they are applied to Elastic Search.

Elasticsearch doesn't support different mappings for different types.

If there are conflicting mappings, this plugin will choose the mapping in the addon last applications listed which follows the guillotina practice of allowing the last listed application to override the settings of the previously list application

elasticsearch:
  index_name_prefix: "guillotina-"
  connection_settings:
    hosts:
      - "127.0.0.1:9200"
    sniffer_timeout: 0.5
    sniff_on_start: true
    use_ssl: true
    http_auth: admin:admin

About

No description, website, or topics provided.

Resources

Code of conduct

Contributing

Security policy

Stars

8 stars

Watchers

7 watching

Forks

Releases

Sponsor this project

Packages

Used by

Contributors

Languages