Octopus

Application for managing the people and organizational structure of the Pirate Party, and potentially other parties or volunteer-based organizations.

Under the hood

Project structure

.
├── api             = app for GraphQL API
├── applications    = app for membership applications, etc.
├── comments        = app for comments in admin
├── directory       = app for directory web UI
├── elections       = app for organizing elections and candidates
├── emails          = app for managing email mailboxes and aliases
├── messages        = app for group messaging by emails, SMS, Zulip
├── notifications   = app for notifications by email, etc.
├── octopus         = app for main admin UI
├── onboarding      = app for onboardin web UI
├── organization    = app for teams and groups
├── payments        = app for membership payments
├── people          = app for people management
├── profiles        = app for personal CV's
├── sync            = app for synchronizations with other systems
├── users           = app for custom user model with SSO, etc.
├── pj_octopus      = Django project for main admin UI
├── pj_onboarding   = Django project for onboardin web UI
├── pj_directory    = Django project for directory web UI
└── pj_api          = Django project for GraphQL API

Django websites consist of applications and projects. Applications are functional modules that serve roles ranging from libraries to web interfaces. Ideally, they are reusable. Projects are sets of installation configurations for a group of applications. In this case, we have three projects for three web interfaces that share the same applications. And another project for serving API.

GraphQL API

The API is built on the GraphQL standard. For pagination, the Relay specification is implemented.

Authorization

To access non-public information in the API, you must include an authorization token in the HTTP header of your request. For example:

Authorization: Bearer 5aMOd4a455JHJKhda4789dsapqPOj

Deployment

Configuration

Since we have four Django projects with configurations, it may be necessary to specify which one to use. This is done through the DJANGO_SETTINGS_MODULE variable, which defaults everywhere to pj_octopus. If you want to run manage.py for another project, you can do it like this:

$ DJANGO_SETTINGS_MODULE="pj_onboarding.settings.dev" python manage.py runserver

You need to set the following environment variables:

variable	default	description
DATABASE_URL		DSN for the database (e.g. postgres://user:pass@localhost:5342/octopus)
OIDC_RP_REALM_URL		OpenID server realm URL (e.g. http://localhost:8080/auth/realms/master/)
OIDC_RP_CLIENT_ID		OpenID Client ID
OIDC_RP_CLIENT_SECRET		OpenID Client Secret
CELERY_BROKER_URL		Celery Broker (e.g. redis://localhost)

In production, you must also set:

variable	default	description
DJANGO_SECRET_KEY		secret encryption key
DJANGO_ALLOWED_HOSTS		allowed hosts (comma-separated list)
DJANGO_SETTINGS_MODULE	pj_octopus.settings.prod	production settings
EMAIL_URL	consolemail://	DSN for the email server (e.g. smtp://user:pass@localhost:25)

Miscellaneous:

variable	default	description
OCTOPUS_ENV	prod	prod/test/dev environment where Octopus runs
SENTRY_DSN		if set, crashes are reported to Sentry
FORUM_DB_DSN		DSN for the forum database
NEXTCLOUD_DB_DSN		DSN for the Nextcloud database
KEYCLOAK_DB_DSN		DSN for the Keycloak database
KEYCLOAK_API_URL		URL to the Keycloak API
KEYCLOAK_API_USER		Keycloak API user
KEYCLOAK_API_PASS		Keycloak API password
KEYCLOAK_API_REALM		Keycloak API realm name
KEYCLOAK_API_USER_REALM		Keycloak user realm name (if different)
ALLOW_CHANGES_IN_REMOTE_SYSTEMS	False	can make changes in remote systems? (Keycloak, Forum, etc.)
FIO_TOKEN		Fio API token for membership payments
BANNED_REGISTRATION_EMAILS	pirati.cz	banned domains for registration emails (comma-separated)
OCTOPUS_MATOMO_ID		Matomo ID for Octopus
ONBOARDING_MATOMO_ID		Matomo ID for Onboarding
DIRECTORY_MATOMO_ID		Matomo ID for Directory
OCTOPUS_URL	https://chobotnice.pirati.cz	URL where Octopus runs
ONBOARDING_URL	https://nalodeni.pirati.cz	URL where Onboarding runs
DIRECTORY_URL	https://lide.pirati.cz	URL where Directory runs
DEFAULT_FROM_EMAIL	Členství <[email protected]>	sender address for Octopus emails
NOREPLY_FROM_EMAIL	[email protected]	no-reply sender address
MESSAGES_FROM_EMAIL	[email protected]	sender address for messages
MESSAGES_FROM_NAME	Zprávy od Pirátů	sender name for messages
VOTING_FROM_EMAIL	[email protected]	sender address for voting emails
VOTING_FROM_NAME	Pirates - Helios Voting	sender name for voting emails
SEND_NOTIFICATIONS	True	send notification messages/emails
SEND_ONBOARDING_EMAILS	True	send onboarding (registration) emails
SEND_MESSAGES	True	send messages
ZULIP_DB_DSN		DSN for the Zulip database
ZULIP_API_SITE	https://zulip.pirati.cz	URL where Zulip runs
ZULIP_API_KEY		API key for Octopus Zulip bot
ZULIP_API_EMAIL		email of Octopus bot
ZULIP_API_KEY_MESSAGES		API key for message bot
ZULIP_API_EMAIL_MESSAGES		email for message bot
ZULIP_API_KEY_VOTING		API key for voting bot
ZULIP_API_EMAIL_VOTING		email for voting bot
ZULIP_REALM_ID	2	Zulip Realm ID
ZULIP_DEFAULT_STREAMS		default Zulip channels for new users (IDs comma-separated)
ZULIP_DEFAULT_STREAMS_MEMBER		default Zulip channels for members (IDs comma-separated)
ZULIP_DEFAULT_STREAMS_SUPPORTER		default Zulip channels for supporters (IDs comma-separated)
MAILTRAIN_DB_DSN		DSN for the Mailtrain database
MAILTRAIN_API_URL		URL to the Mailtrain API
MAILTRAIN_API_TOKEN		API token for Mailtrain
NALODENI_DB_DSN		DSN for the Onboarding database
EMPTY_ALIAS_EMAIL	[email protected]	target for alias-type email accounts without defined destinations
ROUNDCUBE_DB_DSN		DSN for the Roundcube database
HELIOS_URL	https://helios.pirati.cz	URL to Helios
HELIOS_API_TOKEN		API token for Helios
PZ_API_URL	https://z.pirati.cz/api	URL to the Pirate URL shortener API
PZ_API_SECRET_FILE		path to Docker secret with the API key for the URL shortener
PZ_SHORT_URL	https://pi2.cz/	base for shortened URLs
REDMINE_URL	https://redmine.pirati.cz	URL to Redmine
REDMINE_API_KEY		API key for Redmine
REDMINE_DB_DSN		DSN for the Redmine database

Management commands

To create or update permission groups, use:

• load_auth_groups

This command is included in run.sh for running the app in a Docker container.

You need to run Django manage.py commands via CRON in the background:

• clearsessions - deletes expired sessions (daily to weekly)
• clean_duplicate_history --auto - removes duplicate history records, see django-simple-history docs
• background_tasks - various background jobs (cleanup, retrying failed emails, etc.)
• sync_zulip - synchronization with Zulip
• refresh_groups - updates groups (run after adding new group configurations)

Using Django Celery Beat, you must schedule regular tasks:

• messages.tasks.send_scheduled_messages - sends scheduled messages (~ every 5 min)
• payments.tasks.fetch_payments - fetches membership payments from the Fio Bank API (~ every 5 min)
• payments.tasks.process_payments - processes payment notifications and events (~ hourly during the day)
• sync.tasks.sync_groups_forum - syncs groups with the forum (~ every 5 min)
• sync.tasks.sync_groups_nextcloud - syncs groups with Nextcloud (~ every 5 min)
• sync.tasks.sync_groups_keycloak - syncs groups with Keycloak (~ every 5 min)
• sync.tasks.sync_users_redmine - syncs users with Redmine (~ every 5 min)
• sync.tasks.sync_groups_redmine - syncs groups with Redmine (~ every 5 min)

Admin Theme

The theme using django-admin-interface can be loaded with:

python manage.py loaddata admin_interface_theme_pirati.json

The Octopus logo is located in octopus/admin/static/admin/chobotnice.png. It must be uploaded in the theme settings under "Pirates".

There is also chobotnice-test.png for test installations and chobotnice-dev.png for local development versions.

Development

A helper Makefile is defined for common tasks. For help, run:

$ make help

Local installation and running

Create virtualenv for dependency installation

Create the virtualenv:

$ make venv

It creates a virtualenv in the .venv folder. It assumes the default terminal python is Python 3. If not, use Pyenv to install multiple Python versions safely.

Activate the virtualenv

Before working on the project, activate the virtualenv. This cannot be done using make. You must run:

$ source .venv/bin/activate

You can create a shell alias for this. Add to ~/.bash_profile, ~/.zshrc, or similar:

alias senv='source .venv/bin/activate'

Then you can activate it simply with:

$ senv

For a more advanced solution that automatically activates the virtualenv when entering the project directory, use direnv.

Deactivate the virtualenv with:

$ deactivate

Install dependencies

In the activated virtualenv, run:

$ make install

This installs Python dependencies for local project development.

Set environment variables

Environment variables (see configuration above) are loaded from the .env file, which may look like this:

DATABASE_URL=postgres://db:db@localhost:5432/octopus
OIDC_RP_REALM_URL=http://localhost:8080/auth/realms/master/
OIDC_RP_CLIENT_ID=octopus
OIDC_RP_CLIENT_SECRET=abcd

For local development, the settings include the following defaults:

DEBUG = True
ALLOWED_HOSTS = ["*"]

Project management

Database migrations

Apply database migrations:

$ make migrate

When models change, generate migrations with:

$ make migrations

Run development server

Start the Django development server with the Octopus admin interface on port 8008:

$ make run

Then open http://localhost:8008

Similarly, you can run the onboarding web (profile) on port 8009:

$ make onrun

Or the directory web on port 8010:

$ make dirun

Or the API on port 8011:

$ make apirun

Django shell

Start Django shell using shell_plus from Django Extensions:

$ make shell

Tests

The pytest testing framework is used. Run tests with:

$ pytest

Or via make (without parameters):

$ make test

Coverage report:

$ make coverage

Parallel test execution for faster runs:

$ pytest -n auto

Code quality

Code formatting is handled by black. It’s recommended to configure your editor to auto-format files on save.

black formatting runs automatically as part of the pre-commit hooks (see below).

The pre-commit hooks also automatically sort imports using isort.

Pre-commit hooks

The pre-commit framework is used for managing Git pre-commit hooks.

If you have pre-commit installed, run:

$ make install-hooks

This installs the hooks for the project. From then on, each commit will automatically run the defined actions on changed files.

To run hooks manually for all files:

$ make hooks

Dependency upgrade

To upgrade dependencies, use pip-tools (pip install pip-tools):

$ cd requirements/
$ pip-compile -U base.in
$ pip-compile -U prod.in
$ pip-compile -U dev.in

This generates base.txt, prod.txt, and dev.txt.

Alternatively, you can do the same with:

$ make upgrade