Elasticsearch integration to the Docker Compose setup

[pelagiann]

This PR adds Elasticsearch integration to the Docker Compose setup for the noticeboard feature.

[Copilot]

Pull request overview

Adds an Elasticsearch service to the local Docker Compose stack to support the noticeboard feature, and updates the Django image build to install Poetry dependencies without installing the project package itself.

Changes:

• Add an elasticsearch service and persistent volume to docker-compose.yml.
• Start Elasticsearch as part of the scripts/start/development.sh scaffolding services.
• Change Django image dependency install to poetry install --no-root.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 9 comments.

File	Description
scripts/start/development.sh	Includes elasticsearch in the dev startup script’s compose services list.
docker-compose.yml	Defines the Elasticsearch service (image, ports, env, healthcheck) and adds esdata volume.
django/Dockerfile	Updates Poetry install step to avoid installing the root package during image build.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Port 9300 (Elasticsearch transport) is being published to the host. Unless you explicitly need node-to-node transport access from the host, avoid publishing 9300 and keep it internal to reduce attack surface and accidental coupling.

Suggested change

- "9300:9300"

[Copilot]

List indentation for ports here differs from the rest of the file (most sections use ports:\n- "..." without extra indentation). Please align formatting with the existing compose style for consistency and to reduce diff noise.

Suggested change

	- "9200:9200"
	- "9300:9300"
	- "9200:9200"
	- "9300:9300"

[rtb-12]

Reviewed alongside the corresponding app PR (omniport-app-noticeboard#22). Good to see the missing ES service finally land in the compose. A few issues worth fixing before merge — calling out the security one first since it's the most serious:

🔴 Security blocker

1. ES is published to the public internet with no auth

ports:
  - "9200:9200"
  - "9300:9300"
environment:
  - xpack.security.enabled=false

ports: binds to 0.0.0.0 on the host by default. Combined with xpack.security.enabled=false, anyone on the internet who can reach the host on 9200 has full read/write/delete on every index. This is one of the most common ES data-leak vectors (regularly indexed by Shodan).

The other containers don't need ports: to reach ES — they can hit http://elastic:9200 over the internal network overlay. Either drop ports: entirely, or bind to localhost only:

ports:
  - "127.0.0.1:9200:9200"

And keep xpack.security.enabled=true for any non-dev compose; if you want a "dev only with security off" mode, put it behind a docker-compose.override.yml.

Functional issues

2. ES Python deps still aren't in pyproject.toml
This PR adds the service, but the corresponding app PR (#22) needs elasticsearch and django-elasticsearch-dsl added to pyproject.toml. Without that, the Django container will boot and immediately crash on from elasticsearch import Elasticsearch. Worth coordinating the two PRs so they merge as a unit.

3. poetry install --no-root change is unexplained
The Dockerfile change from poetry install to poetry install --no-root looks unrelated to ES. If this is a fix for an unrelated build error (omniport not being a packageable project), please call that out in the PR description — otherwise it looks accidental and reviewers can't tell whether to keep it.

4. Healthcheck depends on curl, which the ES 8 image doesn't ship

test: ["CMD-SHELL", "curl --silent --fail localhost:9200/_cluster/health || exit 1"]

The official docker.elastic.co/elasticsearch/elasticsearch:8.x image is built on UBI minimal and does not include curl. This healthcheck will report unhealthy forever. Either install curl in a derived image, or use a probe that's already in the image, e.g.:

test: ["CMD-SHELL", "wget -qO- http://localhost:9200/_cluster/health | grep -vq '\"status\":\"red\"'"]

(Note: wget may also be absent on some 8.x tags — worth docker run --rm <image> which curl wget to confirm.)

5. Django services don't depends_on elasticsearch
intranet-server and internet-server will start before ES finishes booting (30-60s on first run). Every search request in that window will throw ConnectionError. Add to both Django services:

depends_on:
  elasticsearch:
    condition: service_healthy

Caveat: the file is version: '3.4' — condition was dropped in compose v3 and brought back in v3.4 of compose spec. Should work, but verify with your compose version.

6. chmod accidentally cleared on scripts/start/development.sh
The diff shows old mode 100755 / new mode 100644. The script loses its execute bit; ./scripts/start/development.sh will fail. Restore with git update-index --chmod=+x scripts/start/development.sh.

Operational concerns

7. No JVM heap cap
ES will auto-size its heap based on container memory. On a small dev box, it can chew several GB and starve Postgres / Redis. Standard practice:

environment:
  - ES_JAVA_OPTS=-Xms512m -Xmx512m

Tune for prod separately.

8. container_name: elastic is brittle
Hardcoded container names break scaling (docker-compose up --scale fails) and break under non-default compose project names. The only reason it's pinned is to match the hardcoded Elasticsearch(['http://elastic:9200']) in app PR #22 — but that hostname should come from settings/env anyway (see my comment on #22). If you switch the app to read from env, the service can just use its compose-default name elasticsearch and this container_name: line goes away.

9. Missing restart: always
Every other service in this file has it. Inconsistent.

10. Trailing whitespace on - network

Production deployment caveat

This PR adds ES inside the noticeboard compose stack. That's fine for dev, but it's the opposite of what's wanted in prod (ES on a separate VM so the noticeboard host isn't carrying the index workload). Two suggestions:

1. Move this service definition into a docker-compose.override.yml.example so it's clearly labeled "dev only" and doesn't get accidentally deployed to prod boxes.
2. Add an ELASTICSEARCH_HOST env var on the Django services that defaults to http://elastic:9200 (matching the dev sidecar) but can be overridden to https://search.internal.example.com:9200 in prod. That way the same image runs in both topologies — dev points at the sidecar, prod points at the dedicated VM.

Happy to send a follow-up PR for #1 + #2 if useful.

[rtb-12]

After implementing #54 as an alternative shape and validating the full ES integration end-to-end on staging (against PR #22 + omniport-backend#213), I'd suggest closing this PR in favor of #54.

The two PRs solve the same problem with very different tradeoffs. Quick comparison:

	#52 (this PR)	#54
Service location	Inline in canonical docker-compose.yml	docker-compose.override.yml.example (opt-in for dev)
Port binding	9200:9200 (publicly reachable on host)	127.0.0.1:9200:9200 (localhost only)
xpack.security	disabled with no caveat	disabled, but isolated by localhost binding + explicit "dev only" comment
JVM heap	uncapped — can grow to consume any free RAM	ES_JAVA_OPTS=-Xms512m -Xmx512m
Healthcheck start_period	absent — container reports unhealthy for 30+ seconds during boot	start_period: 30s
container_name: elastic	hardcoded — breaks --scale and non-default project names	uses default service name (works because the app code now reads the host from env)
Env-driven host config	none — code must hardcode 'http://elastic:9200'	noticeboard/elasticsearch_stencil.env + env_file: passthrough — same image works in dev (sidecar) and prod (separate ES VM)
Dev/prod separation	none	override pattern keeps prod compose clean

The biggest functional issue is the public port binding combined with security disabled — anyone reachable to the host on 9200 has full read/write/delete access to all indices. This is a regularly-exploited deployment vector (Shodan continuously scans for it).

If you'd prefer to keep #52's inline approach instead of #54's override, I'd suggest at minimum:

1. Change ports: 9200:9200 → ports: 127.0.0.1:9200:9200 (or drop ports: entirely; expose: is enough for inter-container)
2. Add ES_JAVA_OPTS=-Xms512m -Xmx512m
3. Add start_period: 30s to the healthcheck
4. Drop container_name: elastic (only needed because the app hardcoded the hostname; once omniport-backend#213 reads host from env per its ongoing review, this becomes unnecessary)

But cleanest path: close this and merge #54 instead. Happy to consolidate if useful.

[pelagiann]

Closing this in favour of #54, which solves the same problem in a better shape.

Three things decided it:

1. Elasticsearch should not be inline in the canonical compose file. As written here, every deploy of the stack also deploys Elasticsearch on the same host. Env-driven Elasticsearch wiring + dev override (alternative to #52) #54's override pattern lets a developer opt into a local sidecar while a real deployment points at a separate cluster through the same variables.

2. The connection belongs in the environment. The settings module in Restore the Elasticsearch settings and make the CSRF cookie name configurable omniport-backend#238 already reads ELASTICSEARCH_HOST, ELASTICSEARCH_USER, ELASTICSEARCH_VERIFY_CERTS and ELASTICSEARCH_CA_CERT from os.environ, and its docstring refers to noticeboard/elasticsearch_stencil.env — a file only Env-driven Elasticsearch wiring + dev override (alternative to #52) #54 ships. The application side has effectively already adopted that shape.

3. The security defaults raised in review are real. On the deployment currently running this branch, docker port elastic reports 9200/tcp -> 0.0.0.0:9200 and 9300/tcp -> 0.0.0.0:9300, xpack.security.enabled is false, and an unauthenticated GET /_cat/indices returns 200 from the host address. Env-driven Elasticsearch wiring + dev override (alternative to #52) #54's example override binds to 127.0.0.1 and caps the JVM heap, which is the correct default for anyone copying this.

Nothing here is lost — #54 covers the same ground. It needs a rebase onto current master before it can merge.