When using the horizon.yaml configuration file, the access results in a 503 error.

[hanzhiyan588]

####The problems encountered

1. When deploying using Docker, if the horizon,yaml configuration file is not used, normal access can be achieved but it will prompt that authentication is not configured.
2. When deploying using Docker, using the h
   orizon.yaml configuration file results in a 503 error being returned, and no log output is seen when the debug log level is enabled.

####This is the complete configuration file.

# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# Copy this file to `horizon.yaml` and edit. Hot-reload picks up changes.
#
# Env-var interpolation: `${VAR}` or `${VAR:default}` is expanded BEFORE
# YAML parsing. Use the `:default` form for non-secret values; for
# secrets prefer the bare `${VAR}` form so a missing env var fails loud
# instead of silently falling back to a placeholder.

server:
  host: 127.0.0.1
  port: 8081

oap:
  # OAP query host (port 12800 by default; GraphQL + /status/*). One URL
  # because query traffic is load-balanceable — any OAP node can answer.
  queryUrl: http://oap:12800
  # OAP admin host (port 17128 by default; runtime-rule / dsl-debugging
  # / inspect / status live here). Single URL — point it at a DNS name
  # or VIP fronting your OAP cluster. Runtime-rule lists hit it once
  # (OAP routes cluster-internal); per-node live-debug status uses a
  # DNS lookup on the hostname to discover all node IPs and probes each.
  adminUrl: http://oap:17128
  # OAP's Zipkin v REST host. Defaults to a standalone port (9412 +
  # /zipkin) per the upstream Armeria binding. When OAP is configured
  # to share the GraphQL port (typical for the demo / k8s deploys),
  # use `<queryUrl>/zipkin` instead.
#  zipkinUrl: http://127.0.0.1:9412/zipkin
  timeoutMs: 15000
  # Optional basic-auth credentials for outbound OAP calls.
 # auth:
 #   username: skywalking
 #   password: skywalking

# ────────────────────────────────────────────────────────────────────
# Authentication
#
# Pick ONE backend: `local` or `ldap`. The other block can stay
# populated for break-glass / future use, but it's IGNORED. If you set
# `backend: ldap` and leave `auth.local.users` populated, a warning is
# logged at startup.
#
# Bootstrap rule: if `backend: local` and `auth.local.users` is empty,
# OR `backend: ldap` and `auth.ldap` is missing / `groupMappings` empty,
# the BFF boots but no login succeeds until auth is configured. There is
# no "default admin/admin" password.
# Generate hashes with:   pnpm --filter bff cli:hash
# ────────────────────────────────────────────────────────────────────
auth:
  backend: local
  local:
    users:
      - username: admin123456
        passwordHash: "$argon2id$v=19$m=65536,t=3,p=1$+90YvdOFPPDubbjwEXz0jQ$IxEXvs+C2yck64zCU/9rJDZ01A6A1nfR2Fc3bfmcphw"
        roles: [admin]

  # ── LDAP ──────────────────────────────────────────────────────────
  # Required when `backend: ldap`. Passwords are never stored or read
  # from the directory — login binds AS the user with their typed
  # password, and group membership maps to a Horizon role via
  # `groupMappings` (first match wins, multiple matches union).
  # ldap:
  #   url: ldaps://ldap.corp:636
  #   bindDn: "cn=horizon,ou=services,dc=corp"
  #   # `${VAR}` (required) or `${VAR:default}` (optional). Leave empty
  #   # for anonymous service-bind (only works if the directory permits
  #   # anonymous searches).
  #   bindPassword: "${HORIZON_LDAP_BIND_PW}"
  #   userBaseDn: "ou=people,dc=corp"
  #   # `{username}` is substituted with the typed username (RFC 4515
  #   # escaped). Use `(sAMAccountName={username})` for AD.
  #   userFilter: "(uid={username})"
  #   displayNameAttr: cn
  #   # `memberOf` reads the attr off the user entry (AD-style);
  #   # `search` searches `groupBaseDn` for the user's DN (OpenLDAP-style).
  #   groupStrategy: memberOf
  #   groupBaseDn: ""        # only used when groupStrategy: search
  #   memberAttr: member     # only used when groupStrategy: search
  #   timeoutMs: 5000
  #   tlsInsecure: false     # set true only for dev with self-signed certs
  #   groupMappings:
  #     - group: "cn=horizon-admin,ou=groups,dc=corp"
  #       role: admin
  #     - group: "cn=sre,ou=groups,dc=corp"
  #       role: operator
  #     - group: "cn=platform,ou=groups,dc=corp"
  #       role: maintainer
  #     - group: "*"           # fallback for every authenticated user
  #       role: viewer

  # ── Break-glass ───────────────────────────────────────────────────
  # Optional local admin credential honored ONLY when:
  #   - backend: ldap, AND
  #   - the LDAP probe is currently failing.
  # Every successful break-glass login is audited at WARN level. Leave
  # commented out to disable.
  # breakGlass:
  #   username: emergency-admin
  #   passwordHash: "${HORIZON_BREAK_GLASS_HASH}"
  #   roles: [admin]

# ────────────────────────────────────────────────────────────────────
# RBAC: four built-in roles. Verbs are dot-namespaced; `*:read` grants
# every read across all areas, `area:*` grants every action in one area,
# `*` grants everything. Routes are gated by a single policy table on
# the BFF — see `apps/bff/src/rbac/route-policy.ts`.
# ────────────────────────────────────────────────────────────────────
rbac:
  enabled: true
  roles:
    # Data catalog + read-only overview dashboards. Deliberately NOT
    # `*:read` so a viewer can't accidentally see rule definitions,
    # live-debug sessions, setup screens, or platform internals.
    viewer:
      - metrics:read
      - alarms:read
      - traces:read
      - logs:read
      - topology:read
      - profile:read
      - overview:read

    # Viewer + platform monitoring (OAP cluster + module inspector).
    maintainer:
      - metrics:read
      - alarms:read
      - traces:read
      - logs:read
      - topology:read
      - profile:read
      - overview:read
      - cluster:read
      - ttl:read
      - config:read
      - inspect:read

    # Configures observability: dashboards, alarm rules, DSL/OAL,
    # diagnostics, profiling. Inherits viewer + platform reads so the
    # operator can verify their edits against live data.
    operator:
      - metrics:read
      - alarms:read
      - traces:read
      - logs:read
      - topology:read
      - profile:read
      - cluster:read
      - ttl:read
      - config:read
      - inspect:read
      - overview:read
      - overview:write
      - setup:read
      - setup:write
      - dashboard:read
      - dashboard:write
      - alarm-setup:read
      - alarm-setup:write
      - alarm-rule:read
      - alarm-rule:write
      - rule:read
      - rule:write
      - rule:write:structural
      - rule:delete
      - rule:debug
      - live-debug:read
      - live-debug:write
      - profile:enable

    admin:
      - "*"
  # Where each role lands after login. First role on the user wins.
  # Cluster status lives under /operate/cluster (it's an operator tool
  # against OAP, not an admin / RBAC surface); the prior `/admin/cluster`
  # values 404'd because no route by that name exists.
  landingByRole:
    viewer: /
    maintainer: /operate/cluster
    operator: /
    admin: /operate/cluster

session:
  ttlMinutes: 60
  cookieName: horizon_sid
  cookieSecure: false           # set true behind HTTPS

# State files (audit log, setup state, alarm state, wire debug log).
# Leave the `file:` paths unset and the right location is chosen for you:
#   • published container image → /data/*    (its writable volume)
#   • local binary run          → ./horizon-* (relative to the working dir)
# Set an explicit `file:` under each block ONLY if you need a specific
# location — and make sure the running process can write it (in the image
# that means a path under /data, since the non-root `horizon` user cannot
# write /app).

debugLog:
  enabled: true
  maxBodyChars: 8192
  redactAuthHeaders: true

[wu-sheng]

Hi @hanzhiyan588, thanks for the detailed report — pasting the full config makes this easy to pin down.

This is a bind-address issue, not an auth issue. The relevant line is:

server:
  host: 127.0.0.1

Why it works without horizon.yaml but 503s with it

The published image sets HORIZON_SERVER_HOST=0.0.0.0. That env var is only the default the config schema falls back to — an explicit server.host in horizon.yaml always wins over it.

• No horizon.yaml → the BFF binds 0.0.0.0 (all interfaces) → reachable through -p 8081:8081. You still see the “auth not configured” notice because no users are defined yet, but the listener itself is reachable.
• With your horizon.yaml → the explicit host: 127.0.0.1 overrides the default, so the BFF binds the container’s loopback only. -p 8081:8081 (or any reverse proxy / ingress in front) can’t reach a loopback-only listener, so the connection never lands on the process.

That also explains the empty debug log: the request dies at the network edge before it ever reaches the app, so there’s nothing for the BFF to log. (debugLog records outbound OAP calls, not inbound HTTP, so it wouldn’t capture this regardless.) Whether you see a 503 or a connection reset just depends on what’s in front of the container — a reverse proxy / ingress with the BFF as an unreachable upstream reports 503; a bare docker-proxy gives a reset.

Fix

Bind all interfaces inside the container:

server:
  host: 0.0.0.0
  port: 8081

Or simply remove the server: block entirely and let the image default (0.0.0.0) apply.

This is the first entry under Common mistakes in docs/setup/container-image.md:

server.host: 127.0.0.1 inside the container. Listener binds loopback only; -p cannot route traffic in. Set 0.0.0.0.

On our side

You copied this straight from horizon.example.yaml, which ships with host: 127.0.0.1 — correct for a local binary run, but a footgun when copied verbatim into a container. We’ll add an inline comment there calling out the container case so this is harder to hit.

Let me know if 0.0.0.0 clears it up for you. 👍

[hanzhiyan588]

Thank you for your prompt reply. The issue has been resolved. Thank you again.

[hanzhiyan588]

There is another issue - the OAP health check.
healthcheck:
test: [ "CMD-SHELL", "curl http://localhost:12800/internal/l7check" ]
It seems that it can't be executed because when I entered the container and executed the command "curl http://localhost:12800/internal/l7check", it prompted that "curl" does not have this command.

[wu-sheng]

Do you ask because of skywalking repo's docker/docker-compose? If so, we may need some fix for that. It is out-dated and no one using it for a long time. Usually, e2e has their own docker-compose or uses skywalking-helm

[hanzhiyan588]

Thank you for your prompt reply. The issue has been resolved. Thank you again.

[wu-sheng]

Thanks for the follow-up on the healthcheck, @hanzhiyan588 — you're right, the OAP image (eclipse-temurin JRE) ships no curl/wget/nc, so that healthcheck never actually ran.

Ive opened #13899 to fix the whole quickstart at once:

1. Healthcheck — replaced curl .../internal/l7check with bash's built-in /dev/tcp port probe (no extra binary needed), and widened start_period to 90s so BanyanDB schema install during boot doesn't eat the retry budget.
2. 503 / bind address — the quickstart now ships a docker/horizon.yaml with server.host: 0.0.0.0, so the loopback-only footgun from the original report is gone out of the box.
3. UI wiring — fixed the container port mapping (8080:8081) and replaced the non-existent SW_*_ADDRESS env vars with the mounted horizon.yaml (OAP query/admin/zipkin URLs + a demo admin/admin login).

A note on ordering, since it often comes up: the Horizon UI does not need OAP to be up first. The BFF binds its own port immediately and retries OAP connectivity in the background (the UI-template boot-seed waits for OAP's admin port with a backoff loop), so the login page is reachable right away and data pages fill in once OAP is ready — hence no depends_on: oap gate on the ui service.

Please give #13899 a try and let me know if it clears things up. 👍

[hanzhiyan588]

Yes, this issue has been confirmed. There is no need to wait for the OAP to fully start. However, for the sake of user experience, it is still recommended to add "depends_on: oap"