Skip to content

Commit 0ad7c1c

Browse files
committed
border: NAT gateway resource (border_agent / border_node / dedicated VM)
Introduce the Border control-plane resource: a machine-managed NAT gateway carrying its SNAT/DNAT rules inline (jsonb[]) on one flat resource, with a PaaS builder, user API, BorderBuilder unit tests and network.border.* permission seeds bound to project owners. Deployment shapes, calque of net_lb.type: * default -> border_agent capability on the core node's agent; * node set -> border_node pinned to that node's agent (distributed egress, e.g. a managed realm node); * type.kind: core -> border_node on a dedicated gateway VM the iaas builder provisions (net_border_iaas -> target_node_set). The core node's agent serves only border_agent: border_node is always pinned by the builder, and its driver ships where the data plane lives (realm node / border VM image), mirroring paas_lb_agent vs paas_lb_node. ec-bootstrap-templates restores /etc configs from persisted copies, so the border additions to the universal agent template would never reach an upgraded stand. Hoist the scheduler-section rewrite out of the repo-proxy migration (only reachable on pre-repo-proxy stands) into _ensure_ua_config_current(), run unconditionally on every boot: rewrite [universal_agent_scheduler], insert BorderAgentCapabilityDriver into caps_drivers, sync the persisted copy and try-restart the agent services. Covered by unit tests. Also: single 0066 init migration (final net_border schema + permission seeds), gcl_sdk 3.1.0, drop core's own '/' static route on the shared LB vhost, point the CLI realm endpoint docs at Core's LB.
1 parent 855a3db commit 0ad7c1c

24 files changed

Lines changed: 2525 additions & 1209 deletions

File tree

docs/usage/local_deployment.md

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -126,10 +126,15 @@ field. Use this token as a `Bearer` token in all subsequent API requests.
126126
Configure the `exordos` CLI by registering a realm and a context with the admin credentials:
127127

128128
```bash
129-
exordos settings set-realm local --endpoint http://10.20.0.2:11010 --current
129+
exordos settings set-realm local --endpoint http://10.20.0.2:80/api/core --current
130130
exordos settings set-context local --name admin -u <ADMIN_USERNAME> -p <ADMIN_PASSWORD> --current
131131
```
132132

133+
The CLI authenticates against the `default` client alias (see above), which only Core's load
134+
balancer on port 80 knows how to rewrite. Pointing the realm at the node's own user-api port
135+
(`:11010`) instead skips the load balancer, so the alias can't be resolved and every command fails
136+
with `Can't parse value: uuid=default.`
137+
133138
- `set-realm` registers the platform endpoint under the name `local` and marks it as the active realm.
134139
- `set-context` creates a named context with the admin credentials and marks it as the active context.
135140

docs/usage/local_deployment.ru.md

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -126,10 +126,15 @@ curl --location 'http://10.20.0.2:80/api/core/v1/iam/clients/default/actions/get
126126
Настройте CLI `exordos`, зарегистрировав realm и контекст с учётными данными администратора:
127127

128128
```bash
129-
exordos settings set-realm local --endpoint http://10.20.0.2:11010 --current
129+
exordos settings set-realm local --endpoint http://10.20.0.2:80/api/core --current
130130
exordos settings set-context local --name admin -u <ADMIN_USERNAME> -p <ADMIN_PASSWORD> --current
131131
```
132132

133+
CLI аутентифицируется через алиас клиента `default` (см. выше), который умеет переписывать только
134+
балансировщик Core на порту 80. Если указать в качестве realm порт user-api самой ноды (`:11010`),
135+
балансировщик будет пропущен, алиас не резолвится, и любая команда падает с
136+
`Can't parse value: uuid=default.`
137+
133138
- `set-realm` — регистрирует endpoint платформы под именем `local` и устанавливает его как активный realm.
134139
- `set-context` — создаёт именованный контекст с учётными данными администратора и устанавливает его как активный.
135140

etc/exordos_universal_agent/exordos_universal_agent.conf.j2

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@ caps_drivers =
1717
PasswordCapabilityDriver,
1818
CoreDNSCertificateCapabilityDriver,
1919
LBAgentCapabilityDriver,
20+
BorderAgentCapabilityDriver,
2021
GuestMachineCapabilityDriver,
2122
SSHKeyCapabilityDriver,
2223
RenderAgentDriver
@@ -28,7 +29,8 @@ capabilities =
2829
password,
2930
certificate,
3031
paas_lb_agent,
31-
repo_proxy_installed_element
32+
repo_proxy_installed_element,
33+
border_agent
3234

3335

3436
[CoreDNSCertificateCapabilityDriver]

exordos/manifests/core.yaml.j2

Lines changed: 0 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -268,26 +268,6 @@ resources:
268268
- kind: rewrite_url
269269
regex: "^/api/core/v1/iam/clients/default(.*)"
270270
replacement: f"/v1/iam/clients/{$core.vs.variables.$iam_default_client_uuid:value}$1"
271-
core_lb_static_route:
272-
project_id: "12345678-c625-4fee-81d5-f691897b8142"
273-
parent: $core.network.lb.$core_lb.vhosts.$core_lb_core_http:uuid
274-
condition:
275-
kind: prefix
276-
value: /
277-
allowed_ips:
278-
- 0.0.0.0/0
279-
modifiers:
280-
- kind: auto_header
281-
headers:
282-
- 'Host'
283-
- 'X-Forwarded-For'
284-
- 'X-Forwarded-Port'
285-
- 'X-Forwarded-Proto'
286-
- 'X-Forwarded-Prefix'
287-
actions:
288-
- kind: local_dir
289-
path: /var/www/html
290-
is_spa: true
291271

292272
exports:
293273
local_domain:

exordos_core/cmd/bootstrap.py

Lines changed: 75 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@
2222
import os
2323
import pwd
2424
import re
25+
import subprocess
2526
import sys
2627
import time
2728
import typing as tp
@@ -379,15 +380,10 @@ def _sync_config_to_data_path(content: str, data_path: str) -> None:
379380

380381

381382
def _migrate_installed_elements_configs() -> None:
382-
"""Migrate config files for repo proxy support.
383+
"""Migrate the core config file for repo proxy support.
383384
384-
Idempotently updates two config files:
385-
386-
1. Adds [launchpad] section to the core config file.
387-
2. Adds repo_proxy_installed_element to the capabilities list
388-
in the universal agent config file.
385+
Idempotently adds the [launchpad] section to the core config file.
389386
"""
390-
# 1. Add [launchpad] section to the core config
391387
try:
392388
with open(CORE_CONFIG_PATH, "r", encoding="utf-8") as f:
393389
content = f.read()
@@ -405,45 +401,92 @@ def _migrate_installed_elements_configs() -> None:
405401
else:
406402
LOG.info("[launchpad] section already exists in %s", CORE_CONFIG_PATH)
407403

408-
# 2. Replace [universal_agent_scheduler] section in the UA config
409-
# to ensure repo_proxy_installed_element is present in capabilities.
410-
# The section is fully rewritten to handle any indentation differences.
411-
_UA_SCHEDULER_SECTION = """\
404+
405+
# The parts of the universal agent config that evolve with the core image.
406+
# The scheduler section is fully rewritten to handle any indentation
407+
# differences; keep both literals in sync with
408+
# etc/exordos_universal_agent/exordos_universal_agent.conf.j2.
409+
_UA_SCHEDULER_SECTION = """\
412410
[universal_agent_scheduler]
413411
capabilities =
414412
em_*,
415413
password,
416414
certificate,
417415
paas_lb_agent,
418-
repo_proxy_installed_element
416+
repo_proxy_installed_element,
417+
border_agent
419418
"""
419+
_UA_BORDER_DRIVER_LINE = " BorderAgentCapabilityDriver,\n"
420+
421+
422+
def _ensure_ua_config_current() -> None:
423+
"""Bring the universal agent config up to date with this image.
424+
425+
ec-bootstrap-templates restores /etc configs from the persisted
426+
copies on the data disk, so template changes shipped in a new core
427+
image never reach an upgraded stand on their own. Idempotently
428+
reapply the image-defined parts of the UA config:
429+
430+
1. Rewrite the [universal_agent_scheduler] section with the current
431+
capabilities list.
432+
2. Ensure BorderAgentCapabilityDriver is present in caps_drivers
433+
([universal_agent] holds stand-specific endpoints, so only this
434+
line is inserted, not the whole section).
435+
436+
On changes the result is synced back to the persisted copy and the
437+
agent services (started before ec-bootstrap runs, thus holding the
438+
stale config) are restarted to pick it up.
439+
"""
420440
try:
421441
with open(UA_CONFIG_PATH, "r", encoding="utf-8") as f:
422442
content = f.read()
423443
except FileNotFoundError:
424444
LOG.warning("Universal agent config not found: %s", UA_CONFIG_PATH)
425-
else:
426-
new_content = re.sub(
427-
r"^\[universal_agent_scheduler\].*?(?=^\[|\Z)",
428-
_UA_SCHEDULER_SECTION,
429-
content,
430-
count=1,
431-
flags=re.DOTALL | re.MULTILINE,
445+
return
446+
447+
new_content = re.sub(
448+
r"^\[universal_agent_scheduler\].*?(?=^\[|\Z)",
449+
_UA_SCHEDULER_SECTION,
450+
content,
451+
count=1,
452+
flags=re.DOTALL | re.MULTILINE,
453+
)
454+
455+
if "BorderAgentCapabilityDriver" not in new_content:
456+
new_content = new_content.replace(
457+
" LBAgentCapabilityDriver,\n",
458+
" LBAgentCapabilityDriver,\n" + _UA_BORDER_DRIVER_LINE,
459+
1,
432460
)
433-
if new_content != content:
434-
with open(UA_CONFIG_PATH, "w", encoding="utf-8") as f:
435-
f.write(new_content)
436-
LOG.info(
437-
"Updated [universal_agent_scheduler] section in %s",
438-
UA_CONFIG_PATH,
439-
)
440-
_sync_config_to_data_path(new_content, UA_CONFIG_DATA_PATH)
441-
else:
442-
LOG.info(
443-
"[universal_agent_scheduler] section already up to date in %s",
461+
if "BorderAgentCapabilityDriver" not in new_content:
462+
LOG.warning(
463+
"Could not insert BorderAgentCapabilityDriver into caps_drivers in %s",
444464
UA_CONFIG_PATH,
445465
)
446466

467+
if new_content == content:
468+
LOG.info("Universal agent config already up to date in %s", UA_CONFIG_PATH)
469+
return
470+
471+
with open(UA_CONFIG_PATH, "w", encoding="utf-8") as f:
472+
f.write(new_content)
473+
LOG.info("Updated universal agent config %s", UA_CONFIG_PATH)
474+
_sync_config_to_data_path(new_content, UA_CONFIG_DATA_PATH)
475+
476+
try:
477+
subprocess.run(
478+
[
479+
"systemctl",
480+
"try-restart",
481+
"exordos-universal-agent",
482+
"exordos-universal-scheduler",
483+
],
484+
check=True,
485+
)
486+
LOG.info("Restarted universal agent services to apply the new config")
487+
except (OSError, subprocess.CalledProcessError) as e:
488+
LOG.warning("Failed to restart universal agent services: %s", e)
489+
447490

448491
def _migrate_installed_elements_to_repo() -> None:
449492
"""Migrate installed EM elements to repo proxy.
@@ -737,6 +780,8 @@ def main() -> None:
737780
# working installations are migrated.
738781
_migrate_installed_elements_to_repo()
739782

783+
_ensure_ua_config_current()
784+
740785
if not os.path.exists(SPEC_PATH):
741786
LOG.info("No spec file found at %s", SPEC_PATH)
742787
return

exordos_core/cmd/gservice.py

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,15 @@
3434
default="https://repo.exordos.com/exordos-lbaas/0.1.0/exordos-lbaas.raw.zst",
3535
help="URL to get image for LB dataplane VM",
3636
),
37+
cfg.StrOpt(
38+
"border-image",
39+
default=(
40+
"https://repo.exordos.com/exordos-border/0.1.0/exordos-border.raw.zst"
41+
),
42+
help="URL to get the image for a `core` (VM-based) border gateway. "
43+
"The image must ship the universal agent with "
44+
"BorderCapabilityDriver (border_node capability).",
45+
),
3746
]
3847

3948

exordos_core/gservice/service.py

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -48,6 +48,9 @@
4848
from exordos_core.elements.services import builders as em_builders
4949
from exordos_core.janitor import service as janitor_service
5050
from exordos_core.network import service as n_network_service
51+
from exordos_core.network.border.builders import iaas as net_border_iaas
52+
from exordos_core.network.border.builders import paas as net_border_paas
53+
from exordos_core.network.border.dm import models as border_models
5154
from exordos_core.network.lb.builders import iaas as net_lb_iaas
5255
from exordos_core.network.lb.builders import paas as net_lb_paas
5356
from exordos_core.network.lb.dm import models as lb_models
@@ -123,6 +126,11 @@ def __init__(self, iter_min_period=3, iter_pause=0.1):
123126
project_id=nc.NODE_SET_PROJECT,
124127
)
125128
net_lb_paas_builder = net_lb_paas.LBBuilder()
129+
net_border_iaas_builder = net_border_iaas.BorderIaasBuilder(
130+
instance_model=border_models.IaasBorder,
131+
project_id=nc.NODE_SET_PROJECT,
132+
)
133+
net_border_paas_builder = net_border_paas.BorderBuilder()
126134

127135
# Infra scheduler
128136
infra_scheduler = ua_scheduler_service.UniversalAgentSchedulerService(
@@ -228,6 +236,8 @@ def __init__(self, iter_min_period=3, iter_pause=0.1):
228236
service_builder,
229237
net_lb_iaas_builder,
230238
net_lb_paas_builder,
239+
net_border_iaas_builder,
240+
net_border_paas_builder,
231241
secret_svc,
232242
password_builder,
233243
em_builder,

exordos_core/network/border/__init__.py

Whitespace-only changes.

exordos_core/network/border/builders/__init__.py

Whitespace-only changes.
Lines changed: 110 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,110 @@
1+
# Copyright 2026 Genesis Corporation.
2+
#
3+
# All Rights Reserved.
4+
#
5+
# Licensed under the Apache License, Version 2.0 (the "License"); you may
6+
# not use this file except in compliance with the License. You may obtain
7+
# a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
13+
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
14+
# License for the specific language governing permissions and limitations
15+
# under the License.
16+
17+
import logging
18+
import typing as tp
19+
import uuid as sys_uuid
20+
21+
from gcl_sdk.agents.universal.dm import models as ua_models
22+
from gcl_sdk.infra import constants as sdk_c
23+
from gcl_sdk.infra.dm import models as sdk_models
24+
from gcl_sdk.infra.services import builder
25+
from oslo_config import cfg
26+
27+
from exordos_core.network.border.dm import models
28+
from exordos_core.network.lb.dm import models as lb_models
29+
30+
LOG = logging.getLogger(__name__)
31+
CONF = cfg.CONF
32+
33+
NODE_SET_KIND = lb_models.TargetNodeSet.get_resource_kind()
34+
35+
36+
class BorderIaasBuilder(builder.CoreInfraBuilder):
37+
"""Provisions the dedicated VM of a `core` (VM-based) border.
38+
39+
Calque of the LB iaas builder: one single-replica target_node_set from
40+
``[gservice] border_image`` (an image whose universal agent ships
41+
BorderCapabilityDriver). The paas builder schedules the ``border_node``
42+
capability onto that VM's agent, same as it does for a pinned node.
43+
"""
44+
45+
_name_prefix = "border"
46+
47+
def __init__(
48+
self,
49+
instance_model: tp.Type[models.IaasBorder],
50+
project_id: sys_uuid.UUID,
51+
):
52+
super().__init__(instance_model)
53+
self._project_id = project_id
54+
55+
def create_infra(
56+
self, instance: models.IaasBorder
57+
) -> tp.Collection[ua_models.TargetResourceKindAwareMixin]:
58+
# A pinned border (node set) never provisions its own VM,
59+
# whatever the type says.
60+
if instance.node or instance.type.kind != "core":
61+
return []
62+
63+
node_set = lb_models.TargetNodeSet(
64+
uuid=instance.uuid,
65+
name=f"{self._name_prefix}-{instance.name}",
66+
cores=instance.type.cpu,
67+
ram=instance.type.ram,
68+
replicas=1,
69+
project_id=self._project_id,
70+
status=sdk_c.NodeStatus.NEW.value,
71+
disk_spec=sdk_models.SetRootDiskSpec(
72+
size=instance.type.disk_size,
73+
image=CONF.gservice.border_image,
74+
),
75+
)
76+
return [node_set]
77+
78+
def actualize_infra(
79+
self,
80+
instance: models.IaasBorder,
81+
infra: builder.InfraCollection,
82+
) -> tp.Collection[ua_models.TargetResourceKindAwareMixin]:
83+
if instance.node or instance.type.kind != "core":
84+
return tuple()
85+
86+
nodeset = None
87+
tgt_nodeset = None
88+
for target, actual in infra.infra_objects:
89+
if target.get_resource_kind() == NODE_SET_KIND:
90+
nodeset = actual
91+
target.cores = instance.type.cpu
92+
target.ram = instance.type.ram
93+
target.replicas = 1
94+
tgt_nodeset = target
95+
break
96+
else:
97+
raise ValueError("A core border must have its target_node_set!")
98+
99+
if nodeset and nodeset.nodes:
100+
instance.ipsv4 = [node["ipv4"] for node in nodeset.nodes.values()]
101+
102+
# The paas builder owns the final ACTIVE/ERROR flip (it tracks the
103+
# border_node capability); here only reflect VM provisioning.
104+
try:
105+
if sdk_c.InstanceStatus(nodeset.status) == sdk_c.InstanceStatus.ERROR:
106+
instance.status = sdk_c.InstanceStatus.ERROR.value
107+
except (ValueError, AttributeError):
108+
pass
109+
110+
return (tgt_nodeset,)

0 commit comments

Comments
 (0)