add all resources route, update docs, add cron for creating elements inventory

Author: slashburygin

.github/workflows/inventory.yml

@@ -0,0 +1,22 @@
+name: Build elements inventory
+
+on:
+  workflow_dispatch:
+  schedule:
+    - cron: '5 * * * 0'
+  push:
+
+permissions:
+  contents: read
+
+jobs:
+  Build:
+    runs-on: self-hosted
+    strategy:
+      fail-fast: true
+    steps:
+      - uses: actions/checkout@v6
+        with:
+          fetch-depth: 30
+      - name: Build elements inventory
+        run: python3 genesis_core/cmd/elements_inventory.py

Makefile

@@ -0,0 +1,17 @@
+SHELL := bash
+REPOSITORY := https://repository.genesis-core.tech
+ifeq ($(SSH_KEY),)
+	SSH_KEY = ~/.ssh/id_rsa.pub
+endif
+
+all: help
+
+help:
+	@echo "build_core       - build genesis core"
+	@echo "bootstrap        - bootstrap genesis core"
+
+build_core:
+	genesis build -i $(SSH_KEY) -f . --inventory --manifest-var repository=https://repository.genesis-core.tech
+
+bootstrap:
+	genesis bootstrap -i output/inventory.json -f -m core --admin-password admin --cidr 10.20.0.0/22

docs/em/cli.md

@@ -0,0 +1,23 @@
+# Examples
+
+## Build element from repository
+
+ Build a Genesis element. The command build all images, manifests and other artifacts required for the element. The manifest in the project may be a raw YAML file or a template using Jinja2 templates. For Jinja2 templates, the following variables are
+ available by default:
+
+- {{ version }}: version of the element
+- {{ name }}: name of the element
+- {{ images }}: list of images
+- {{ manifests }}: list of manifests
+
+Go to project directory and run the following command:
+
+```bash
+genesis build <element_name>
+```
+
+## Install element from repository
+
+```bash
+genesis e install <element_name>
+```

docs/element_manager/manifest.md docs/em/manifest.mddocs/element_manager/manifest.md renamed to docs/em/manifest.md

@@ -6,8 +6,6 @@ Welcome to Genesis Core!
 
 The Genesis Core is an open source software that offers a one turnkey solution to deal with infrastructure at all levels - from bare metal and virtual machines to applications and services.
 
-Refer to the [wiki](https://github.com/infraguys/genesis_core/wiki) for more detailed information.
-
 ## 📦 Installation
 
 There are several ways to install Genesis Core and depend on your purpose you can choose one of them.
@@ -19,7 +17,7 @@ This stand may be used for development purposes as well if you are focusing on a
 
 ## Basic usage
 
-In a case you would like to run Genesis Core on your own infrastructure, you can use the [basic guide](https://github.com/infraguys/genesis_core/wiki/BasicUsage) for more details.
+In a case you would like to run Genesis Core on your own infrastructure, you can use the [basic guide](./usage/basic_usage.md) for more details.
 
 ## 🚀 Development
 

docs/element_manager/service.md docs/em/service.mddocs/element_manager/service.md renamed to docs/em/service.md

@@ -0,0 +1,69 @@
+# Certificates
+
+Certificates are a part for the Secret Manager service. The service allows to issue and manage certificates, store them in specified storage and use them for different purposes.
+
+The current implementation only support `dns_core` method (provider) to issue and manage certificates. This method supposed DNS challenges via Core DNS that is
+available from the internet.
+
+Examples:
+
+```bash
+curl --location 'http://10.20.0.2:11010/v1/secret/certificates/' \
+--header 'Content-Type: application/json' \
+--header 'Authorization: Bearer MY_TOKEN' \
+--data-raw '{
+    "name": "my-cert",
+    "project_id": "00000000-0000-0000-0000-000000000000",
+    "method": {
+        "kind": "dns_core"
+    },
+    "constructor": {
+        "kind": "plain"
+    },
+    "email": "user@genesis-core.tech",
+    "domains": ["test0.cdns.genesis-core.tech"]
+}'
+```
+
+The main fields are:
+
+- **name** - name of the certificate.
+- **project_id** - it's a project the certificate belongs.
+- **method** - the method (provider) to issue and manage the certificate.
+- **constructor** - In the context of the certificates, the constructor object creates and stores the certificate. The `plain` means create and store in the plain format.
+- **email** - the email address to use for the certificate.
+- **domains** - the list of domains to use for the certificate.
+
+Also it's possible to specify domains with wildcards.
+
+```bash
+curl --location 'http://10.20.0.2:11010/v1/secret/certificates/' \
+--header 'Content-Type: application/json' \
+--header 'Authorization: Bearer MY_TOKEN' \
+--data-raw '{
+    "name": "my-cert",
+    "project_id": "00000000-0000-0000-0000-000000000000",
+    "method": {
+        "kind": "dns_core"
+    },
+    "constructor": {
+        "kind": "plain"
+    },
+    "email": "user@genesis-core.tech",
+    "domains": ["*.test1.cdns.genesis-core.tech", "test1.cdns.genesis-core.tech"]
+}'
+```
+
+## Methods / Providers
+
+The Genesis Core supports the following methods / providers to issue and manage certificates:
+
+### dns_core
+
+The `dns_core` provider allows to issue and manage certificates via Core DNS. It means Core DNS service should be available from the internet to accept ACME challenges. The main logic to communicate with Let's Encrypt is implemented in the [GCL CertBot plugin](https://github.com/infraguys/gcl_certbot_plugin) look at it for more information but the main steps are:
+
+- Create or get private client key.
+- Initiate a client with the key.
+- Request a certificate for domains.
+- Pass the DNS challenge.
+- Some final preparation.

docs/index.md

@@ -0,0 +1,185 @@
+# Basic usage
+
+This guide describes how to install and use Genesis Core on existing infrastructure. The existing infrastructure can be a local machine or several servers.
+It's assumed Linux(Ubuntu) is used as the OS on your machines.
+
+## Requirements
+
+Before you install and use genesis core you need to install several requirements:
+
+### Packages
+
+Install necessary packages:
+
+#### Ubuntu
+
+Install packages
+
+```bash
+sudo apt update
+sudo apt install qemu-kvm qemu-utils libvirt-daemon-system libvirt-dev mkisofs -y
+```
+
+Add user to group
+
+```bash
+sudo adduser $USER libvirt
+sudo adduser $USER kvm
+```
+
+### Libvirt
+
+Create a libvirt storage pool or use `default` if it already exists.
+
+Check pools:
+
+```bash
+sudo virsh pool-list --all
+```
+
+Create a new pool if no one exists or you would like to use another one.
+
+```bash
+sudo virsh pool-define-as default dir --target "/var/lib/libvirt/images/"
+sudo virsh pool-build default
+sudo virsh pool-start default
+sudo virsh pool-autostart default
+```
+
+Check the status with virsh-info:
+
+```bash
+sudo virsh pool-info default
+```
+
+## Installation
+
+The simplest way to install Genesis Core is to get a prebuilt virtual machine image with all necessary dependencies. Take the the [latest image here](http://repository.genesis-core.tech:8081/genesis-core/latest/genesis-core.qcow2).
+
+### Local machine / Development
+
+Install Genesis DevTools first. For more information about it, see [Genesis DevTools](https://github.com/infraguys/genesis_devtools). The devtools allows to build and run the genesis core locally. Since we already downloaded the latest image, we don't need to build it from scratch but if you need to build an image from source look at the [instructions here](https://github.com/infraguys/genesis_devtools?tab=readme-ov-file#build).
+
+The devtools has `bootstrap` command that will start the Genesis Core locally.
+
+```bash
+genesis bootstrap -i genesis-core.raw -m core
+```
+
+For more information about the `bootstrap` command, see [instructions here](https://github.com/infraguys/genesis_devtools?tab=readme-ov-file#bootstrap).
+
+The installation is ready at address `10.20.0.2` but at this time it's not very useful since it cannot run any workload. Let's add a local machine as a hypervisor to solve this problem.
+
+#### Get admin token
+
+Before we can add a hypervisor, we need to get an admin token. The command to get the token is:
+
+```bash
+curl --location 'http://10.20.0.2:11010/v1/iam/clients/00000000-0000-0000-0000-000000000000/actions/get_token/invoke' \
+    --header 'Content-Type: application/x-www-form-urlencoded' \
+    --data-urlencode 'grant_type=password' \
+    --data-urlencode 'username=<YOUR_ADMIN_USERNAME>' \
+    --data-urlencode 'password=<YOUR_ADMIN_PASSWORD>' \
+    --data-urlencode 'client_id=GenesisCoreClientId' \
+    --data-urlencode 'client_secret=GenesisCoreClientSecret' \
+    --data-urlencode 'scope=' \
+    --data-urlencode 'ttl=86400'
+```
+
+The return value is `json` object with the `access_token` field. Copy the token and use it in the next steps.
+
+#### Add hypervisor
+
+##### Libvirt via SSH (preffered)
+
+Create key in genesis core's VM for root user:
+
+```console
+# ssh-keygen
+```
+
+Copy public key to hypervisor (recommended user is `ubuntu`)
+
+##### Libvirt via TCP connect
+
+We need to get access to libvirt via a tcp connection. By default the tcp connection is closed, so we need to enable it.
+
+**NOTE:** For development purposes we can use raw tcp connection. Don't use it in production.
+
+Edit libvirt configuration file `/etc/libvirt/libvirtd.conf`, add these lines;
+
+```bash
+listen_tcp = 1
+listen_addr = "0.0.0.0"
+auth_tcp = "none"
+```
+
+Run commands to enable libvirt tcp connection:
+
+```bash
+sudo systemctl stop libvirtd
+sudo systemctl enable --now libvirtd-tcp.socket
+sudo systemctl start libvirtd
+```
+
+Check the libvirt is listening the tcp socket:
+
+```bash
+sudo systemctl status libvirtd.service
+sudo systemctl status libvirtd.service | grep "libvirtd-tcp.socket"
+```
+
+##### Configure ZFS for storage
+
+It's recommended to create distinct dataset for zvols:
+
+```bash
+zpool create rpool ... # create pool itself
+zfs create -o volmode=dev rpool/disks
+virsh pool-define-as --name rpool --source-name rpool/disks --type zfs
+virsh pool-start rpool
+virsh pool-autostart rpool
+```
+
+##### Config hypervisor in core
+
+Add the machine as a hypervisor, replace `XXXX` with the token from the previous step.:
+
+```bash
+curl --location --globoff 'http://10.20.0.2:11010/v1/hypervisors/' \
+--header 'Content-Type: application/json' \
+--header 'Authorization: Bearer XXXX' \
+--data '{
+    "driver_spec": {
+        "driver": "libvirt",
+        "iface_mtu": 1500,
+        "network_type": "network",
+        "network": "genesis-core-net",
+        "storage_pool": "default", // for qcow
+        "storage_pool": "rpool", // for ZFS
+        "connection_uri": "qemu+tcp://10.20.0.1/system", // for TCP connect
+        "connection_uri": "qemu+ssh://ubuntu@10.20.0.1:22/system?no_verify=1", // for SSH connect
+        "machine_prefix": "dev-"
+    },
+    "avail_cores": 4,
+    "avail_ram": 4096,
+    "all_cores": 4,
+    "all_ram": 4096,
+    "status": "ACTIVE"
+}'
+```
+
+- storage_pool - name of the libvirt storage pool to use.
+- connection_uri - libvirt connection uri. If you need to add another machine use different ip address.
+- all_cores - total number of cores you would like to allocate to the hypervisor.
+- all_ram - total amount of ram you would like to allocate to the hypervisor.
+- avail_cores - use the same value as `all_cores`.
+- avail_ram - use the same value as `all_ram`.
+
+### Production
+
+The production installation guide will be added soon.
+
+## Usage
+
+Follow the [usage guide](https://github.com/infraguys/genesis_core/wiki/Usage) for more information.

docs/secret/certificates.md

@@ -19,6 +19,8 @@
 import ipaddress
 import logging
 import os
+import pwd
+import grp
 import sys
 import time
 import typing as tp
@@ -45,7 +47,8 @@
 
 LOG = logging.getLogger(__name__)
 LOG.setLevel(logging.INFO)
-GCTL_CFG_DIR = "/home/ubuntu/.genesis"
+USER = "ubuntu"
+GCTL_CFG_DIR = f"/home/{USER}/.genesis"
 SPEC_PATH = "/mnt/cdrom/spec.json"
 MANIFEST_PATH = "/mnt/cdrom/core.yaml"
 MAIN_SUBNET_UUID = sys_uuid.UUID("c910a7e1-61ae-4d56-bdd6-a59faa3cbda3")
@@ -242,7 +245,13 @@ def _install_core_manifest(spec: dict[str, tp.Any], core_element_name: str = "co
 
     # Create a configuration file for gctl
     os.makedirs(GCTL_CFG_DIR, exist_ok=True)
-    with open(os.path.join(GCTL_CFG_DIR, "genesisctl.yaml"), "w") as f:
+    # chown GCTL_CFG_DIR to ubuntu user
+    uid = pwd.getpwnam(USER).pw_uid
+    gid = grp.getgrnam(USER).gr_gid
+    os.chown(GCTL_CFG_DIR, uid, gid)
+
+    config_path = os.path.join(GCTL_CFG_DIR, "genesisctl.yaml")
+    with open(config_path, "w") as f:
         yaml.safe_dump(
             {
                 "schema_version": 1,
@@ -263,8 +272,10 @@ def _install_core_manifest(spec: dict[str, tp.Any], core_element_name: str = "co
             },
             f,
         )
+    os.chown(config_path, uid, gid)
 
-    os.system(f"genesis --config {GCTL_CFG_DIR}/genesisctl.yaml elements install {manifest_path}")
+    os.system("genesis autocomplete")
+    os.system(f"genesis --config {config_path} elements install {manifest_path}")
 
 
 def _init_secrets(spec: dict[str, tp.Any]):