Repository class refactoring by arthurmco · Pull Request #95 · viniciusferrao/cloysterhpc

arthurmco · 2025-01-27T14:11:03Z

This is a HUGE refactoring adding some structure to the code base (namespace and folders) and removing old RepoManager code

Generalized RepoManager over the repository type and the runner,
Removed RepoManger and Runner from cluster
Replace the old implementation of RepoManager with the new implementation
Remove the dependency Runner from Cluster, if something is depending on the cluster's runner to run commands we'll find out, this something should be running commands on its own
Added getRepoManager and getRunner for retrieving the respective singletons
I added namespaces cloyster::models and cloyster::services to start splitting the code according to the model-view-presenter architecture
I moved the files so they correspond to their namespaces
Refactored RepoManager to use glibmm KeyFile for parsing
Refactored RepoManager public API
Refactored RepoManager implementation
Removed XCAT code from repo manager
Added KeyFile implementation hiding glibmm KeyFile (files.h/files.cpp)
Added concepts.h header for general concepts
Standarise dry run logging messages
Isolate presenter and viewer in its own cmake target
Add CATTUS_SKIP_DISK_CHECKSUM environment to skip checksum of the ISO
Add --test-conf-file for testing KeyFile implementation against arbitrary files (it loads and prints the file)

viniciusferrao · 2025-01-27T14:19:39Z

+
+void ELRepoFile::read()
+{
+    printf("reading\n");


viniciusferrao · 2025-01-27T14:25:50Z

+        m_repositories.push_back(std::move(repo));
+    }
+
+    for (const auto& r : m_repositories) {


I would avoid using only r as variable name. Usually we use it for well defined counters, like: i, j and k. Or for iterators, like it.

This is only code for toying with glibmm

In real life, this would not be like this

(This comment applies to every printf/fmt::format comment above)

viniciusferrao · 2025-01-27T14:27:43Z

Refactor the repository class based on @dhilst 's modeling

The modeling was done by the three of us. You included!

dhilst · 2025-01-27T14:34:15Z

+class ELRepoFile : public RepoFile<ELRepo> {
+private:
+    Glib::RefPtr<Glib::KeyFile> loadFile();
+
+public:
+    ELRepoFile(const std::filesystem::path& path)
+        : RepoFile(path)
+    {
+    }
+
+    void read() override;
+    void write() override;
+};


We can have an abstract RepoFile which extends the File, adds the m_repositories generic member, and with parse unparse methods, read is super read followed by parse and write is unparse followed by super write

std::filesystem::path is convertible to std::string or const char*. We must use it. It's the right API.

However the file class that be dependent on std::filesystem::path and not the derived one.

Just to be clear

parse loads data from the file into the m_repositories member

unparse loads the m_repositories member into the file

Also, @arthurmco, add overloads to parse and unparse which I/O into istringstream ostringstream instead of directly into the file so we can test without touching the disks. I don't know if this is possible because I don't know the interface of glibmm if it can load/unload keyfiles from strings, can it?

This is possible; glibmm can load keyhole data from a string.

dhilst · 2025-01-30T13:48:31Z

+std::vector<ELRepo> ELRepoFile::parse(const std::stringstream& ss)
+{
+    m_file = Glib::KeyFile::create();
+    m_file->load_from_data(ss.str().c_str());
+    return this->parseData();
+}


Cool, now

Add m_repositories member as a vector of ELRepo with a getter

The getter should load the data into the member if the member is empty and return at the end

Inside of ELRepo?

Inside ELRepoFile

dhilst · 2025-01-30T13:50:52Z

+struct ELRepo {
+    std::string group;
+    std::string name;
+    std::optional<std::string> baseURL;


dhilst · 2025-01-30T14:04:20Z

+    auto get_optional_string
+        = [this](auto group, auto key) -> std::optional<Glib::ustring> {
+        return m_file->has_key(group, key)
+            ? std::make_optional(m_file->get_string(group, key))
+            : std::nullopt;
+    };


I don't see why this captures this, anyway, move it to functions.cpp

dhilst

The API looks good, there are a few points

Make sure to remove the prints
Rename the ELReo to ELCloneRepo
Remove the PR from draft

viniciusferrao

Another issue that I've observed. The virtual classes should be on its own files. This needs to be fixed.

viniciusferrao · 2025-01-30T13:54:28Z

+
+std::vector<ELRepo> ELRepoFile::parse()
+{
+    this->read();


Do we need "this"?

viniciusferrao · 2025-01-31T14:55:44Z

 void copyFile(std::filesystem::path source, std::filesystem::path destination);

+std::optional<Glib::ustring> readKeyfileString(Glib::RefPtr<Glib::KeyFile> file,
+    const std::string_view& group, const std::string_view& key);


std:string_view is already a pointer to a string. It should not be passed as const ref.

I didn't know that. I thought it was pointer+length. It will be fixed

viniciusferrao · 2025-02-01T02:03:58Z

 }

+std::optional<Glib::ustring> readKeyfileString(Glib::RefPtr<Glib::KeyFile> file,
+    const std::string_view& group, const std::string_view& key)


No need to be const ref.

See this comment

viniciusferrao

I ended up adding a lot of comments.

I think we should think a little bit more on this implementation.

viniciusferrao · 2025-02-07T01:09:08Z

The class name must match de filename. And also the #ifndef guards.

I would recommend using a namespace to avoid clashing.

viniciusferrao · 2025-02-07T01:15:30Z

+    virtual void read() { }
+    virtual void write() { }


This should be = 0? I can't find the implementation in other files, only the specialization. It should be pure virtual.

viniciusferrao · 2025-02-07T01:19:22Z

+    {
+    }
+
+    virtual void read() { }


I think read will not modify the state of the object right? Should it be const?

viniciusferrao · 2025-02-07T01:26:31Z

+    virtual void read() { }
+    virtual void write() { }
+
+    virtual ~GenericFile() = default;


Since you defaulted a destructor, AFAIK, the move operations are by default deleted. I think we should define the move operations:

GenericFile(GenericFile&&) = default; GenericFile& operator=(GenericFile&&) = default;

viniciusferrao · 2025-02-07T02:06:11Z

+        repo.enabled = enabled;
+        repo.gpgcheck = gpgcheck;
+        repo.gpgkey = gpgkey;
+        repositories.push_back(std::move(repo));


emplace_back.

viniciusferrao · 2025-02-07T02:11:25Z

+}
+
+[[nodiscard]] const std::vector<ELCloneRepo>&
+ELRepoFile::getRepositoriesConst() const


Having the qualifier in the method name is a very bad approach.

viniciusferrao

That's a big one. There's some comments that were probably not from this PR originally, we should mark those as a BUG to be fixed later.

For everything else can you evaluate @dhilst? Don't be economic when naming things, you can abuse it. Avoids guessing. The compiler will optimize anyway.

viniciusferrao · 2025-03-12T21:19:16Z

+#ifndef CLOYSTER_CONCEPTS_H
+#define CLOYSTER_CONCEPTS_H


Suggested change

#ifndef CLOYSTER_CONCEPTS_H

#define CLOYSTER_CONCEPTS_H

/*

* Copyright 2025 Vinícius Ferrão <vinicius@ferrao.net.br>

* SPDX-License-Identifier: Apache-2.0

*/

#ifndef CLOYSTERHPC_CONCEPTS_H_

#define CLOYSTERHPC_CONCEPTS_H_

viniciusferrao · 2025-03-12T21:19:57Z

+ * @brief IsParser<P, T> means: P can parse and unparse Ts from streams. The
+ * parsers are allowed to throw exceptions
+ */
+template <typename Parser_, typename I, typename O>


I is input and O is output? If yes I think we should use more descriptive names.

viniciusferrao · 2025-03-12T21:21:01Z

+ * The parsers are not allowed to throw exceptions, it must return errors of
+ * type E
+ */
+template <typename Parser_, typename T, typename E>


What is T and E?

Also the naming with _ at the end, is that a convention?

I removed this concept, anyway T is the parsed type and E the error type

viniciusferrao · 2025-03-12T21:21:20Z

+
+}
+
+#endif // CLOYSTER_CONCEPTS_H


Suggested change

#endif // CLOYSTER_CONCEPTS_H

#endif // CLOYSTERHPC_CONCEPTS_H_

viniciusferrao · 2025-03-12T22:09:39Z

+        { "AlmaLinux-8.8-x86_64-dvd.iso", "OracleLinux-R8-U8-x86_64-dvd.iso",
+            "rhel-8.8-x86_64-dvd.iso", "Rocky-8.8-x86_64-dvd1.iso",
            "Rocky-9.5-x86_64-dvd.iso" }) };


I think this should be better organized or generated since it's a pattern and not hardcoded like that. I know it's a problem from legacy code. But maybe this should be removed.

viniciusferrao · 2025-03-12T23:33:30Z

+        case OS::Platform::el8:
+        case OS::Platform::el9:
+        case OS::Platform::el10:
+            return "/etc/yum.repos.d/cloyster.repo";


I think this will explicit return as const char*. Should we force it to return an std::string?

viniciusferrao · 2025-03-12T23:33:51Z

+    std::vector<std::string> dependencies;
+
+    std::size_t version = osinfo.getMajorVersion();
+    std::string powertools = "powertools";


That logic is not good at all.

I didn't get it, how exactly it is not good?

viniciusferrao · 2025-03-12T23:34:35Z

+void RepoManager::enable(const std::string& repoid)
+{
+    if (cloyster::dryRun) {
+        LOG_WARN("Dry Run: Would enable repository {}", repoid);


Should all dry run statements be LOG_INFO instead? It's not a warning.

This is because I standardized all the dry-run logging messages to be warning and have Dry Run: as the prefix, for the sake of consistency

viniciusferrao · 2025-03-12T23:35:43Z

+            return std::vector<std::string>(
+                       { "-beegfs", "-elrepo", "-epel", "-openhpc",
+                           "-openhpc-updates", "-rpmfusion-free-updates" })


That's temporary?

It is not. These are the repositories enabled by default in EL distros, it comes from old/refactored code. Is it wrong?

* Add script to build rpms * Fix debug build * Link against host Glibmm using pkg-config * Extract glibmm from diskImage, move it to files.cpp --------- Co-authored-by: Daniel Hilst <daniel@versatushpc.com.br>

sonarqubecloud · 2025-03-17T17:15:48Z

Quality Gate failed

Failed conditions
20 Security Hotspots

See analysis details on SonarQube Cloud

dhilst · 2025-03-18T17:18:25Z

@viniciusferrao Changing the smart pointers in the postfix (for the client bus) requires refactoring and changing the IService interface. Replacing shared with unique pointers in the presenter requires a major refactoring too because these classes should not have ownership of these objects

I will address these problems in another PR

* Adding OFED option and Cluster singleton with global getter * Add singleton class * Use Singleton<T> class for other singletons * Make all Dry Run message LOG_INFO * Remove print and test option * Implement DOCA kernel driver installation into the netboot image * Fix OFED installation in headnode * Fix review comments * Generalize testing and fix createHTTPRepo function * Remove CATTUS_... environment variables in favor of --force + --skip flags * Rename BaseRunner to IRunner for consistency * Wrap magic_enum lib * Remove package_manager in favor of IOSService interface * Deduplicate code * Run clang-format * Remove hwinfo, cloyster/hardware.h and tests The tests are breaking because of the refactoring, I'll redo the tests after I finish it * Remove inifile in favor of KeyFile * Removing old repository code * Add comments and FIXME * Use boost to parse options * Run clang-format * Decouple message bus from implementors * Fix test command handling * Fix warning in options.h * Restores runtime behavior after refactoring (WIP) * Replace boost::program_options with CLI11 * Implement repository sanity checking logic in repos.cpp * WIP generalizing repository addition * Restore testing code infra * Add test on the CI * Move singletons initalization to its own file * [WIP] Add libbacktrace to the dependencies * Decouple I/O from repository config classes to make them testable * Remove stacktrace attempt There is no libstdc++exp.a in Rocklinux repositories, stacktrace requires it. * Continuing the repositories core logic implementation * Remove old repository installation for xCAT and OFED * Add test files * Add ScriptBuilder service * Update NFS with script builder * Refactor NFS to use ScriptBuilder for installation and image customization. * WIP Fixing repositories bugs after the revamp * Fixing image generation and bugs after revamp * Fix the tests --------- Co-authored-by: Daniel Hilst <daniel@versatushpc.com.br>

sonarqubecloud · 2025-07-02T14:22:01Z

Quality Gate failed

Failed conditions
22 Security Hotspots

See analysis details on SonarQube Cloud

This and #95 were big refactoring, a new RPM was generated and sent for acceptance testing, there is no point into review currently as it is now

WIP: Repository class refactoring

a01df8c

arthurmco requested a review from dhilst January 27, 2025 14:11

arthurmco self-assigned this Jan 27, 2025

viniciusferrao suggested changes Jan 27, 2025

View reviewed changes

dhilst reviewed Jan 27, 2025

View reviewed changes

Comment thread include/cloysterhpc/services/repo.h Outdated

dhilst reviewed Jan 27, 2025

View reviewed changes

Comment thread include/cloysterhpc/services/repo.h Outdated

dhilst reviewed Jan 27, 2025

View reviewed changes

Comment thread src/services/repo.cpp Outdated

arthurmco added 2 commits January 30, 2025 08:31

Add parse() and unparse() methods, fix more things

6fe6003

Add stringstream read specialization, as suggested by @dhilst

99f79ce

dhilst reviewed Jan 30, 2025

View reviewed changes

Comment thread include/cloysterhpc/services/repo.h Outdated

dhilst reviewed Jan 30, 2025

View reviewed changes

arthurmco added 3 commits January 31, 2025 09:38

Move some common functions to functions.h

3f0f219

Remove this->

37f8b60

Fix repo class

ab4ba11

dhilst approved these changes Jan 31, 2025

View reviewed changes

dhilst previously requested changes Jan 31, 2025

View reviewed changes

viniciusferrao suggested changes Feb 1, 2025

View reviewed changes

arthurmco added 2 commits February 3, 2025 10:30

Fix const ref and declaration name

25df8ed

Fix naming

de966e5

arthurmco requested a review from dhilst February 4, 2025 13:44

arthurmco marked this pull request as ready for review February 4, 2025 13:45

arthurmco added 2 commits February 4, 2025 12:37

Rename required files

4895e2e

fixup! Rename required files

a834f94

viniciusferrao suggested changes Feb 7, 2025

View reviewed changes

dhilst assigned dhilst and unassigned arthurmco Feb 7, 2025

Testing RepoManager refactoring

924dcb1

dhilst force-pushed the repo-class branch from 79954f1 to 924dcb1 Compare February 18, 2025 14:19

dhilst added 2 commits February 18, 2025 11:51

Run clang-format

e0121c6

Testing release build

c4623cb

dhilst force-pushed the repo-class branch from 169037c to c4623cb Compare February 19, 2025 19:23

dhilst marked this pull request as ready for review February 19, 2025 20:21

dhilst added 3 commits February 19, 2025 17:21

Run clang-format again

59d485e

Remove empty file

0023eaf

Remove commented code

9cefe91

dhilst changed the title ~~WIP: Repository class refactoring~~ Repository class refactoring Feb 19, 2025

dhilst force-pushed the repo-class branch from 398f732 to 49c5382 Compare February 20, 2025 03:31

Add RPMRepoManager sketch implementation

9d58572

dhilst force-pushed the repo-class branch from 49c5382 to 9d58572 Compare February 20, 2025 13:40

dhilst added 2 commits February 24, 2025 18:07

Replace old code with RPMRepositoryManager in repos.cpp

9072ce4

Remove unused code

17de642

dhilst force-pushed the repo-class branch 2 times, most recently from 9d0b02b to ecb7a02 Compare February 25, 2025 00:45

Save RPMRepositoryFile only once for multiple enable calls

a696419

dhilst force-pushed the repo-class branch from ecb7a02 to a696419 Compare February 25, 2025 01:08

Run clang-format

9e9a860

viniciusferrao previously requested changes Mar 12, 2025

View reviewed changes

RPM Build script && Use host glibmm (#96)

ae1436e

* Add script to build rpms * Fix debug build * Link against host Glibmm using pkg-config * Extract glibmm from diskImage, move it to files.cpp --------- Co-authored-by: Daniel Hilst <daniel@versatushpc.com.br>

dhilst mentioned this pull request Mar 24, 2025

RPM Build script && Use host glibmm #96

Merged

dhilst merged commit 163866b into master Jul 2, 2025
1 of 6 checks passed

-#ifndef CLOYSTER_CONCEPTS_H
-#define CLOYSTER_CONCEPTS_H
+/*
+ * Copyright 2025 Vinícius Ferrão <vinicius@ferrao.net.br>
+ * SPDX-License-Identifier: Apache-2.0
+ */
+#ifndef CLOYSTERHPC_CONCEPTS_H_
+#define CLOYSTERHPC_CONCEPTS_H_

	#endif // CLOYSTER_CONCEPTS_H
	#endif // CLOYSTERHPC_CONCEPTS_H_

Conversation

arthurmco commented Jan 27, 2025 • edited by dhilst Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

viniciusferrao commented Jan 27, 2025

Uh oh!

Uh oh!

dhilst Jan 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dhilst Jan 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

dhilst Jan 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dhilst left a comment

Choose a reason for hiding this comment

Uh oh!

viniciusferrao left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

viniciusferrao left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

arthurmco commented Jan 27, 2025 •

edited by dhilst

Loading

dhilst Jan 27, 2025 •

edited

Loading

dhilst Jan 27, 2025 •

edited

Loading

dhilst Jan 30, 2025 •

edited

Loading