Ground System and Remote Dev Environments

[ugo94490]

Hi everyone,

Recently we had several ideas for improvements with @ThibFrgsGmz and we wanted to post them here to get some feedback from the community

GDS (GroundSystem)

Currently, the cFS-GroundSystem provided with the framework is only a development tool.

We think it would be good to first migrate the GDS to a browser-based application.

This would allow to have a development environment closer to a "remote development environment".

For example with F Prime it is possible to run the framework in a container with GitPod (see nasa/fprime#788).

We thought of several others features that could be added to the GDS that would allow us to test our software created with cFS more easily.

For example the launch of validation script that would allow us to:

• to launch commands at specific time or time relative to previous command.
• define the expected telemetry value at a more or less precise range.
• generate test reports.

We have also thought it would be nice to be able to visualize the telemetry as a graph.
Either by creating this chart system from scratch or by using a tool like YAMCS / OpenMCT ?

We have seen that there was already a software that includes the different test functionalities mentioned above (COSMOS implemented in OpenSatKit), unfortunately this software is not a web application and therefore does not fit into a "remote development environment" initiative.

This new initiative seems to us very important as it allows to fully integrate cFS in a DevOps circuit.

It will allow all teams to have a "code anywhere" and "ready to code" environment which is essential for the operational team for example which does not necessarily manipulate all these tools / libs / languages.

They will only have to launch GitPod / CodeSpaces / Docker for example to test the software via this web interface.

We would be delighted to hear your opinions on these various improvements and to discuss with you in order to improve them or to add others

[stanislaw]

Quick comment on this specific point:

We have seen that there was already a software that includes the different test functionalities mentioned above (COSMOS implemented in OpenSatKit), unfortunately this software is not a web application and therefore does not fit into a "remote development environment" initiative.

Check out this: BallAerospace/COSMOS#1316.

[ugo94490]

Thank you very much I had not noticed this issue !

[astrogeco]

I love this idea and its something that I would like to have available for the cFS community. Unfortunately I have yet to see a web based ground system that can do both command and telemetry. I might be wrong but OpenMCT only seems to do the telemetry side, granted, I haven't spent much time with it. Once COSMOS 5 is out we'll have an option but I think having something "bare bones" like the current python ground system that runs natively on a web browser (maybe even serverless?) would be a huge advantage.

[ugo94490]

Actually, OpenMCT does not have a native command interface.
It only allows to visualize telemetry but there is a plugin that allows to link it to YAMCS in order to send commands.

COSMOS 5 seems to be a good option, moreover it will allow to "delegate" the GDS part. The contributors of the COSMOS repo are very active so the cFS GDS would stay permanently updated via a submodule (would replace the current cFS-GroundSystem submodule) and would have more and more functionality.

We think that some improvements can be made to COSMOS such as the creation of validation scenarios described abstractly in XML (like for EDS) which would seem to be easier to use for operational/production people.
Apart from that COSMOS looks really complete.

I think a temporary ground system even serverless would be a big advantage as it would allow to start creating a remote development environment now, and it will be updated later with the integration of COSMOS 5.

[ThibFrgsGmz]

A small addition regarding the validation scenarios / test cases described in XML that can be used by both developers and operational staff.

Another reason why we are thinking about XML is that an interesting implementation appears in this publication A Testing Framework for Critical Space SW, on page 5.

We think that a production person would handle an XML markup language more easily than a programming language, but maybe we are wrong.

[astrogeco]

I'm partial to YAML because it's easy to read

[jphickey]

IMO neither XML nor YAML is well-suited to describing something like a test case. It's a markup language (the "ML") not a sequence language.

The simple test cases can probably be described (where inputs and expected outputs are relatively fixed), but as soon as you get into test cases that require some type of conditional (if this, then that result), or if a result is relative (expected commandcounter = last commandcounter + 1) then a markup language falls apart.

Embeddable script languages like Lua offer a lot of flexibility in this regard. It "reads" pretty easily, and you can also embed snippets of code into other non-code documents. It's pretty simple in the basic use-cases, but its still a full blown scripting language when you need it. I like Lua.

[mprather]

We reviewed OpenMCT last year to see if it could be leverage within an existing application. At the time, it would not work well with other apps (e.g. ability to only use a few select classes/features was no there) - it was expecting to effectively be the app itself.

As for testing languages and YAML/XML... I haven't seen a good testing framework that solely relies on descriptive markup. I would recommend staying with a native code framework. This makes it much easier for development teams while leveraging the full power of the native language while minimizing the number of tools and languages needed to get the system working.

[astrogeco]

@ugo94490 on a side note, can we rename this topic to something like "Ground System and Remote Dev Environments" since GDS isn't a common acronym?

[ugo94490]

It's done !

[dmccomas]

The NASA/Goddard apps have test scenarios and test scripts. I would recommend using some of these as inputs to evaluate a potential validation technology/platform. I think you'll find, as Joe mentioned, that many of the scenarios require conditional logic. In my testing experience I found STOL, the scripting language used by Goddard ground systems, to be limiting. I was on one mission that used Python as the test scripting language and IMO our testing improved in terms of productivity and in the quality of test scripts because we could leverage python language features.

…

On Tue, Jul 20, 2021 at 3:43 PM Joseph Hickey ***@***.***> wrote: IMO neither XML nor YAML is well-suited to describing something like a test case. It's a markup language (the "ML") not a sequence language. The simple test cases can probably be described (where inputs and expected outputs are relatively fixed), but as soon as you get into test cases that require some type of conditional (if this, then that result), or if a result is relative (expected commandcounter = last commandcounter + 1) then a markup language falls apart. Embeddable script languages like Lua offer a lot of flexibility in this regard. It "reads" pretty easily, and you can also embed snippets of code into other non-code documents. It's pretty simple in the basic use-cases, but its still a full blown scripting language when you need it. I like Lua. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#307 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAMGKCB7QDIFS4I72KJHBJDTYXGWPANCNFSM5AVKHIFA> .

[ThibFrgsGmz]

Python is indeed an interesting choice. It is also the language used to describe validation/regression scenarios in the ESA TASTE framework (https://taste.tuxfamily.org/wiki/index.php?title=Technical_topic:_Advanced_testing_with_MSC_and_Python_scripts).

From your experience with COSMOS with OpenSatKit, is Python better than Ruby for this kind of scripting?

[astrogeco]

From my limited experience, I think Python's large userbase makes it a strong choice.

[dmccomas]

From a language perspective I think they both bring more than enough horsepower for testing so it's hard to state one is better than the other. Culturally I agree that Python' large userbase especially in the flight software community makes it a good choice.
I also think the test framework is very important. I was prototyping some COSMOS 4.* extensions in OSK but never matured them. COSMOS 5.0 is a complete departure from 4.* and the long term support for COSMOS 4.* is unknown. I'm in the process of refactoring OSK into multiple modules so COSMOS will not be part of the core module.

[avan989]

Is the idea to implement this as "Ground System As A Service", hosted on AWS for people to test? or is the idea to create an local web applications running on the local machine that acts as a ground system?

[ugo94490]

We were thinking more of a local web application, so that each person will be free to do what he wants with it (host it online or not, make changes, etc...)

[avan989]

I think it’s doable but it would be tough to find someone with enough time to implement the entire thing. Is the idea to get the cfs team at nasa to build it or something that the open source community would do?

[jphickey]

FWIW, much of the "local web service" concept is already implemented using EDS, if you want to go that route. The actual Software Bus - CGI bridge app exists but unfortunately wasn't part of the set that went through release authorization, so its NASA-only right now... but it is definitely doable and has been done.

[ThibFrgsGmz]

@avan989 We are not necessarily looking to assign this development task to anyone in particular but rather to see if others share our idea. If this is indeed the case, we could imagine that a detailed issue is created, and that a collaborator takes the action to initiate this work. We could imagine a partner company taking on an intern to initiate something for example.

[ThibFrgsGmz]

@jphickey The link between S-EDS and the "local wel service" concept is not so obvious to me, could you elaborate if you don't mind?
I understand that the S-EDS concept is more or less still at the Work In Progress stage on the CCSDS SOIS side and that feedback from the SAVOIR group is being done to improve the DoT. Once the discussions around the data descriptive format standard are completed, we could more easily interface a Ground System with a Space System.

What does the NASA-only application you mention actually do?

PS: what does the acronym CGI stand for, please? I have trouble seeing what it means.

[astrogeco]

I think the word "ground system" is throwing us all off. I wonder if there's a better word for what @ugo94490 is asking for, which, in my understanding, is a web-browser based tool to send commands and read telemetry during developers. I think the main advantage of this over our current Python tool is that it could be easy to containerize and portable.

[ThibFrgsGmz]

@jbohren-hbr Indeed, this is perhaps the highest priority barrier in that this type of standard provides an abstraction to detail the specification of a system's data types and the constraints associated with each. This allows a set of interactions between systems/components to be defined by thinking in terms of Service Data Units (SDUs) without having visibility into the implementation elements, including the programming language.

I think SOIS EDS is the best choice for two reasons that immediately come to my mind.

• The European SAVOIR-SAFI group working on standardizing functional interfaces for AOCS sensors and actuators specifies that device manufacturers shall provide an EDS used to generate the sensor/actuator driver, which in turn is used by the software to connect the device to the bus (see here https://essr.esa.int/project/osra-onboard-software-reference-architecture accessible only to Europeans if I remember correctly).
• The International Deep Space Standard for Software, specifically ISwSIS, specifies the use of SOIS EDS for onboard interface services (see here https://nasasitebuilder.nasawestprime.com/idss2/wp-content/uploads/sites/45/2020/10/software_baseline_final_9-2020.pdf).

I don't really know why the adoption of SOIS EDS is so slow.

• Is there a lack of user-friendly tools in Python for example, easy to use and to integrate in a buildchain?
• Does the DoT not meet their needs because they don't cover all the mission-specific requirements?
• Do they prefer JSON or YAML to XML for reasons of readability for example?
• Is it a question of technological heritage and do companies prefer to reuse their proprietary tools?

[jbohren-hbr]

I don't really know why the adoption of SOIS EDS is so slow.

I think right now a challenge is (a) making the existing tooling accessible, and (b) standardizing an invocation method of that tooling for a cFS system.

Right now, even the EdsLib documentation reads:

this tool is generally not intended to be executed manually [ref]

@jphickey's example integration of NASA's EdsLib with cFS [ref] shows how EdsLib can be integrated with the cFS buildsystem. That example fully generates all of the platform and message-specific headers from EDS xml files.

Unfortunately, the implementation is so tightly coupled with the cFS sources that anyone who started using it would have to maintain the tooling and additional headers themselves.

Maybe a good first step is creating a modularized version of the EDS support using EdsLib so that people can start using it for their platform configuration headers and app headers. This would be something you could drop into a cfs mission definition folder without having to modify the core modules.

[astrogeco]

This thorough writeup and thoughtful conversation completely encapsulate the cFS community's potential. I'm grateful for every bit of knowledge and insight from you all. I don't have any steps forward right now but I'm definitely using this thread to advocate for more support for the community from an organizational perspective.

[ThibFrgsGmz]

@jbohren-hbr In my opinion, the source generation tool should be totally decoupled from the cFS framework otherwise it will be difficult to be adopted by other space actors.

The tool should take a suite of SEDS with a source code template, defined with CookieCutter for example, to make it flexible and adaptable over time.

The developer would then be free to define the source code templates (C, C++, Ada, Rust...) that respect the coding/naming rules of his framework/company.

---

By the way, regarding the Remote Dev Env, we are completely in line with @astroesteban 's demonstration on cFS, it goes exactly in the same direction as we imagined/have sketched.

[jbohren-hbr]

@jbohren-hbr In my opinion, the source generation tool should be totally decoupled from the cFS framework otherwise it will be difficult to be adopted by other space actors.

Yeah, ideally there's a clear separation managed by a well-defined interface in the build infrastructure, and EdsLib is designed that way. I didn't mean to imply that EdsLib couldn't be used outside of cFS. However, the current instantiation of cfe which has been modified to leverage EdsLib is not backwards-compatible with the traditional non-EDS cFS API.

For example, EdsLib can (and should) be used to do more than just generate interface headers. It's also generating runtime sources and other facilities which address some common pitfalls around using cFS like message ID portability and target endainness management.

Currently, the build-time dependency graph for cfe when using EdsLib looks something like the following:

                               ┌──────────┐
                           ┌───┤eds2cfetbl│◄─────────────────────────────────────────────────────────────────┐
                           │   └──────────┘                                                                  │
                           │                                                                                 │
                           │   ┌──────────┐                                                                  │
                           │   │          │◄─────────────────────────────────────────────────────────────────┤
                           ├───┤missionlib│                                                                  │
                           │   │          │◄────────┐                                                        │
             ┌────────┐    │   └──────────┘         │                                                        │
             │ EdsLib │◄───┤                        │                                                        │   ┌───────┐
             └────────┘    │   ┌──────────┐         │                                                        ├───┤  cfe  │
                           └───┤ sedstool │◄────────┤                                                        │   └───────┘
                               └──────────┘         │ ┌───────────────────────┐    ┌────────────────────┐    │
                                                    ├─┤ cfe generated sources │◄───┤ cfe static sources │◄───┘
  ┌────────────────────┐      ┌───────────────┐     │ └───────────────────────┘    └────────────────────┘
  │ mission XML config │◄─────┤ cfe XML files │◄────┘
  └────────────────────┘      └───────────────┘

So the source generation tools (EdsLib and sedstool) are independent, but there is information in the mission EDS XML configuration which influences array sizes and other parameters in the cfe EDS XML files, which are managed within the cfe sources, and are finally used to generate headers and source files. Then there is a set of buildtime and runtime facilities like missionlib which are necessary to build and run cfe with EDS support. Finally, eds2cfetbl is needed by any cfe apps which generate tables.

In this way, even though EdsLib can be used outside of cfe, the current EDS-aware version of cfe can't easily be used without EdsLib. We're interested in helping address this challenge to make EDS utilization in cFS more accessible and less risky for adoption by the community.