Moon Rover - Synchronize cFS Time with Linux Host OS

[HLCHowland]

Hello,
I am working on a moon rover for CMU and am trying to figure out how to get the time in cFS to be synchronized with the time on our lander. The lander will act as a relay for our very small rover. The rover and the lander communicate through WiFi and the lander will keep the rover's Linux install time-synchronized by serving NTP. Every couple of minutes the rover will poll the lander's NTP server to keep itself in check, but I cannot figure out the best way to make sure that the cFS time is correct as well.

How do I synchronize the cFS time to that of the host Linux OS so that the time on the lander and the time in cFS are roughly the same?

Should I be doing this by just updating the STCF? This does not seem to be the right way, and I cannot find anything helpful in time.h or osapi-os-core.h. Also, where is the MET tracked on a Linux system?

[skliper]

Typical approach is to set CFE_PLATFORM_TIME_CFG_SOURCE and CFE_PLATFORM_TIME_CFG_SRC_TIME both true and then add an app that samples Linux OS time and calls CFE_TIME_ExternalTone and CFE_TIME_ExternalTime periodically.

For MET on Linux as long as CFE_PLATFORM_TIME_CFG_VIRTUAL is true, the MET is simulated via an internal timer. See CFE_TIME_Global.VirtualMET.

[HLCHowland]

Got it, thank you so much!

[9a3eedi]

In terms of design, would it make more sense to do this as an app, or as a PSP module that gets included only for a specific platform?

[2shaar2059]

@skliper Not sure I understand how I can use CFE_TIME_ExternalTone(). A couple of questions:

• What is the tone, and why does it need to be provided? How does cFS use the tone?
• What do you mean by "simulated via an internal timer"? I see hints in cFE's source code about updating the time state machine at a 1Hz "tone" (not sure what tone means), but presumably, the time is updated by asking the Linux OS for the current time. How is time simulated in that case if it is just tied to the OS?
• What happens if, for whatever reason, my custom time synchronization app fails to send a tone (maybe my time synchronization app thread hit a critical error, stalled, got restarted, preempted, etc.)? Will the MET lag for the duration that my time synchronization app is not functional?
• Related to the previous point: I tried printing the cFS time at a 1Hz rate to the console without changing the time through the cFE time API. Then, I sent the cFS process a SIGSTOP, waited for 64 seconds, CONTINUED the cFS process, and noted down the time it printed out once it continued. The time did not lag by 64 seconds even though it couldn't do the 1Hz tone updates. Why was the time able to keep up?
• Without me doing any explicit time synchronization or changing MET or STCF (either indirectly or directly through the API), can I expect the time difference between CFE_TIME_GetMET() and Linux's gettimeofday() (with appropriate microsecond to subsecond conversion) not to drift? I understand that cFE's time service periodically (1Hz) updates the MET using the Linux OS time, so I wouldn't expect any drift between the two. If this is the case, is it sufficient to note down the offset between the two when the cFS Flight Software process starts up and then add that offset after-the-fact to all data products my cFS Flight Software produced to get their timestamp as if Linux timestamped them in the first place?

Thanks!

[skliper]

FYI I'm not sure who the current time expert is for cFS, @dmknutsen may be able to help. I'm no longer directly supporting cFS, but here's a few quick answers to hopefully help out a little at least.

• What is the tone, and why does it need to be provided? How does cFS use the tone?

The "tone" is a very accurate pulse that marks the point at which the "time at tone is/was" message applies. It's typically provided by whatever system is providing time "truth" like a GPS or radio within a system that needs accurate "wall clock" time. Without a valid "tone" and "time at tone" message time services freewheels (basically just does the best it can w/ internal time). Typically the tone is sent at the second boundary of your "truth" time, and the message comes either before or after the tone by a few 100 ms (give or take). cFS uses the tone and associated time at tone to synchronize it's internal tracking of "wall clock" time with "truth".

• What do you mean by "simulated via an internal timer"? I see hints in cFE's source code about updating the time state machine at a 1Hz "tone" (not sure what tone means), but presumably, the time is updated by asking the Linux OS for the current time. How is time simulated in that case if it is just tied to the OS?

In a flight system there's often an FPGA or other source of Mission Elapsed Time (MET), which basically just counts up at 1 Hz. I think the PSP is where getting the real MET is implemented when available. If there isn't a MET, cFE can simulate it by just incrementing a counter at 1 Hz, but it's initialized at 0 when the software is started. Note there's both the "tone" and an internal 1Hz timer, both tied to interrupts... if the tone goes away, the 1 Hz timer is still being processed.

• What happens if, for whatever reason, my custom time synchronization app fails to send a tone (maybe my time synchronization app thread hit a critical error, stalled, got restarted, preempted, etc.)? Will the MET lag for the duration that my time synchronization app is not functional?

Time services will freewheel. The internal 1Hz timer that will continue to update, you just won't be synchronized w/ the time truth. What the MET does depends on how you are configured... if simulated I think it just increments based on the internal 1Hz timer.

• Related to the previous point: I tried printing the cFS time at a 1Hz rate to the console without changing the time through the cFE time API. Then, I sent the cFS process a SIGSTOP, waited for 64 seconds, CONTINUED the cFS process, and noted down the time it printed out once it continued. The time did not lag by 64 seconds even though it couldn't do the 1Hz tone updates. Why was the time able to keep up?

Time services uses CFE_PSP_GetTime. For linux the PSP is using the timebase_posix_clock. As long as the timebase is still running whenever cFS wakes up it'll use that time.

• Without me doing any explicit time synchronization or changing MET or STCF (either indirectly or directly through the API), can I expect the time difference between CFE_TIME_GetMET() and Linux's gettimeofday() (with appropriate microsecond to subsecond conversion) not to drift? I understand that cFE's time service periodically (1Hz) updates the MET using the Linux OS time, so I wouldn't expect any drift between the two. If this is the case, is it sufficient to note down the offset between the two when the cFS Flight Software process starts up and then add that offset after-the-fact to all data products my cFS Flight Software produced to get their timestamp as if Linux timestamped them in the first place?

If there is something updating the reference for gettimeofday like NTP or similar they could drift. timebase_posix_clock is the monotonic POSIX clock. TBH I don't really know if gettimeofday uses the same clock (or something in sync) for Linux in absence of external inputs, but I'd expect so.

Thanks!

[2shaar2059]

Thanks Jake. I went back and tried the following in my time synchronization app:

    CFE_TIME_SysTime_t linux_TAI_time = get_Linux_TAI_time();
    CFE_TIME_SysTime_t currentSTCF    = CFE_TIME_GetSTCF();
    CFE_TIME_SysTime_t currentMET     = CFE_TIME_GetMET();
    CFE_TIME_SysTime_t NewMET = CFE_TIME_Subtract(linux_TAI_time, currentSTCF);
    CFE_TIME_ExternalTone();
    CFE_TIME_ExternalMET(NewMET);

get_Linux_TAI_time() is a custom function that fetches the TAI time from Linux.

However, I am not able to build my code with this. I set CFE_PLATFORM_TIME_CFG_SOURCE, CFE_PLATFORM_TIME_CFG_SRC_MET, and CFE_PLATFORM_TIME_CFG_SERVER to true. But then I got the following error:

OUR_SOFTWARE_REPO/cfe/fsw/cfe-core/src/time/cfe_time_verify.h:140:6: error: #error Cannot define both CFE_MISSION_TIME_CFG_FAKE_TONE and CFE_PLATFORM_TIME_CFG_SOURCE as true!
     #error Cannot define both CFE_MISSION_TIME_CFG_FAKE_TONE and CFE_PLATFORM_TIME_CFG_SOURCE as true!

After setting CFE_MISSION_TIME_CFG_FAKE_TONE to false:

OUR_SOFTWARE_REPO/cfe/fsw/cfe-core/unit-test/time_UT.c: In function 'Test_External':
OUR_SOFTWARE_REPO/cfe/fsw/cfe-core/unit-test/time_UT.c:1346:23: error: 'CFE_TIME_TaskData_t {aka struct <anonymous>}' has no member named 'ClockSetState'; did you mean 'ClockSource'?
     CFE_TIME_TaskData.ClockSetState = CFE_TIME_SetState_NOT_SET;
                       ^~~~~~~~~~~~~
                       ClockSource
OUR_SOFTWARE_REPO/cfe/fsw/cfe-core/unit-test/time_UT.c:1361:23: error: 'CFE_TIME_TaskData_t {aka struct <anonymous>}' has no member named 'ClockSetState'; did you mean 'ClockSource'?
     CFE_TIME_TaskData.ClockSetState = CFE_TIME_SetState_WAS_SET;
                       ^~~~~~~~~~~~~
                       ClockSource
OUR_SOFTWARE_REPO/cfe/fsw/cfe-core/unit-test/time_UT.c:1365:22: error: 'CFE_TIME_TaskData_t {aka struct <anonymous>}' has no member named 'AtToneMET'
     CFE_TIME_TaskData.AtToneMET.Seconds = 10;
                      ^
OUR_SOFTWARE_REPO/cfe/fsw/cfe-core/unit-test/time_UT.c:1366:22: error: 'CFE_TIME_TaskData_t {aka struct <anonymous>}' has no member named 'AtToneMET'
     CFE_TIME_TaskData.AtToneMET.Subseconds = 0;
                      ^
OUR_SOFTWARE_REPO/cfe/fsw/cfe-core/unit-test/time_UT.c:1367:22: error: 'CFE_TIME_TaskData_t {aka struct <anonymous>}' has no member named 'AtToneDelay'
     CFE_TIME_TaskData.AtToneDelay.Seconds = 0;
                      ^
OUR_SOFTWARE_REPO/cfe/fsw/cfe-core/unit-test/time_UT.c:1368:22: error: 'CFE_TIME_TaskData_t {aka struct <anonymous>}' has no member named 'AtToneDelay'
     CFE_TIME_TaskData.AtToneDelay.Subseconds = 0;
                      ^
OUR_SOFTWARE_REPO/cfe/fsw/cfe-core/unit-test/time_UT.c:1369:23: error: 'CFE_TIME_TaskData_t {aka struct <anonymous>}' has no member named 'AtToneLatch'; did you mean 'ToneDataLatch'?
     CFE_TIME_TaskData.AtToneLatch.Seconds = 0;
                       ^~~~~~~~~~~
                       ToneDataLatch
OUR_SOFTWARE_REPO/cfe/fsw/cfe-core/unit-test/time_UT.c:1370:23: error: 'CFE_TIME_TaskData_t {aka struct <anonymous>}' has no member named 'AtToneLatch'; did you mean 'ToneDataLatch'?
     CFE_TIME_TaskData.AtToneLatch.Subseconds = 0;
                       ^~~~~~~~~~~
                       ToneDataLatch
OUR_SOFTWARE_REPO/cfe/fsw/cfe-core/unit-test/time_UT.c:1387:23: error: 'CFE_TIME_TaskData_t {aka struct <anonymous>}' has no member named 'ClockSetState'; did you mean 'ClockSource'?
     CFE_TIME_TaskData.ClockSetState = CFE_TIME_SetState_WAS_SET;
                       ^~~~~~~~~~~~~
                       ClockSource
OUR_SOFTWARE_REPO/cfe/fsw/cfe-core/unit-test/time_UT.c:1391:22: error: 'CFE_TIME_TaskData_t {aka struct <anonymous>}' has no member named 'AtToneMET'
     CFE_TIME_TaskData.AtToneMET.Seconds = 10;
                      ^
OUR_SOFTWARE_REPO/cfe/fsw/cfe-core/unit-test/time_UT.c:1392:22: error: 'CFE_TIME_TaskData_t {aka struct <anonymous>}' has no member named 'AtToneMET'
     CFE_TIME_TaskData.AtToneMET.Subseconds = 0;
                      ^
OUR_SOFTWARE_REPO/cfe/fsw/cfe-core/unit-test/time_UT.c:1393:22: error: 'CFE_TIME_TaskData_t {aka struct <anonymous>}' has no member named 'AtToneDelay'
     CFE_TIME_TaskData.AtToneDelay.Seconds = 0;
                      ^
OUR_SOFTWARE_REPO/cfe/fsw/cfe-core/unit-test/time_UT.c:1394:22: error: 'CFE_TIME_TaskData_t {aka struct <anonymous>}' has no member named 'AtToneDelay'
     CFE_TIME_TaskData.AtToneDelay.Subseconds = 0;
                      ^
OUR_SOFTWARE_REPO/cfe/fsw/cfe-core/unit-test/time_UT.c:1395:23: error: 'CFE_TIME_TaskData_t {aka struct <anonymous>}' has no member named 'AtToneLatch'; did you mean 'ToneDataLatch'?
     CFE_TIME_TaskData.AtToneLatch.Seconds = 0;
                       ^~~~~~~~~~~
                       ToneDataLatch
OUR_SOFTWARE_REPO/cfe/fsw/cfe-core/unit-test/time_UT.c:1396:23: error: 'CFE_TIME_TaskData_t {aka struct <anonymous>}' has no member named 'AtToneLatch'; did you mean 'ToneDataLatch'?
     CFE_TIME_TaskData.AtToneLatch.Subseconds = 0;
                       ^~~~~~~~~~~
                       ToneDataLatch
OUR_SOFTWARE_REPO/cfe/fsw/cfe-core/unit-test/time_UT.c:1411:23: error: 'CFE_TIME_TaskData_t {aka struct <anonymous>}' has no member named 'ClockSetState'; did you mean 'ClockSource'?
     CFE_TIME_TaskData.ClockSetState = CFE_TIME_SetState_WAS_SET;
                       ^~~~~~~~~~~~~
                       ClockSource
OUR_SOFTWARE_REPO/cfe/fsw/cfe-core/unit-test/time_UT.c:1415:22: error: 'CFE_TIME_TaskData_t {aka struct <anonymous>}' has no member named 'AtToneMET'
     CFE_TIME_TaskData.AtToneMET.Seconds = 10;
                      ^
OUR_SOFTWARE_REPO/cfe/fsw/cfe-core/unit-test/time_UT.c:1416:22: error: 'CFE_TIME_TaskData_t {aka struct <anonymous>}' has no member named 'AtToneMET'
     CFE_TIME_TaskData.AtToneMET.Subseconds = 0;
                      ^
OUR_SOFTWARE_REPO/cfe/fsw/cfe-core/unit-test/time_UT.c:1417:22: error: 'CFE_TIME_TaskData_t {aka struct <anonymous>}' has no member named 'AtToneDelay'
     CFE_TIME_TaskData.AtToneDelay.Seconds = 0;
                      ^
OUR_SOFTWARE_REPO/cfe/fsw/cfe-core/unit-test/time_UT.c:1418:22: error: 'CFE_TIME_TaskData_t {aka struct <anonymous>}' has no member named 'AtToneDelay'
     CFE_TIME_TaskData.AtToneDelay.Subseconds = 0;
                      ^
OUR_SOFTWARE_REPO/cfe/fsw/cfe-core/unit-test/time_UT.c:1419:23: error: 'CFE_TIME_TaskData_t {aka struct <anonymous>}' has no member named 'AtToneLatch'; did you mean 'ToneDataLatch'?
     CFE_TIME_TaskData.AtToneLatch.Seconds = 0;
                       ^~~~~~~~~~~
                       ToneDataLatch
OUR_SOFTWARE_REPO/cfe/fsw/cfe-core/unit-test/time_UT.c:1420:23: error: 'CFE_TIME_TaskData_t {aka struct <anonymous>}' has no member named 'AtToneLatch'; did you mean 'ToneDataLatch'?
     CFE_TIME_TaskData.AtToneLatch.Subseconds = 0;
                       ^~~~~~~~~~~
                       ToneDataLatch

I should mention our project uses cFE version "Development Build: 6.7.0+dev298" from late 2020 and we have not changed that for stability reasons. I would imagine there is still a way to get CFE_TIME_ExternalMET() working with our version of cFE.

Also, I see mention of a command that sets the MET or STCF here. Which is preferred for synchronization with an external source such as the linux host OS time:

1. CFE_TIME_ExternalTime() and CFE_TIME_ExternalTone()
2. sending a command with command code CFE_TIME_SET_TIME_CC and SC_TIME_SET_STCF_CC

Thank you.

[skliper]

What's preventing you from adopting the design suggested in the original answer? Your approach of updating MET externally does not follow recommended design practices, MET is expected to be monotonic and if it is external it's typically coming from a more accurate clock... it's not expected to jump. If you really need to update the way you are proposing instead of following the suggested answer, consider updating the spacecraft time correlation factor (although time services will do this for you if you follow the suggested answer).

[hjones2]

@skliper this rover is communicating with its lander over WiFi (using COTS WiFi components). It has no other radios. It must synchronize to the lander's time. Depending on what lander it ends up on, it does not necessarily have time at the tone available from the lander. Our default is that the lander will provide NTP.

Also, there are cases in which the computer running cFS should update its time from something other than the lander. If the system reboots outside of communications with its lander, a second computer onboard (not running cFS) has a more consistent reboot time, and it sends timestamped messages over serial to the computer running cFS. Using this time (based on second computer's last known time + consistent reboot time) should be accurate enough to use for ephemeris data so the system (moving slowly on the lunar surface) can use its sun sensor orientation information to inform its estimated pose when traveling back into lander communications range. I don't think that "freewheeling" time would give us what we need in this case.

[skliper]

Whatever the time source is, I suggest you use it to update the SCTF, not MET. You can use the technique described in the original response regardless of the source. Just write the app to get it from wherever needed, and set the cfe time using the commands described.