Skip to content

Add support for CelesTrak CSV instead of TLE#4890

Merged
alex-w merged 4 commits into
masterfrom
tle-to-csv
May 3, 2026
Merged

Add support for CelesTrak CSV instead of TLE#4890
alex-w merged 4 commits into
masterfrom
tle-to-csv

Conversation

@10110111
Copy link
Copy Markdown
Contributor

The 5-digit TLE field for NORAD id is going to be used up soon. This PR tries to implement the CelesTrak's CSV format that should be used for all downloads.

The current approach is to generate the TLE from the CSV entries, because it seems that too much code relies on this format. The NORAD ids in the generated output will be 0, and the actual id will be stored separately.

Currently the code is completely untested, I'm making a PR just to track the progress of this and possibly to discuss different approaches to the solution.

@alex-w alex-w added the subsystem: plugins The issue is related to plugins of planetarium... label Apr 28, 2026
@gzotti gzotti added this to the 26.2 milestone Apr 29, 2026
@gzotti gzotti moved this from Backlog to In progress in Plugin: Satellites Apr 29, 2026
@gzotti gzotti linked an issue Apr 29, 2026 that may be closed by this pull request
@gzotti gzotti added the importance: high Obvious error, very annoying, but no crash label Apr 29, 2026
@10110111 10110111 force-pushed the tle-to-csv branch 2 times, most recently from 63efc9c to ce31930 Compare April 30, 2026 02:56
@10110111 10110111 changed the title WIP adding support for CelesTrak CSV instead of TLE Add support for CelesTrak CSV instead of TLE Apr 30, 2026
Comment thread plugins/Satellites/src/Satellites.cpp
@10110111
Copy link
Copy Markdown
Contributor Author

So, this seems to work, except for the things I don't know how to check:

  • Satellites::add(const TleData& tleData) doesn't seem to ever be called, but I've removed a call to getSatIdFromLine2 from there, because sat id is no longer to be expected in a TLE (the ones generated from CSV are zero).
  • satOpStatusMap in Satellites::parseTleFile doesn't appear to ever be used from my checks, so I didn't implement anything similar in Satellites::parseCsvFile.
  • While it seems that TLE's satnum field is no longer fetched from anywhere but parseTleFile, I don't know how to know this for sure.

Also, parsing time seems to have increased, particularly due to the need to parse the Epoch field, which is now specified in the form like 2026-04-28T21:52:36.305184 instead of the old fractional days since year start. This may mean that the parsing should later be moved to a separate thread, so that it doesn't freeze the UI.

@10110111 10110111 marked this pull request as ready for review April 30, 2026 03:31
@github-actions github-actions Bot requested review from alex-w and gzotti April 30, 2026 03:31
@alex-w
Copy link
Copy Markdown
Member

alex-w commented Apr 30, 2026

So, this seems to work, except for the things I don't know how to check:

  • Satellites::add(const TleData& tleData) doesn't seem to ever be called, but I've removed a call to getSatIdFromLine2 from there, because sat id is no longer to be expected in a TLE (the ones generated from CSV are zero).

Please try press the button Add (Plus) in the GUI of Satellites plugin

  • satOpStatusMap in Satellites::parseTleFile doesn't appear to ever be used from my checks, so I didn't implement anything similar in Satellites::parseCsvFile.

This was a Celestrak's feature for TLE and apparently it removed now from other formats - probably this is time to drop support of it (including filters) :-/

  • While it seems that TLE's satnum field is no longer fetched from anywhere but parseTleFile, I don't know how to know this for sure.

Bad news :( The NORAD ID number is important data for satellite observers, but at least the COSPAR/NSSDC ID is here. Well, the search tool and infostrings should be updated too in this case. Plus apparently the TLE block in the plugin's GUI should be removed or, at least, hidden.

Also, parsing time seems to have increased, particularly due to the need to parse the Epoch field, which is now specified in the form like 2026-04-28T21:52:36.305184 instead of the old fractional days since year start. This may mean that the parsing should later be moved to a separate thread, so that it doesn't freeze the UI.

Please do it.

@10110111
Copy link
Copy Markdown
Contributor Author

  • While it seems that TLE's satnum field is no longer fetched from anywhere but parseTleFile, I don't know how to know this for sure.

Bad news :( The NORAD ID number is important data for satellite observers, but at least the COSPAR/NSSDC ID is here. Well, the search tool and infostrings should be updated too in this case. Plus apparently the TLE block in the plugin's GUI should be removed or, at least, hidden.

I didn't mean that the NORAD id is no longer available: it is, it's just taken from the CSV NORAD_CAT_ID field, rather than the TLE. What we should make sure is that the TLEs are never used as the source of NORAD id (except when the original information is downloaded as TLE, but in this case only in Satellites::parseTleFile and nowhere else).

What I did mean is that, since the TLE strings are available in Satellite::tleElements and are passed back and forth between Satellite, Satellites etc., I'm struggling to prove that they aren't ever used for extraction of the NORAD id.

@alex-w
Copy link
Copy Markdown
Member

alex-w commented Apr 30, 2026

Aha, OK, the new behavior is fine for me. Please check unit tests - some methods was added for UT

@10110111
Copy link
Copy Markdown
Contributor Author

10110111 commented May 2, 2026

Looks like CelesTrak is becoming more and more restrictive. Not only does it forbid downloading Starlink and Active GROUPs more than once in 2 hours (see the discussion at the bottom of this page), but it also has banned my IP address for two hours due to my repeated once in a few minutes updates of other groups from Stellarium for debugging:

403 - Forbidden: Access is denied.
Access from IP address XXX.XXX.XXX.XXX was blocked for celestrak.org/NORAD/elements/gp.php?GROUP=weather&FORMAT=CSV.
We have detected excessive downloads for files in the /NORAD/elements directory and access has been temporarily blocked. Access will be automatically restored once the excessive downloads have ceased for 2 hours.

Please note that orbital data files are only checked for updates every 2 hours and most orbital data only updates 2-3 times a day (or less). Please check your scripts to ensure they are operating properly.
Please contact ts.kelso@celestrak.org if you have questions.

In fact, they not only blocked me from access to the elements, but even to the docs on data formats and to the very discussion of the limits (see the first link above).

@10110111
Copy link
Copy Markdown
Contributor Author

10110111 commented May 2, 2026

So with the manual parsing of the CSV epoch parsing times went back to sensible: 21 s → 0.39 s (though in the future Satellites and Exoplanets should still move their parsing etc. to separate threads: GUI thread is not meant for this).

10110111 added 4 commits May 3, 2026 15:09
This patch also simplifies the code by early returning and avoiding
redundant file close/remove operations that will be done by the
destructor anyway.
Copy link
Copy Markdown
Member

@alex-w alex-w left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you very much!

@alex-w alex-w merged commit 2ca7ce5 into master May 3, 2026
28 of 29 checks passed
@alex-w alex-w deleted the tle-to-csv branch May 3, 2026 12:08
@github-project-automation github-project-automation Bot moved this from In progress to Done in Plugin: Satellites May 3, 2026
@alex-w alex-w added the state: published The fix has been published for testing in weekly binary package label May 5, 2026
@github-actions
Copy link
Copy Markdown

github-actions Bot commented May 5, 2026

Hello @10110111!

Please check the fresh version (development snapshot) of Stellarium:
https://github.com/Stellarium/stellarium-data/releases/tag/weekly-snapshot

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

importance: high Obvious error, very annoying, but no crash state: published The fix has been published for testing in weekly binary package subsystem: plugins The issue is related to plugins of planetarium...

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

Add support for OMM format for orbits of satellites

3 participants