Implement core process using Rust

[AllanOricil]

Have you considered implementing the core logic using Rust and then create native binaries for each platform and architecture?

Maybe a very simple implementation to test some scenarios to estimate performamce gains, memory footprint and cpu usage would me interesting to validate if this makes sense. Then multiply this estimate by the number of jobs that run in a period of time using sgd to estimate the total amount of memory, cpu and time saved.

You could still keep this plugin to ease distribution through sf cli. It would be used to parse arguments and call the native binary

sf cli (js) -> plugin (js) -> native binary (rust)

[scolladon]

I've never thought about it.
I wonder what would would be the performance gain?
Is there project interested in those performance gain?

For context :

• This plugin is not performance critical
  • The time spent in deployment is far more important than the time spent running the plugin
  • Some project are no longer deployable in full because of metadata deployment size limitation, making this plugin useful
• It is a portable plugin, running in multiple environment (node version, os version, git version) and in CI context, which is easy with node and sf cli
• It is an open source, free plugin, I'm not paid for that even if it is useful and whatever time I spend on it, whatever its quality and its stability

For information :
This is where the plugin is the less fast :

• IO
  • Reading from the terminal (spawning git commands)
  • Reading files (sfdx-project.json, metadata files containing sub metadata)
  • Writing files (copying increment metadata content)
• xml parsing (for profile, translation, customlabels, permissionset, sharingrules, etc)
  The other aspects of the plugin are "ok" performance-wise (even if I'm pretty sure rust would beat it at breakfast even for those aspects)

[AllanOricil]

I wonder what would would be the performance gain?
Is there project interested in those performance gain?

It would be valuable to collect metrics while consumers are running the core functionality of this plugin. Could you, or someone else that is interested to know the answer, set up a Sentry account, or a similar tool, to start gathering telemetry for each execution? Ideally, we should capture metrics for each significant part of the process.

Some key areas where I believe it’s important to collect metrics include:

• XML and JSON parsing
• Git operations
• I/O operations
• File system interactions

By gathering this data over a period of time, we can more confidently assess whether optimizing these areas is worthwhile. Depending on the scale—such as the number of jobs processed or the average volume of operations per job—improving the efficiency and speed of this automation could make a significant impact.

Some project are no longer deployable in full because of metadata deployment size limitation, making this plugin useful

"Edited: I made a mistake"
Yes, that makes sense. But some projects still perform a deploy validation based on the diff between LATEST_RELEASE_TAG..HEAD every now and then to prepare for the next release. I remember I worked on a project that this type of deployment would run once a week (manually) or on every merge to main to ensure it stays green before the release day.

It is a portable plugin, running in multiple environment (node version, os version, git version) and in CI context, which is easy with node and sf cli

The goal of this proposal is that nothing must change for consumers using the sdg plugin. The distro package of your plugin would contain the binary for the architecture of the consumer, as well as the necessary js build of the sf plugin. The only difference is that the core part would be execute by the binary, with native machine code, instead of using js. The js part would just hand the cli parameters to the binary.

It is an open source, free plugin, I'm not paid for that even if it is useful and whatever time I spend on it, whatever its quality and its stability

The quality and stability of the sgd plugin is over 9000, really. You and the other maintainers are doing an amazing job, really. It is just an Idea I had based on what I've been seeing other tools doing. For example, rollup builds are already extremely fast, however the rollup core team is reimplementing it using Rust. They are doing it because at scale it makes a ton of sense. Of course the order of magnitude of the number of builds with rollup are way bigger than salesforce diff deployments, but maybe this smaller amount of jobs could still be relevant. That is the hypothesis.

[scolladon]

I have a feature to add a logger.

I'll add an acceptance criteria to check it can logs performance information (anonymously, without customer data, containing just metrics) inside an external system (I'll need to spike which one to use) for some specific part of the application.
I need to find a way for customer to optout from this telemetry sending (via an environment variable and a parameter probably).

I don't know when I'll be able to do that, neither what problems it will show.
Thanks you for this suggestion, we'll see where it leads.

[AllanOricil]

https://github.com/AllanOricil/nrg-cli/blob/main/src/telemetry.ts

This is a very simple example of telemetry module using sentry I added to a CLI I created. It disables the module globally in the whole application based on an env variable.