Allow finer grained code upload

[hnestmann]

cartridge level
code level

[jjuanmendoza]

Any idea on how to approach those ?
I'm trying to deploy only artifact deltas as their weight is becoming a problem and deployments are currently taking several hours to settle down.
Thanks

[jbachelet]

We could imagine a CLI statement that allows us to upload only specific folders within the cartridges (even a specific folder found across all cartridges from a specific "root" folder, i.e. all static/ folders).
This way, we would be able to send only the frontend changes without the backend for example, or only the images and fonts folders without any code changes (as those resources are the more time-consuming during the deployment).
We could also imagine a way to ignore some files/folders from the deployment (like the scss folder and the client folder from SFRA as those are only used for development purpose)

Finally, you can imagine that you deploy all images/fonts during the night, and only deploy the code folders during the normal process of deployment.

[jjuanmendoza]

No, basically I'm thinking on a diff at client level where changes are uploaded one by one (on a per file compiled/source/image basis even they are all transferred as single zip and later on positioned at the right folder) or removed (in case they've disappeared) from the code_version.
9X% of new deployments have minimal file differences in terms of deltas
It'd be transparent to the deployment (and maybe parametrisable if you want a full or incremental release). My aim was to find out current sfcc-ci support to achieve such behaviour.

[hnestmann]

@jjuanmendoza That has nothing to do with the ticket at hand :) which is about what Jordane explains.

I don't think it would help if you download the currently active code version to do your diff. So I think your build server has to rememeber the currently active code state and you diff the changes files from there.
I wouldn't put this in sfcc-ci though - rather an additional external script

[jjuanmendoza]

@hnestmann
right now after diffing (yes from the latest deployed version) I'm launching curl -x COPY or DELETE on a file basis to achieve something similar
after reading this issue title i thought it's implementation would allow me to leverage sfcc-ci for incremental deployments (my final goal)

incidentally, the use of a script goes without saying, the potential support that sfcc-ci shall provide me in order to achieve incremental deployments (like "allowing finer grained code upload" but their might be others) was my whole point.
I can move my comment wherever it makes more sense but this is a recurrent problem that big implementations are hitting and that custom scripts are trying to fix by leveraging some old/modern tools (prophet watcher, dw grunt-gulp, sfcc-ci, etc).

[hnestmann]

Found a quick hack You can deploy single cartriges by putting those in a code.zip which has a normal folder structure like code/your_cartridge. The server will merge them

[jjuanmendoza]

thanks @hnestmann !
correct me if I'm wrong cuz i walked that path some weeks ago and I might have overseen something important:
1.- such a merge behaviour comes directly from the zip command (https://linux.die.net/man/1/zip) search for "When given the name ..."
2.- for heavy code versions the whole .zip still needs to be uploaded even though only certain files from i.e. 2 cartridges have been added/changed/removed
Thanks

[hnestmann]

I think there is a misunderstanding

Say you have a sandbox with an existing 'code' version. Which contains cartridges 'app_storefront' and 'modules'. Now you use sfcc-ci to deploy to 'code' version with a zip that only contains 'code/plugin_serviceworker'. After the deployment, your sandbox will carry three cartridges.

• app_storefront
• modules
• plugin_serviceworker

[jjuanmendoza]

great thanks, that answers my second question.
what about if that new code version contains (in addition to code/plugin_serviceworker) a 'modules' cartridge with less files that the one in the existing 'code' version ?

[hnestmann]

That I didn't try out, but you can if you want.

I would assume it works like windows file system. By overwriting existing files, adding new files and keep files which are not included in the zip.

[jjuanmendoza]

right, that's why i was referring to std zip command's behaviour (as the default action is to add or replace zipfile entries from list)

That's one of the reasons why we need to add some script "glue" in order to use it for incremental ci (we need a preprocess to determine whether any files have been removed and if so either zip -d them or upload the whole new code version)

[hnestmann]

I don't connect the dots here it seems. The merging happens on unzipping on the sandbox. Not in the CI command directly.

[jjuanmendoza]

You're right.
Unfortunately for a true incremental ci we need to be able to track such file removals (like the ones i was raising) as part of a new version of the existent code version. That causes to slow down the whole process as we need to compare it against the deployed version. Unless we discard deltas with file deletions I haven't been able to workaround such an expensive operation (compare against deployed code version) before deploying a new/updated code version.

[hnestmann]

In my opinion, for developer sandboxes this is a less of an issue and for actual test instances I would always push my clients to have a CI server which deploys everything and does do an integrity check for everything that needs to be deployed.
However what you can do write a little script that provides an index file of a given cartridge and deploys that onto the sandboxes
deployinfo.json
{"version": "1.2.3",files":[{"modules/server/server.js": "size": "12kb", "hash": "abc","lastModified": "2019-04-08T10:06:35.279Z"}]

However this is way out of scope for this CLI tool, and should rather be in the build-suite which has a build orchestration focus. We are happy to receive contributions

[jjuanmendoza]

Thanks @hnestmann
Actually some deployments are taking us 4-5 hours (main part of it from transferring the heavy .zip file regardless changes/deletions/additions only affect 2-3% of the files) per pipeline hence we cannot go with the "deploy everything" path. The deployinfo.json is another solution we've been trying but has it's own downsides. Nevertheless I see your point and I agree this is out-of-scope for this CLI tool.
I'll let you know in case we end up with anything else beyond gluing/scripting sfcc-ci/build-suite