GIT error on push

[fnagel]

Since a few weeks I receive following error via the GitHub hook settings:

{
    "messages": [
        "The following errors were reported:",
        "Git command \u0022~\/.toast\/armed\/bin\/git clone --depth 1 --single-branch --branch '5.0.0' 'https:\/\/github.com\/fnagel\/t3extblog.git' \/var\/www\/html\/gizzle\/tmp\/a019d1ccea4f0f98f18f3ea65f44b5ec3677c90c\/t3extblog\u0022 failed! Code 128, Message was: \u0022\u0022 (0)\n"
    ]
}

Most of the time, the TER release works as expected no matter what, but sometimes I need to hit "redeliver" a few times.

I've updated GIT on that server and check the file permissions. Manual GIT clone commands work without issues.

Any ideas why this is happening?

[NamelessCoder]

Error 128 from git means "Git did not exit cleanly" can have many different causes. The directory that is created (commit hash folder in tmp) should contain the state of the repository at the commit that dispatched the web hook event - perhaps you can investigate there next time this happens, before you redeliver the event?

Side question: when this happens, is the commit that dispatched the event, a force-push?

Side question 2: when this happens, I assume it's when you publish a new tag. If so, is the commit that the tag references already pushed - or do you push + push tags together? Could help to do that in two separate actions...

[fnagel]

Thanks for your quick answer!

The directory that is created (commit hash folder in tmp) should contain the state of the repository at the commit that dispatched the web hook event

Already did and the directory contains a valid copy of the repo at the wanted commit.

Side question: when this happens, is the commit that dispatched the event, a force-push?

No. I'm sure as the release I did this morning was definitely not a force push.

Side question 2: when this happens, I assume it's when you publish a new tag. If so, is the commit that the tag references already pushed - or do you push + push tags together?

I normally push both together, the commit and the tag. Never had any issues with that before. But I could try to push that in two steps next time.

I've updated the code base (I was on a8c0382, missing three commits) and updated the vendors. That probably helps too...

[NamelessCoder]

I always push the commit first (then wait for CI to verify) and then publish the tag - I don't know if that makes any difference but I've never had to redispatch requests except in cases where TER was temporarily down/broken.

There is one other thing to check: if this happens at times where you've done a lot of pushes, the git client that runs in the gizzle installation may hit the request limiter on GitHub - and when you try manually or redeliver, the limit may have expired. I think there should be a specific error message when that happens, but a lot can change in three years ;)

[fnagel]

Just tried it again with the tag pushed separately and got this in the Webhook UI: We couldn’t deliver this payload: Service Timeout. Pushed "redeliver" and got the OP error again. This time I did not push "redeliver" more than once but waited a little longer. A few minutes later the release was public on TER. Seems, despite the error, the actual deployment works.

[fnagel]

Pushed another small (file count and overall size) extension (this time tag and commits in a single push) and it worked without issues.

[fnagel]

Pushed another not so tiny extension (again: tag and commits in a single push) and it worked without issues. Mhhh, seems I need to monitor this a little more...

[fnagel]

Pushed a big extension (beautyofcode, comes with two JS libs -- so quite a lot files) in a single push and got a new error:

Skipped html/gizzle/tmp/c66186ce91f7b31e03cb6e68acb44226484d2159/beautyofcode/ext_emconf.php
Skipped html/gizzle/tmp/c66186ce91f7b31e03cb6e68acb44226484d2159/beautyofcode/Resources
Skipped html/gizzle/tmp/c66186ce91f7b31e03cb6e68acb44226484d2159/beautyofcode/ext_conf_template.txt
Skipped html/gizzle/tmp/c66186ce91f7b31e03cb6e68acb44226484d2159/beautyofcode/Documentation
Skipped html/gizzle/tmp/c66186ce91f7b31e03cb6e68acb44226484d2159/beautyofcode/Configuration
Skipped html/gizzle/tmp/c66186ce91f7b31e03cb6e68acb44226484d2159/beautyofcode/Tests
Skipped html/gizzle/tmp/c66186ce91f7b31e03cb6e68acb44226484d2159/beautyofcode/ChangeLog
Skipped html/gizzle/tmp/c66186ce91f7b31e03cb6e68acb44226484d2159/beautyofcode/ext_localconf.php
Skipped html/gizzle/tmp/c66186ce91f7b31e03cb6e68acb44226484d2159/beautyofcode/Classes
Skipped html/gizzle/tmp/c66186ce91f7b31e03cb6e68acb44226484d2159/beautyofcode/bower.json
Skipped html/gizzle/tmp/c66186ce91f7b31e03cb6e68acb44226484d2159/beautyofcode/class.ext_update.php
Skipped html/gizzle/tmp/c66186ce91f7b31e03cb6e68acb44226484d2159/beautyofcode/LICENSE.txt
Skipped html/gizzle/tmp/c66186ce91f7b31e03cb6e68acb44226484d2159/beautyofcode/ext_tables.php
Skipped html/gizzle/tmp/c66186ce91f7b31e03cb6e68acb44226484d2159/beautyofcode/README.md
Skipped html/gizzle/tmp/c66186ce91f7b31e03cb6e68acb44226484d2159/beautyofcode/composer.json
{
    "messages": [
        "The following errors were reported:",
        "Extension contains no files. (0)\n"
    ]
}

After hitting the "redeliver" button multiple times (getting the the OP error again, removing the whole tmp folder, hit it again, repeat) but it never worked. First time I needed to use the zip upload on the TER website again :-/ Strange enough, last time I released that extension it worked without issue (even I already had the OP issues at that time).

So, this issue is still not really reproducible.

[fnagel]

Oh boy -- it's getting even more strange :-D

Pushed another extension. Got an "We couldn’t deliver this payload: Service Timeout" error with no response at all. Checked the server: Repository folder has been created and (a few minutes later) the TER release is public. No "redeliver" button used.

Checked error log and found this:
[Fri Jan 03 14:37:33 2020] [warn] [client 2a02:908:1961:68a0:680e:4515:1210:f6a0] mod_fcgid: stderr: PHP Warning: hash_hmac(): Unknown hashing algorithm: in /html/gizzle/fluidtypo3-gizzle/vendor/namelesscoder/gizzle/src/Payload.php on line 210

[NamelessCoder]

I assume you use a self-hosted version of gizzle for this. Could you try the following:

• Raise time limit allowed for FCGI communication (separately from max_execution_time)
• Ensure server has a GitHub token or uses SSH with a public key associated with an account
• For the "skipped XYZ" thing, make sure git-ignore doesn't contain a bad wildcard
• For the "unknown hash" thing, make sure the request contains X-Hub-Signature header with a value of something like sha1=1234567890abcdef and that your server supports the algorithm indicated therein (expected default is sha1)

[NamelessCoder]

HMAC validation business logic: https://github.com/NamelessCoder/gizzle/blob/f1980299d9520e13b76c8ddaed8511e81f588d8d/src/Payload.php#L207

[fnagel]

I assume you use a self-hosted version of gizzle for this.

Yes. And everything worked fine the last few years. Issues tarted a few month ago but I did not change anything at all. But it is possible my hosting provider updated something.

Raise time limit allowed for FCGI communication (separately from max_execution_time)

Uhhh, I'm not really sure how to do this. I've added following to my fcgi-bin\php-fcgi-starter file:

PHP_FCGI_MAX_REQUESTS=10000
export PHP_FCGI_MAX_REQUESTS

Limits in my php.ini:

max_execution_time = 3600

max_input_time = 3600
max_input_vars = 3000
memory_limit = 256M
post_max_size = 300M
upload_max_filesize = 290M
max_input_nesting_level = 64

Ensure server has a GitHub token or uses SSH with a public key associated with an account

That is the case.

For the "unknown hash" thing, make sure the request contains X-Hub-Signature header with a value of something like sha1=1234567890abcdef and that your server supports the algorithm indicated therein (expected default is sha1)

Not that easy to test the request headers but sha1 is definitely available. Tested using hash_algos().

For the "skipped XYZ" thing, make sure git-ignore doesn't contain a bad wildcard

No changes here and looks good IMO:

vendor
.idea
.secret
.typo3credentials
.token
composer.lock
web/.htaccess

...and thanks for your help! I wish you a good new year!