Applying `phase/solved` and then closing (merging) a pr still results in a comment

[wooorm]

Initial checklist

• I read the support docs
• I read the contributing guide
• I agree to follow the code of conduct
• I searched issues and couldn’t find anything (or linked relevant results below)

Affected packages and versions: TODO

Steps to reproduce

remarkjs/remark#851 (comment)

Expected behavior

No comment

Actual behavior

comment