Any recipe against skill set and playwright test decay?

[davrot]

My goal was to understand if this new tech (speckit) can be used beneficially. My hope was that it keeps the context brain rot under control but I guess I was mistaken to expect that. I tried to build a simple overleaf plugin (equation editor like in word). Since 5 days, I am trying to get the AI to build this simple module. I am kind of losing the rest of my mind, since the copilot with spec kit loses its marbles over time. I thought that the idea of the spec kit was to prevent this with the .md file?

I defined a skill set (I gave it two ways how to build the overleaf module: A. modify the source code, build the dockerfile and restart the docker container B. Hotfix files inside the docker container, compile the website inside and killall -2 node). In the beginning both worked fine. Then it started to compile the source code directly and destroyed the source code. I had to restore from my backups. I told it not to do it but it continue to do it anyway. Then it started to build and introduce a local server. I asked it again and again not to do that. It doesn't cared.

"We" agreed to have for every playwright test a separate sub-folder for the screenshots. In the beginning it played along. Then it decided to throw the screenshots all over the place.

I had 12 nicely defined playwright tests. The emphasis is on "had". After some time all the tests we doing the same, fusing into each other. Currently I am trying to convince it to accept my rule for the tests and rebuild them as they where. No chance.

I asked it now many many many times to not use env variables like NODE_PATH (because I can not auto approve them)

cd /data_1/dev_latex_editor/services/web/modules/latex-editor/demo && NODE_PATH=/data_1/dev_latex_editor/playwright-tests/node_modules node capture-minimized.js

It made me even a bash script. It used it 2 times or so and then went back to env variable.

And so on...

I would say that the experiment was a complete failure. An example of finest AI brain rot. Except it missed something important!?! In the end I asked Claude to help me write the messages for Copilot. Claude kind of "gave" up and told be to be firm with Copilot.

:-(