Benchmark methodology & setup

[Miloslav]

Hi @PHPantom-dev team,

Congrats on the project! It's always great to see new tooling emerging in the PHP ecosystem.

We noticed the benchmark table in your README.md (21K PHP files, 1.5M LOC). Could you share which codebase or script was used for that 21K files benchmark, and how the test was executed?

Use case

We’d love to run these performance tests internally against PHP Tools to analyze potential bottlenecks and compare results under the exact same conditions and environment.

Thanks!
Best regards,
DEVSENSE Team

[AJenbo]

Hi @Miloslav and the DEVSENSE team,

Thanks for reaching out, and thank you for the kind words! It’s great to see active interest and engagement across the PHP tooling ecosystem.

The codebase referenced in the README is a private commercial Laravel application, so I am unfortunately unable to share that specific codebase publicly.

However, since the PhpStorm team has also reached out with similar questions, I think maybe we should try and find a open-source projects to use for benchmarking so everyone can replicate them and compare.

The system I run the benchmark on is a Ryzen 5950x running Ubuntu 26.06. The timing is total CPU time consumed, rather than wall time. I do this to eliminate the "works fast on my machine" problem, it also gives a good view of total energy usage. However, it does introduce some trade-offs:

• Threading: Tools that utilize multi-threading poorly can appear artificially favorable in CPU time, even though a larger fraction of their execution time is felt directly as latency by the user.
• Overhead: On a 32-thread CPU, thread management overhead is more pronounced for tools that attempt to take advantage of a high core count.

Execution & Readiness Criteria

• PhpStorm: Caches are cleared, and the IDE is relaunched from the menu.
• LSPs: Caches are manually cleared, and the server process is launched cold via Zed.
• Measuring "Readiness": Time is tracked until total system resource consumption settles back to baseline idle and the system responds to a hover on code that utilizes the framework.

Known Nuances & Caveats:

• Background Indexing vs. Usability: Waiting for the system to settle can obscure when a server becomes functionally responsive. For instance, PHPantom becomes usable/interactive after ~1s, but continues background indexing across the workspace for ~4s.
• Full Diagnostics: On the current main branch, PHPantom runs full-project diagnostics in the background by default upon launch, which extends total CPU time to ~40s (this probably won't be enabled by default by the next release).

Ways to improve

It would be great to collaborate on selecting target OSS repositories and writing a reproducible benchmark script we can all run. If you have specific projects or methodology suggestions in mind, feel free to share them here.

I would prefer including a large open-source Laravel project in the suite, as framework-specific support is where we focus heavily and where LSPs spend significant time processing the codebase.

Best regards,

• Anders

[DarkGhostHunter]

In terms of OSS repositories, there are a lot of candidates, but I think there should be a selection of four:

1. Simple "Hello world" using a micro-framework (great to find problems)
2. A library (Wordpress is my candidate; great to most test code in source rather than in vendor)
3. A Symfony based project
4. A Laravel base project

I believe these four should suffice all scenarios, especially the last two in terms of framework usage.

[AJenbo]

Since there is no real way to "finish" this issue I'll convert it to a discussion.

[AJenbo]

I just found out that Zed was disabling composerNodes which causes performance issues with Devsens: zed-extensions/php#132

It should be fixed now, I'll make sure to update my benchmark and let you know once I have done so.