Output window for DMD server debug output?

[TurkeyMan]

VisualD is really just not working well for me still.

Goto definition almost never works, even in situations where there are no templates in sight.
It's also still extremely slow. Even when goto definition does work, it can often take several seconds for it to resolve and actually go there.
Everything in the C++ experience is instantaneous, but we seem to have a lot of really long latencies by comparison, which suggests there is something fundamentally different about the architecture?

Anyway, back on topic; I wonder if it would be worth directing the DMD Semantic Server output to a VS Output window?

I was just now trying to remember the output file name from last time we discussed this to look for errors. I've been looking for the issue (open or closed) where we had that discussion and I can't find it... could you remind me where it's hiding?

Using an output window would make it much simpler to quickly diagnose and report specific issues without special knowledge, and for instance, my colleagues who aren't motivated to investigate the log files might also casually take notice in passing...?

[rainers]

I recently also had the impression that the semantic engine is quite a bit slower than the compiler (which in addition generates code). Part of that is probably due to the use of the GC instead of dmd's bump allocator (it still consumes tons of memory sometimes, I recently saw it at 50 GB making it not respond at all). Another issue is that the engine is built with dmd instead of LDC, because the latter does not support generating pointer information for the DATA/BSS segments to be used by the precise GC.

I tried to make it more incremental recently, but that made it even less stable. Maybe some tiny edits should not cause most of the code to be recompiled, but it is difficult to locate the proper parts that might be affected by a change.

Everything in the C++ experience is instantaneous,

Not for me, I'm also switching to text search quite often because VC's intellisense takes forever. Granted, some of the C++ projects I work on are maybe 100 times larger than dmd, for example.

could you remind me where it's hiding?

If there is a folder %temp%\dmdserver, it logs the communication in there. I doubt it makes a lot of sense to make this more visible in the UI, at least not without making the output more manageable.

[rainers]

On second thought I added forwarding a shortened version of the logging to the Visual D output pane, and it showed some quirks immediately :) There is a queue of requests to forward to the server and wait for their completion and this easily gets stalled. I suspect it is mostly waiting for the GC that has to deal with way too much memory. Still investigating...

[TurkeyMan]

Oh excellent. I'd love to see this output when applied to my code. Having that output in a kinda real-time sense will make it much easier to report actionable issues.

Why does DMD seem to consume enormous loads of memory? Other compilers aren't as bad as DMD; is there something fundamentally wrong with DMD's architecture that makes it such a memory hog?

I suspect we might just be seeing the "real" cost of using a GC; people are always focused on the GC speed itself, but the real cost is that it causes people to write in a not-memory-conscious way, and also in such a way that disregards consideration to memory pooling and/or lifetimes.

[rainers]

Sorry for the delay, it is now released with https://github.com/dlang/visuald/releases/tag/v1.5.0-beta1

I have made a couple of optimizations, should be a bit faster but no general changes. The dmdserver log now also contains some GC collection timings, showing that my goal I was hoping to reach at DConf 2019(?) of less than 50 ms for 1 GB of memory is still not reached, even with parallel scanning.

Why does DMD seem to consume enormous loads of memory?

DMD uses a region allocator that never frees any memory (CTFE has it's own region allocator that can be reused), but it has to keep the full AST in memory anyway, in case a function gets executed. Other languages might be able to only keep declaration information, but prune function bodies or initializers. In dmd code generation is also a separate pass only run after all semantic analysis is done.

[TurkeyMan]

Hmm. Do we know general stats like which nodes are most numerous, or which allocations dominate the pool? I wonder if structure layout optimization is worth thinking about.

I wonder if we could introduce some managed containers and reduce pressure on the GC? I gotta find some time to push on the move semantics some more...

[rainers]

There has been some effort in reducing the size of the AST nodes. Some structs are also maintaining free lists, e.g. Scope, and data known to be temporary is often allocated using C malloc and free.

When fixing parallel scanning I've made some benchmarks building phobos unittests with/without GC here: dlang/dmd#21428 (comment). The GC saves less than 50% of memory in that case.