Glitches in Warp terminal when using --follow

[ulken]

Screen.Recording.2023-02-02.at.13.03.57.mov

1. It's rendered at the bottom, instead of at the top, which is the case in other terminals like Terminal.app and Hyper.
2. It glitches every time it re-renders (the cursor jumps back and forth).

I realize the problem is most likely in Warp, but I just wanted to check if you know of any obvious reason why it behaves the way it does and/or if something can be done in klog to handle "both" cases.

Other long-running commands like htop behaves as expected. Also it doesn't output a bunch of extra newlines after the process exits (might be related).

Just started using klog, but I must say it's awesome in its simplicity!

[ulken]

I realize the --diff --now part is unnecessary. --follow is enough to reproduce.

[jotaen]

Thanks for the detailed report. I have never used Warp, so nothing rings a bell with me here spontaneously.

The “offending” lines would be in the with_repeat.go helper function, which continuously keeps rendering the terminal screen:

klog/klog/app/cli/lib/with_repeat.go

Lines 22 to 35 in 73250d9

	// Call handler function repetitively
	print("\033[2J") // Initial screen clearing
	ticker := gotime.NewTicker(interval)
	defer ticker.Stop()
	secondsCounter := int64(0) // Choose large type because of overflow
	for ; true; <-ticker.C {
	secondsCounter += 1
	print("\033[H\033[J") // Cursor reset
	err := fn(secondsCounter)
	if err != nil {
	return err
	}
	}
	return nil

More specifically, it uses two escape sequences \033[2J for clearing the screen and \033[H\033[J for resetting the cursor to the top left position. That happens once per second, which is also why it flickers in your video. (Which it shouldn’t do either, though.) Maybe Warp interprets or responds to these escape sequences in a different way compared to other Terminals?

I cannot observe the flickering or alignment issues anywhere else, but I also have to admit that I’m not an expert for escape sequences. So maybe htop and the like have a different, more robust approach here.

Please let me know in case you find something out here, maybe there is a simple and reasonable way to fix this for Warp after all. (The only thing I’d be a bit hesitant about would be to introduce a new library for this, or to add lots of workaround-code.)

I realize the --diff --now part is unnecessary. --follow is enough to reproduce.

Yes, that’s correct. The --follow flag activates the withRepeat helper:

klog/klog/app/cli/today.go

Lines 34 to 45 in 73250d9

	if opt.Follow {
	return lib.WithRepeat(ctx.Print, 1*gotime.Second, func(counter int64) app.Error {
	err := handle(opt, ctx)
	if counter < 7 {
	// Display exit hint for a couple of seconds.
	ctx.Print("\n")
	ctx.Print("Press ^C to exit")
	ctx.Print("\n")
	}
	return err
	})
	}

Just started using klog, but I must say it's awesome in its simplicity!

Thanks, happy to hear that 😃

[ulken]

Same things goes for the pause command, but that's probably expected.

[ulken]

Opened an issue over at Warp as well: warpdotdev/Warp#2597

[ulken]

Yeah, htop is more low-level and written in C. Uses ncurses under the hood.

[jotaen]

Opened an issue over at Warp as well: warpdotdev/Warp#2597

Terrific, thanks! Maybe that yields a lead.

Same things goes for the pause command, but that's probably expected.

Yes, klog today and klog pause both use that WithRepeat helper. (There are no other instances of it, though.)

[ulken]

https://stackoverflow.com/a/71453783/1728686 might provide some insights

[ulken]

The answers here are also relevant: https://unix.stackexchange.com/questions/81167/prevent-text-screen-blinking-when-doing-clear

[ulken]

Here are some good tips I think are worth following/adapting. This guy knows what he's talking about. His entire business revolves around terminal CLIs/TUIs and he's quite established.

https://www.textualize.io/blog/posts/7-things-about-terminals

There are a few factors which reduce flicker in the terminal. The first is the terminal emulator that you are using. Modern terminals use hardware-accelerated rendering and will synchronize updates with your display to reduce flicker. Although in my experience there are other factors which have a greater impact on reducing flicker than your choice of emulator. Even on older terminals, you can generally get flicker-free animation with this one little trick (actually three).

The first trick is "overwrite, don't clear". If you clear the "screen" and then add new content, you risk seeing a blank or partially blank frame for a brief moment. It's far better to overwrite the content in the terminal entirely so that there is no intermediate blank frame.

The second trick would be to write new content in a single write to standard output. It may be convenient to make multiple file.write calls to display an update, but as with the blank frame you risk a partial update becoming visible.

The third trick would be to use the Synchronized Output protocol; a relatively new addition to the terminal protocol, but already supported by many terminals. Details in the link, but the gist is that you tell the terminal when you begin and end a frame. It can then use this information to deliver flicker free updates.

Nr. 3 might be overkill, but Nr. 1 and Nr. 2 should be pursued, IMO.

[ulken]

I've read up some more on this and would be willing to have a go (ha ha) at some of the issues. Would you be interested in a PR or two, if/when I find the time?

Note: I'm new to Golang, but hoping, with your, Google and Copilot's guidance, I can navigate my way to some sort of solution.

Not leaving any ETAs, just checking if it's worth pursuing or not.

[jotaen]

Thanks for offering your help! For this issue here, please feel free to pick it up and fiddle around. As mentioned above, the starting point is the WithRepeat helper. A few spontaneous hints:

• The WithRepeat helper doesn’t have test coverage, neither directly nor implicit, so the only way to QA it is by trying it out in various terminals and OS’es.
• Terminal escape sequences are a bit funky sometimes, e.g. there already were some weird issues in the past.
• I’m trying to support Windows on a best-effort basis, so if things work there too (most notably PowerShell and Windows Terminal) then that’s great, but if it’s causing too many complications then it’s okay to make pragmatic sacrifices.

Not sure if you were referring to other issues/discussions as well, so just in case: I’m generally open for contributions, but I’m also pretty deliberate about code structure and feature scope, mostly in order to keep the maintenance burden within reasonable bounds. So in case you are motivated to work on other things as well, please reach out beforehand so that we can discuss a plan. I’d also be open for collaborative efforts, although I have to admit that I probably won’t have much capacity at the moment to work on bigger topics.

[ulken]

Thanks for offering your help! For this issue here, please feel free to pick it up and fiddle around. As mentioned above, the starting point is the WithRepeat helper.

Thanks 🙏

The WithRepeat helper doesn’t have test coverage, neither directly nor implicit, so the only way to QA it is by trying it out in various terminals and OS’es.

OK, I'll try out a few terminals, but I'm Mac only. Do you have any helpful Windows-users?

Terminal escape sequences are a bit funky sometimes, e.g. there already were some weird issues in the past.

Noted.

I’m trying to support Windows on a best-effort basis, so if things work there too (most notably PowerShell and Windows Terminal) then that’s great, but if it’s causing too many complications then it’s okay to make pragmatic sacrifices.

Hum, OK, that limits possibilities. I'll see what gives. Maybe I'll just run my own fork if it ends up working as desired (but non-portable).

Not sure if you were referring to other issues/discussions as well, so just in case: I’m generally open for contributions, but I’m also pretty deliberate about code structure and feature scope, mostly in order to keep the maintenance burden within reasonable bounds. So in case you are motivated to work on other things as well, please reach out beforehand so that we can discuss a plan. I’d also be open for collaborative efforts, although I have to admit that I probably won’t have much capacity at the moment to work on bigger topics.

I hear you. I was already under that impression. No worries, I'll do my best to adhere to your principles. If you end up not liking it, that's totally fine.

My available time is very limited as well (no. 2 is a month a half), so yeah.

[jotaen]

Hum, OK, that limits possibilities. I'll see what gives. Maybe I'll just run my own fork if it ends up working as desired (but non-portable).

Just in case there might have been a misunderstanding: I’m positive to make sacrifices in regards to Windows portability. So if things happen to work on Windows without much extra effort, then that’s cool, but if proper Windows support creates lots of complications, then it’s okay for me to accept that it won’t work perfectly on Windows.

I can also help out with testing on Windows, as I have a Windows VM set up for testing purposes. So don’t worry too much about it for now.

I think this issue here would be very valuable to fix, because what you found out in regards to terminal best practices seems convincing to me, especially the topic about overwriting the screen incrementally instead of clearing it. So I appreciate if you find some time to look into it, and hopefully we’re able to improve it 🙏 If you have questions regarding the code structure or project setup, please let me know.

[jotaen]

@ulken I briefly wanted to check in, since it has been a while: is this still an issue for you? I probably won’t get around to fixing this anytime soon, so I’m trying to determine whether the ticket is still worth to keep open. If you (or anyone else) is still affected, though, I’m happy to keep the ticket open.

[ulken]

@ulken I briefly wanted to check in, since it has been a while: is this still an issue for you? I probably won’t get around to fixing this anytime soon, so I’m trying to determine whether the ticket is still worth to keep open. If you (or anyone else) is still affected, though, I’m happy to keep the ticket open.

The behavior is definitely still there, but I never use the feature. At least partially due to the issue at hand.

But it's fine by if you want to do some fall cleaning. I won't be able to get around to it anytime soon.

[jotaen]

I’ve looked into this a little, just to get a feeling for the scope. TLDR, I’d close this ticket for now, as it seems tricky to fix, and it only affects very specific terminals.

Some findings regarding the “tricks” you referenced earlier:

• Re trick 1 “Overwrite, don’t clear”: That, unfortunately, would require more refactoring in the way klog commands print to the terminal. Part of the problem is that klog’s printing mechanism is relatively naive, as it just flushes things to the terminal whenever there is something to say. Overwriting would require to keep track e.g. of newlines, in order to reliably clear leftovers. I’m currently not sure whether the complexity would be worth that.
• Re trick 2 “Buffering”: Writing in a single pass, as opposed to issuing multiple writes, only slightly alleviated the flickering on Warp, but the problem is still very obvious.
• Re trick 3 “Synchronized output”: The synchronized output you mentioned earlier seems interesting, because it’s literally just emitting one \033[?2026h before the render cycle, and \033[?2026l afterwards. Unfortunately, it doesn’t seem to be widely supported yet. Warp hasn’t implemented it, and in contrast to what this overview page says, I also couldn’t verify support in Mac Terminal or iTerm2 (although flickering isn’t an issue in these two anyway). I also couldn’t verify whether this technique would have any effect at all, as Warp is the only terminal right now where the flickering is known to be happening.

If there are news on the Warp side of things, or if anyone has new ideas, feel free to post them here, then we can reconsider this issue. For now, I’d put it on halt as “won’t fix”.