Is this the right tool for stroke rendering? And how does hardware acceleration work.

[KySpace]

Hi! I hope this will not end up irrelevant.

Motivation

I'm interested in building a stylus-whiteboard toy app that renders strokes based on the sampled points with their pressure, position and stylus angle etc.. I hope to better utilize these properties to simulate various kinds of pens and brushes and record them effectively in a vectorized way (as opposed to rasterized brushes in painting apps or simple strokes in whiteboard apps). I once assumed that I could take these sampled points as vertices and pass them to GPU to let the fragment shaders determine their boundaries, but then this doesn't seem possible since GPUs like triangles, so I've heard, see discussion

I realized that what I really want is to (for every frame) plot pixel by pixel over the entire screen, given a selection of strokes with their sampled points and I just assumed that doing it in parallel on GPU will be faster, but maybe doing it on CPU is necessary and will be more versatile (Correct me if I'm wrong).

I've read the introduction and some issues here. It seems that this crate is the right tool? Currently, I don't have much concept on what would be the performance bottleneck. I guess I can try it out but I'd appreciate if someone point out that I'm off track earlier.

Scenario model, with questions

Say, the screen I'm looking at right now has ~6 million pixels, and there are 2000 characters displayed, and I can scroll and zoom in pretty smoothly and the rendering is sharp. From what I learned, these glyphs are eventually turned into little triangles to be rendered on GPU.

• What if, instead, I do it with pixels, and draw on each of these 6 million pixels, according to the neighboring glyph/stroke, ignoring the tessellated triangles (provided that I have a method to tell which glyphs or strokes are relevant). Would the performance be on-par with the GPU? I can argue that I saved resources making triangles and use less points to record a glyph, if I do it smartly.
  • I see some demos for pixels have pixels larger than screen pixel, but I do want to render to screen pixels most of the time. Will it be slow if I render at screen resolution/size? Can I align them easily? Will they be sharp?
• What does it mean that pixels is hardware accelerated? Does it mean that GPU is used eventually? But is pixels still doing tessellation into triangles? Or does it use some other GPU features that draw pixels efficiently that I don't know? Compared to CPU rendering, what makes it faster in the end?
  • If writing to pixels is on the CPU side, is some parallelism involved
• Using pixels, am I to write the pixel rendering functions in rust or wgsl? Am I storing the sampled points of strokes in memory or GPU buffer? Or both?
• Are the pixel values stored on memory or GPU Buffer? Or either way?
• I would like to render it on a HTMLCanvas. Is it possible/easy to do, like choosing the wgpu's SurfaceTarget?
• While I render the pixels pixels to the canvas, can I also render other things on top/below it using wgpu?

Thank you!

[parasyte]

Hi!

This crate is for real-time pixel plotting. Most prominently for writing emulators for old computers and game systems, and for games that want to simulate the style of old games with low resolution graphics (typically 2D and low color depth). It has also been used for various tools of all sorts. But that sets the stage for the primary target: low resolution pixel plotting and animation.

The use case you describe is more closely related to 2D vector graphics. And pixels is not the ideal base for vector workloads. The biggest hurdle is that all you get here are simple tools for displaying a texture on screen and methods to update it. We don't even provide basic pixel plotting logic, since there are other crates which can do that if you need to. I have used pix to create a smooth sprite-based game, and the tools it provides compliments pixels perfectly for that sort of aesthetic.

You are correct that triangles are the most common GPU primitive, but there is some subtlety! GPUs also have a line primitive (e.g., for wireframe) and a point primitive (which can be used for particle effects). See: https://docs.rs/wgpu/latest/wgpu/enum.PolygonMode.html

Triangles are rasterized when the vertex list is processed in "fill" mode, otherwise it's rasterized as a contiguous line or individual points (with arbitrary radius). pixels uses a single over-large triangle to display its texture. There is more discussion about the technique in #180, including many references to interesting articles that describe in details the GPU architectures. (Mobile GPUs are quite different, they are usually called "tiled rendering GPUs". There is an adorable manga PDF from ARM that ELI5's how a tiled GPU works: https://interactive.arm.com/story/the-arm-manga-guide-to-the-mali-gpu/page/1)

The hardware acceleration that pixels uses is texture scaling and any optional post-processing shader effects. That's it.

While you can draw lines and points with a shader, it is not going to be as pretty as a real vector graphics library. For that, you have a few options:

• raqote - See also the raqote-winit example.
• tiny-skia - See also the tiny-skia example.
• piet
• vello

vello is the newest of these, and it has its own wgpu rasterizer, so you don't need to rasterize to a texture separately (and that allows it to do hardware accelerated rasterization). Given your large texture requirements, you probably want something like this instead of software texture rasterization.

Directly answering some of your questions:

What if, instead, I do it with pixels, and draw on each of these 6 million pixels, according to the neighboring glyph/stroke, ignoring the tessellated triangles (provided that I have a method to tell which glyphs or strokes are relevant). Would the performance be on-par with the GPU? I can argue that I saved resources making triangles and use less points to record a glyph, if I do it smartly.

It will be ridiculously slow. 😢 #180 explains some relevant comparisons between CPU and GPU rasterization.

I see some demos for pixels have pixels larger than screen pixel, but I do want to render to screen pixels most of the time. Will it be slow if I render at screen resolution/size? Can I align them easily? Will they be sharp?

That's the texture scaling, and it's one of the key features along with maintaining integer scaling and original aspect ratio. CPU rasterization with large textures will always be terribly slow and there is nothing you can do to improve it as much as a GPU. Alignment and sharpness are the same thing, and it's paradoxically both easy to spot and easy to miss when wrong.

What does it mean that pixels is hardware accelerated? Does it mean that GPU is used eventually? But is pixels still doing tessellation into triangles? Or does it use some other GPU features that draw pixels efficiently that I don't know? Compared to CPU rendering, what makes it faster in the end?

The hardware acceleration was already answered above. There is absolutely no tessellation in this crate. That's a technique completely unrelated to putting a texture on a display surface. pixels can only be faster than crates with a similar CPU rasterization workload that lack scaling and post processing. The CPU rasterization is otherwise the same; it's the application that handles it, not pixels.

If writing to pixels is on the CPU side, is some parallelism involved

Only as much as the rasterization is optimized for SIMD architectures. But again, pixels has nothing to do with that. You provide the rasterizer, and pixels will show your pixel buffer on screen with a few features and constraints.

Using pixels, am I to write the pixel rendering functions in rust or wgsl?

Not at the moment. See #170 for discussion.

Are the pixel values stored on memory or GPU Buffer? Or either way?

Both! Since pixels targets software rasterizers, the image data is initially written in main memory by the CPU and transferred to GPU memory as a texture.

I would like to render it on a HTMLCanvas. Is it possible/easy to do, like choosing the wgpu's SurfaceTarget?

It's more complicated than that. You use winit to create the WebGL2RenderingContext or GPUCanvasContext and wgpu to compile shaders and run command buffers that target it. pixels will nicely abstract all of that away, but it does so by only presenting an API to manipulate a single texture. Most other abstractions tend to be open ended and provide a high level API for 3D scenes. In the case of vello, the API you are presented with is a 2D vector encoder. But they can all more or less draw to an HTML Canvas without needing to piece all of the pipelines together yourself.

While I render the pixels pixels to the canvas, can I also render other things on top/below it using wgpu?

Yes. The minimal-egui and imgui-winit examples both show how you can integrate wgpu workloads with a pixel buffer. (The workloads are both GUIs, but it can just as well be a 3D scene or whatever you want.)

[KySpace]

Thanks for the great reply and introductions! I understand now that calculating on a pixel basis is a more suitable job of GPU. I now consider some alternative methods!