Paukan & NPRWizard
Paukan
Hey, I’ve been trying to squeeze edge‑based shading into a real‑time pipeline without drowning the frame buffer. Got any clever tricks to keep the outlines sharp but low‑overhead?
NPRWizard
Sure thing, buddy! First off, ditch that full‑blown Sobel on every frame. Grab the depth or normal buffer from the first pass, run a cheap 3‑pixel kernel on the GPU, and write the result straight into the stencil buffer. That way you only do one extra compute pass, and the GPU can skip all the heavy blending until the final composite. Next, punch those outlines with a two‑tone pass—just a flat color and a sharp step. Use a low‑precision float for the edge map to cut memory bandwidth. If you’re hunting for ultra‑low cost, consider a signed distance field for simple shapes; that gives you crisp lines with almost no extra work. Finally, lock the line width to integer pixel multiples; the GPU will keep those strokes clean without anti‑aliasing overhead. Happy to dig deeper if you need the nitty‑gritty math behind the stencil trick!
Paukan
Sounds solid, but I’d double‑check the stencil clear order—if you leave any old depth data in, the edge pass will get confused. Also, keep the kernel size odd; even‑sized kernels tend to shift the outline by half a pixel and ruin that integer‑pixel width rule you mentioned. If that’s okay, I can sketch out the exact shader code you’d need.
NPRWizard
Nice catch on the stencil and kernel parity—those little quirks can wreck a clean edge if overlooked. I’d love to see your shader sketch, especially how you’re blending the stencil pass into the final image. Don’t hesitate to throw in any hiccups you hit, I’ll help iron them out with some retro‑style fixes!
Paukan
Here’s a quick, no‑frills sketch. The first pass writes the edge map to the stencil.
```glsl
// Vertex – same for both passes
layout(location = 0) in vec3 aPos;
layout(location = 1) in vec3 aNormal;
uniform mat4 uMVP;
out vec3 vNormal;
void main() {
vNormal = aNormal;
gl_Position = uMVP * vec4(aPos,1.0);
}
```
**Stencil pass (compute)**
```glsl
layout(local_size_x = 16, local_size_y = 16) in;
layout(r8) uniform readonly image2D depthTex;
layout(r8) uniform writeonly image2D stencilTex;
void main() {
ivec2 id = ivec2(gl_GlobalInvocationID.xy);
float d = imageLoad(depthTex, id).r;
// cheap 3‑pixel kernel – just a difference with neighbors
float dL = imageLoad(depthTex, id + ivec2(-1,0)).r;
float dR = imageLoad(depthTex, id + ivec2(1,0)).r;
float dU = imageLoad(depthTex, id + ivec2(0,-1)).r;
float dD = imageLoad(depthTex, id + ivec2(0,1)).r;
float edge = max(abs(d-dL), abs(d-dR), abs(d-dU), abs(d-dD));
// threshold to binary edge
float outVal = edge > 0.02 ? 1.0 : 0.0;
imageStore(stencilTex, id, vec4(outVal));
}
```
**Final composite (fragment)**
```glsl
layout(location = 0) out vec4 fragColor;
uniform sampler2D baseTex;
uniform sampler2D stencilTex;
uniform vec3 outlineColor = vec3(0.0,0.0,0.0);
in vec2 texCoord;
void main() {
vec3 base = texture(baseTex, texCoord).rgb;
float edge = texture(stencilTex, texCoord).r;
vec3 color = mix(base, outlineColor, edge);
fragColor = vec4(color,1.0);
}
```
The only trick is the `edge > 0.02` threshold – tweak it to control width. If the stencil looks too thin, lower the threshold or run a 5‑pixel kernel. If it’s too thick, raise it. That should keep the line clean and integer‑pixel‑aligned without extra AA. Let me know if the compute stage stalls; I’ll look at memory layout or packing.
NPRWizard
Nice skeleton—keeps it lean but still solid for a real‑time edge pass. A couple quick tweaks: use `int` instead of float for the stencil write; that packs each pixel into one bit if you go 8‑bit or even use a packed uint for speed. Also, clamp your depth samples to avoid going off‑edge at borders—maybe wrap or use `clampToEdge`. If you bump into stalls, try dispatching with an even multiple of the local size and read from the depth texture in groups; that lets the GPU keep its caches hot. And if you ever want a thicker line without raising the threshold too much, pad the kernel to 5×5 but still only write the central pixel—keeps the stencil sparse. Happy shading!
Paukan
Thanks for the nitty‑gritty. Switching to an 8‑bit stencil with packed bits will cut bandwidth, but just remember the GPU’s read/write patterns – you want to avoid scattered writes. For the 5×5 pad, keep the central write and throw the rest into a small separate buffer if you need to adjust width on the fly. Also, if the depth texture has LOD issues, bind it with linear filtering disabled for the stencil pass. That should keep the stalls under control. Let me know if you hit any lock‑step oddities.
NPRWizard
That’s the spirit! Love how you’re keeping the stencil tight and the writes local—exactly what an old‑school rasterizer would have wanted. If the depth LOD still gives you hiccups, drop the nearest‑neighbour sampler, or better yet, lock the depth texture to point filtering and let the stencil pass run like a punch‑driven line art filter. Remember, you can always treat the 5×5 pad like a tiny “pen pressure” buffer; that way you keep the stroke clean and the bandwidth low. Happy edge‑hunting—just don’t let the GPU’s cache decide the story!
Paukan
Got it, will lock that filter to point and keep the pen‑pressure buffer tight. If the cache starts acting like a rogue bartender, I’ll add a small uniform counter to force a flush. Thanks for the heads‑up.