V/G shader to properly generate rectangle from 1-2 vertices

Post Reply
Posts: 103
Joined: Sun Mar 08, 2009 7:24 pm

V/G shader to properly generate rectangle from 1-2 vertices

Post by Discostew » Sat Mar 07, 2015 7:37 pm

Before I start, I want to say that I'm doing this to help StapleButter with BlargSNES. I'm attempting to develop a routine for displaying Mode7 using GPU acceleration, as it currently is handled via software-rendering (which is slow).

What I'm looking to do is take either 1 or 2 vertices I input, and output 2 polygons that form a rectangle. The rectangle itself is to have a width/height of 8 pixels (prior to matrix calculations), so in a case where I have 2 points, they would be on opposite corners of the shape, so the geometry shader would use them to generate the two other vertices to form an axis-aligned rectangle (using one's X and the other's Y). This is how BlargSNES currently handles tiles for modes other than Mode7, as they are axis-aligned. I am attempting to incorporate a model-view matrix that'll define the scale/rotation for a group of tiles (from either full-screen or per-scanline), but the results are not correct because I implemented it via the vertex shader.


The image itself is not rotating the tiles, but is instead squishing either horizontally or vertically because the 2 vertices are processed with model-view matrix calculations in the vertex shader (which now I know is done before the geometry shader, not after). I tried to shift such calculations to the geometry shader, but it didn't like that, resulting in a blank screen and eventually led to a crash.

Here's the code that would produce the results of the image above.

Code: Select all

; setup constants
	.const c9, 0.0, 0.0, 0.0078125, 1.0
; setup outmap
	.out o0, result.position
	.out o1, result.color
	.out o2, result.texcoord0
; setup uniform map (not required)
	.uniform c0, c3, projMtx
	.vsh vmain, end_vmain
	.gsh gmain, end_gmain
; v0: vertex (x, y, and optional z)
; v1: texcoord
; c0-c3: projection matrix
; c4: texcoord scale
		mov r2, v0 (0x4)
		mov r2, c9 (0x3)

		; my inclusion of modelview-matrix calculation, which this (and projMtx) may need to be in the geometry shader if possible?
		dp4 r1, c5, r2 (0x0)
		dp4 r1, c6, r2 (0x1)
		dp4 r1, c7, r2 (0x2)
		dp4 r1, c8, r2 (0x3)

		; result.pos = projMtx * temp.pos
		dp4 o0, c0, r1 (0x0)
		dp4 o0, c1, r1 (0x1)
		dp4 o0, c2, r1 (0x2)
		dp4 o0, c3, r1 (0x3)

		; result.texcoord = in.texcoord * (uniform scale in c4)
		mul r1, c4, v1 (0x5)
		mov o1, r1 (0x5)
		; turn two vertices into a rectangle
		; setemit: vtxid, primemit, winding
		; v0 = vertex 0, position
		; v1 = vertex 0, texcoord
		; v2 = vertex 1, position
		; v3 = vertex 1, texcoord
		; x1 y1
		setemit vtx0, false, false
		mov o0, v0 (0x5)
		mov o1, c9 (0x8)
		mov o2, v1 (0x5)
		; x2 y1
		setemit vtx1, false, false
		mov o0, v2 (0x6)
		mov o0, v0 (0x7)
		mov o1, c9 (0x8)
		mov o2, v3 (0x6)
		mov o2, v1 (0x7)
		; x1 y2
		setemit vtx2, true, false
		mov o0, v0 (0x6)
		mov o0, v2 (0x7)
		mov o1, c9 (0x8)
		mov o2, v1 (0x6)
		mov o2, v3 (0x7)
		; x2 y2
		setemit vtx0, true, true
		mov o0, v2 (0x5)
		mov o1, c9 (0x8)
		mov o2, v3 (0x5)
;operand descriptors
	.opdesc x___, xyzw, xyzw ; 0x0
	.opdesc _y__, xyzw, xyzw ; 0x1
	.opdesc __z_, xyzw, xyzw ; 0x2
	.opdesc ___w, xyzw, xyzw ; 0x3
	.opdesc xyz_, xyzw, xyzw ; 0x4
	.opdesc xyzw, xyzw, xyzw ; 0x5
	.opdesc x_zw, xyzw, xyzw ; 0x6
	.opdesc _y__, yyyw, xyzw ; 0x7
	.opdesc xyzw, wwww, wwww ; 0x8
I'm still in the early stages of learning shaders, so while I have some concepts down, there is still a lot I don't understand, and this is one of them. If all goes well, I'd attempt to do this with only one vertex input and have the geometry shader generate 3 additional vertices, as the width/height never change.

Posts: 210
Joined: Thu Feb 03, 2011 10:47 pm

Re: V/G shader to properly generate rectangle from 1-2 verti

Post by mtheall » Sun Mar 08, 2015 4:31 am

This approach is simply not going to work for affine transformations. Your geoshader assumes that the two added vertices are axis-aligned with the two input vertices, which is simply not true for affine transformations (except for obvious cases like 90 degree rotation or horizontal flip, among others). You need to pass the affine matrix to the geoshader and calculate the missing vertices with this transformation.

The problem I see you immediately running into is that many games use Mode 7 to create a perspective transformation by adjusting the affine matrix per-scanline. You will probably have to rely on hacks that know a game is creating a certain perspective via scanline tricks and use a pre-calculated perspective transform.

An alternative is to have the emulator calculate all of the vertices and pass them all to the vertex shader, ditching the geoshader altogether. Of course, you can take this approach for only Mode 7 backgrounds and still use your geoshader approach for the other backgrounds since they will be axis-aligned.

Posts: 103
Joined: Sun Mar 08, 2009 7:24 pm

Re: V/G shader to properly generate rectangle from 1-2 verti

Post by Discostew » Sun Mar 08, 2015 5:57 am

Yeah, I started to see that as I kept trying and trying. I resorted to throwing in 4 vertices (and figured out why it wasn't working before). I understand that there may be problems with per-scanline sets, but I'll tackle that when I get to it. First I want to get full-screen, unmodified-scanline Mode7 working.

So far everything looks good, except I can't get the positioning for the layer correct. I had to use the inverse of the affine matrix to get the scaling/orientation correct, but getting the position correct is lost to me atm.

Posts: 103
Joined: Sun Mar 08, 2009 7:24 pm

Re: V/G shader to properly generate rectangle from 1-2 verti

Post by Discostew » Sun Mar 08, 2015 8:50 pm

Ok, I believe I got the calculations for positioning correct. In much the same way I had to calculate the right affine matrix (by getting it's inverse), I sorta did the reverse on calculating the position. Now I can work on figuring out which tiles are to be rendered, which if I've thought this out carefully, would be kinda like rendering a flat-shaded polygon in software, but mapping the routine to the layer to grab the right tiles.

Post Reply

Who is online

Users browsing this forum: No registered users and 2 guests