Well, it’s been a whole load of time since I posted, mainly due to revision and lazyness on my part. So I thought I’d post a quick update on what I’m doing – not that anyone will be all too interested.

So I’m back in Walsall after spending a week down at Sarah’s. Unfortunately, things haven’t been brilliant on the revision front. Whilst I’m mildly optimistic about Combinatorics, Introduction to Topology and to a lesser extent Topics in Mathematical Biology, I know that Measure Theory is going to be awful. I’ve not done a lot of work for it, and to be honest, I just don’t understand a lot of the material.

However, on the upside, I’ve finished all of my Topology and Topics exercise sheets, which is something that has never been done before. I’m now at the stage where, a week before the dreaded exams, I’m getting my act together and doing three past exam papers a day. So maybe there’s hope yet; we shall see, I suppose. I figure that even if Measure Theory goes badly, I can still pull it up with some better marks from the other modules.

However, I didn’t really want to talk about exams so I’ll talk about something equally as boring – programming. Been playing around quite a bit with two things over the holidays. Firstly – and this has been on the agenda for some time now – coding an OpenGL GStreamer application to map video onto a cube or some other random object. Surprisingly, this was initially very easy, once I’d found out how to actually get the contents of the GStreamer buffer into my program.

Basically I use decodebin to get the video playback working, and then pipe it through ffmpegcolorspace to get a nice RGB texture. Then, I use fakesink and a handoff function to actually generate the texture to screen and map it onto a cube. The hardest bit was actually finding out about the handoff function – it’s buried deep inside the GStreamer Plugin Guide, which is quite annoying. However, it does look quite cool. You can even take a look at it for yourself if you’d like.

Even so, I wasn’t quite satisfied. The biggest speed hump is the ffmpegcolorspace conversion, as generically most video files use some kind of YUV format to store video frames. Typically, and somewhat ashamingly, there’s no general way of dealing with YUV textures in OpenGL – all of the extensions out there generally suck, and don’t work for different versions. Plus, none of them cope with 4:2:0 planar YUV, only 4:2:2 pixel-packed for the most part. So some form of conversion to RGB is required, and a significant amount of CPU overhead is incurred in the process.

For the last couple of weeks I’ve been looking at various ways of efficiently using the GPU to do this conversion for me. Colour matrices were the first option I looked at. Basically, given some texture, one can use a colour matrix to remap colours in any given format. However, the major problem is going from 4:2:0 planar to 4:4:4 pixel packed, and that itself incurs quite a significant overhead.

Blending textures is possibly the best way of doing this. Basically, one generates 5 different textures along with using a simple colour-lookup table. Firstly, we lay down the Y channel as a luminance channel. Then the additive U, then subtractive U, and the same for V. In effect, just layering one channel on top of another to simulate the YUV->RGB conversion in the GPU.

However, there are again problems with this. The naive approach is to redraw texture upon texture, which means that every pixel needs to be redrawn 5 times every frame – at 25fps, that’s 125 times/second! In order to get the most out of the GPU, I used the ARB_multitexture extension, which allows you to layer textures extremely quickly in one pass. I also delved into the world of shaders to try and work out whether it might even be worth usnig only 3 textures and doing the RGB conversion through a simple ARB fragment program. This is something I’m still looking into.

Unfortunately, although this actually works and I’ve tested it, it’s still very very slow. However, I’m using quite a crap Intel 915GM based GPU under Linux. The i915 driver is not particularly good at the moment, and upgrading to Mesa 6.5 seems to have made matters even worse. glxgears is showing only 600fps – I should be getting at least 1300 according to other guys with the same card. So I’m guessing – and hoping – that the problem lies with the drivers and not my card. When I get back, I’m going to check it out on my Radeon 9800 and we’ll see how that copes.

Anyway, that’s probably enough for now. I’m going to reboot and see if I can get this all working nicely.

10:00 pm | Posted in GStreamer, Linux, Maths, Revision, Warwick Blogs | 3 Comments » | Show comments » | Hide comments » |