I’m a complete amateur to DSLR photography. But I’m keen to learn, and so I’ve been trying to set myself tasks for learning bits and pieces of new information over time. Over the past week, I’ve been trying to find out what RAW is, and how I can create a decent workflow using it. This post is more of a reminder for me, but I hope that some other people can find it useful as well.

So, what is RAW?

Simply put, a RAW photograph allows you to store the unaltered, “raw” output of the sensor of the camera. When I first got the EOS-350D, like a lot of people I mainly ignored the fact that I could shoot in RAW. I didn’t see the point; mainly I justified this by saying “ooh, I can get 650 photos on the memory card if I shoot with JPEG!”

What’s the difference? Well, when shooting with JPEG, your camera will take the sensor output, apply the white balance to the image and then encode the photo using the JPEG compression algorithm. The critical thing to note is that this process is irreversible, and makes further post-processing more difficult down the road.

When shooting RAW, the white balance information is stored in the file, but not applied to the image itself. So, if you shoot with auto white-balance and decide down the line that you should have applied a daylight setting instead, this isn’t a problem.

People that know me might know that I can get really very obsessive about having things like photographs organised, whilst still retaining all of the data that was embedded with the image. I find it very difficult to even throw away bad photos just in case I do lose something. So clearly, I’m going to want to shoot RAW - just in case.

At this point, I ran into a few obstacles. Firstly, there is no set standard for RAW images (more on this later). Instead, the manufacturers all have their own different versions of what they deem to be “raw”. Since I own an EOS-350D, the raw of choice is the CR2 file, so I’m only going to talk about this from now on. Also, the CR2 format is closed but has been reverse engineered to a point.

Secondly, I need to organise my photos. Up until this point, I’d been using the Canon ZoomBrowser software to retrieve my images from the camera and store them locally. This stores the pictures in folder such as 2006_08_02. I have a big problem remembering dates, so clearly I need to be able to re-organise into descriptive names as well as retaining some sort of chronological order.

After some debating, I eventually created a list of objectives.

• Find (or write) a piece of software which could be used to grab all of the photos off of a memory card reader. Basically scan it for JPEGs and Canon’s CR2 RAW format.
• Find a program that’s actually capable of organising photos in a sensible fashion and something that I find easy to use.
• Use some mythical beast to manipulate the CR2’s and export them as JPEGs.
• Eventually find some way of integrating the whole lot into WordPress in a very easy and doable fashion.

At this point, I had a big decision to make: Linux, or Windows?

As you all know, I think Linux is amazing. There are some very good tools out there for organising photos and dealing with RAW. For completeness and future reference, I’ve got a small list.

• dcraw: a command line utility for converting pretty much every RAW format under the sun.
• ufraw: a GIMP plugin centred designed to utilize dcraw for importing and post-processing RAW files.
• F-Spot: a nice Mono GTK application for organising photos by tags, similar to Photoshop Elements.
• exiftool: a Perl module for extracting EXIF/XMP/etc information from photos.

The problem is, they just don’t fit together in a way that I’m comfortable with. Everything takes absolutely ages, and when I have 3,000 photos waiting to be collected together and organised, time is of the essence. So, unfortunately, Linux is out for the time being.

This (obviously) leaves me with Windows. Interestingly enough, objective 3 is by far the easiest decision. Photoshop CS2 with the Camera Raw plugin absolutely rocks. It’s comprehensive enough so that I can alter the stuff that I want to play with, but not so complex that it makes me bang my head off of the desk.

After a couple of days of searching, I found Picasa for objective 2. It’s an amazing application with a lot of potential, and naturally it’s straight from the guys at Google. I can set it up to sit there and watch the My Pictures folder, and since it supports a variety of RAW formats, an awful lot of time gets saved. Also, since I can organize by folder, I solve the problem of crappy folder names.

The one problem is that whilst it works, Picasa doesn’t seem to apply any sort of colour space to the image, so everything turns out dark. But hopefully this should get better in later versions.
This basically left me with objective 1, since number 4 is going to be solved in a couple of days time. By a complete stroke of genius, I ran across this blog by an author I don’t recognise. He uses both the same camera (and so has the CR2 problem) and uses Picasa for general organisation. There were two posts in particular that caught my eye.

Firstly, this post talks a lot about DNG, the open-specification format for RAW images from Adobe. As the other post is quite detailed, I’ll not talk about it very much. The second post talks about automated download from a memory card. Both of these things pretty much solve my woes with the CR2 images and also a method of grabbing them from the card. Basically:

• Have a program sitting in the background.
• When it’s activated, select the folder to scan from and upload to.
• Filter out CR2 images and process with the Adobe DNG converter.
• Verify conversion was okay.
• Copy image to folder by date created timestamp (can’t be bothered with parsing EXIF).
• Empty the card.

So this is the new project for tomorrow. I’m writing this application in C#, so we’ll see how it goes. My workflow then becomes:

• Take photos.
• Grab card out of camera, slap into the machine and hit the big go button.
• Launch up Picasa, which automagically scans and adds the photos.
• Remove the crap photos.
• Post-process if necessary.

Looks like it’s going to be a fun day!

12:28 am | Posted in Computing, Linux, Photography | 6 Comments » | Show comments » | Hide comments » |

Over the past few days I’ve been trying to procrastinate as much as is humanly possible – for example, playing GTA: San Andreas for very large periods of time. However, last night I turned my attention to the website backend that I’ve been developing, and so far the results have been promising.

Basically, the entire concept revolves around a single “site” object, which is set up by a global include script. The constructor sets up the smarty templating, a PDO object and starts PHP sessions. It also checks for login requests and the like. Then, when the page is loaded it simply registers itself with the site object (so it can do some pretty GUI things and also track the user through the pages) and then gets on with the business of doing useful things.

Since PHP5 (finally) supports destructors, it then grabs the page content from the output buffer, slaps it all into a smarty template automagically and then outputs without the script page ever having to do a single thing. Also, error handling can be done really easily – an exception handler catches any PDO errors and, if the site–>error() function is called, it sets an error flag and exits. On exit, the destructor is called and it can do the appropriate stuff.

One final thing I want to do is create a PDO proxy object which will catch and log the queries sent to the database for statistics information. The pseudo–function __call is very useful in this respect.

On the Ubuntu side of things, I’ve finally got it all set up the way I want it. Took me ages to sort out the supposedly simple keyboard layout problem, but in the end I just edited the xorg.conf file manually instead of messing around. And, to be honest, I’m very pleased with the outcome. I don’t have to bother messing around with lots of stuff, and now I even have mplayerplugin working correctly. Amazing. Still have to sort out hibernation, but I think that’s a problem with my grub.conf and 915resolution.

I’m also taking the liberty of upgrading software on my server, and getting it all working properly. It’s currently on the 2.6.12 kernel and has an uptime of 158 days (which I’m very proud of), but I’m afraid that it really needs an update. God only knows how many security holes have been found now, so I’ll probably update that tonight and reboot it when I get home. Also decided that enough is enough, and completely removed X support – it’s really not needed at all on a server.

Looking to the future, I’m planning on replacing the role of my home server by my current desktop machine. Ideally I want my home server to be a small box which only deals with router requests and has no other processes running, purely for security reasons. I’m planning on replacing my desktop at the end of next year, although depending on my financial situation, I may decide to replace it in October.

On the hardware front I’ll definately be going for some kind of dual–core Athlon, but I’m not sure which yet. To be honest, I would absolutely love a dual 7800GTX SLi machine – just for kicks – but that might be a little pricey. It’s basically going to come down to what I can afford. Hopefully the price of hard–drives will have come down dramatically, and I’d like to re–use the drives in my current machine for the new one and have some kind of home–network hard–drive option on the server. We’ll just have to wait and see what happens, I suppose.

2:41 pm | Posted in Hardware, Linux, Randomness, Warwick Blogs | No Comments »

Well, it’s quite amazing that I’m actually writing two entries in one day, but what the heck – it’s for a good reason.

I decided that instead of trying (and probably failing) to get Xgl working with my i915 driver, I’d give AIGLX a shot instead. There was a nice howto and repository on the Ubuntu forums which I followed, and that got it all set up in a matter of minutes. I’m now quite pleased to say that everything’s up and running and working much better than I ever expected.

cube/rotate work extremely well, and the transparency effect when opening/closing windows is very nice. Minimize also works really effectively, but unfortunately the switcher is broken for the time being (alt–tab works, but I can’t seem to get out of it – maybe I’m just being dumb). miniwin is also broken, but the other plugins work great. wobbly is fun for a while, but not really of much use other than for nice eye–candy.

The only downside is that resizing windows is quite slow at the moment. Hopefully with successive releases this will improve, but for the time being I’m definately keeping this enabled. It’s really great to see something that’s truly revolutionary. I just wish that we’d moved over to xorg a long time ago, instead of keeping the XFree behemoth alive.

2:38 pm | Posted in Linux, Warwick Blogs | No Comments »

Well, first impressions are really very good. Considering that I installed from an alpha version of the LiveCD, installation went very smoothly. Managed to reformat using the inbuilt GParted widget and get to the very end of the installation process before grub didn’t want to install. Luckily I have quite a lot of experience with grub due to Gentoo installs, so that was easily rectified.

Ubuntu has, amazingly enough, managed to auto–detect pretty much everything. Even ACPI sleep works – something I never got working under Gentoo. Need to sort out hibernation but that’s probably not going to get used that much. Right now I’m running a dist–upgrade, and then its time to try out AIGLX and Compiz for some nice prettyness.

Anyway, time to transfer some documents back over and get the laptop set up.

1:05 pm | Posted in Linux, Warwick Blogs | No Comments »

Well, after having wasted about three hours trying to fix an obscure Gentoo–related wireless problem, I’ve come to the conclusion that it’s probably time that I tried another distribution for the first time in two years. Gentoo is an amazing distribution, and I really do love working with it. But when you’re trying to get work done, and you can’t because of some silly little problem, it does get quite annoying. Equally annoying is when emerging a package that subsequently breaks everything.

So, I’m going to back up my documents tonight, make a ghost image of my Gentoo partition and install Ubuntu (Flight 6). I’m actually really looking forward to trying out Gnome 2.14 properly, and I might even give Xgl a go if I can get it working nicely with the Intel drivers.

Currently downloading the LiveCD which I intend to try out for a while, and then use the brand–spanking new installer to (hopefully) install without wiping my entire drive.

It’ll be a shame to lose Gentoo after all of these years but it’s getting to the point where I just need something that works out of the box – to a certain degree – without me spending half my time editing configuration files. I shall definately be keeping it on my server boxes, but I think that Ubuntu is definately the way to go on my laptop.

Well, that’s my evening sorted out.

9:02 pm | Posted in Linux, Warwick Blogs | No Comments »

So the first exam, Topics in Mathematical Biology, is all out of the way and done. My impressions were that it was fairly similar to the last two years’ worth of past papers, and having done those and the question sheets, it made it slightly easier. However, I couldn’t help but feel that the questions themselves were quite bulky. I found there was an awful lot to do for each question – especially the first, which took me about 45 minutes!

I attempted questions 1–3 and 5. I only answered question 1 in its entirety due to immense time constraints, but I got the vast majority of the work done. For the most part it seemed fairly correct. Unfortuantely I couldn’t quite remember how the very end of the singular pertubation question was supposed to work, so even though I’d got down to the inner and outer solutions, I couldn’t match them up properly to get the composite which was annoying. However, I gave it a decent shot so we’ll see what happens.

Question 3 was actually quite interesting and I very much enjoyed it. I did manage to complete all of the parts barr the very last bit, for which I basically scrawled something down when we were told to put our pens down. Managed to get the same answers as the question, so that’s got to count for something.

Didn’t attempt question 4 since I know next to nothing about lambda-omega wavetrains, so sat down and did question 5. Didn’t do the initial 3-mark starter question as I meant to come back to it later but again, didn’t get around to it which was a bit annoying. Got through most of it with ease, but unfortunately discovered a mistake in the very last part which means that I’m going to have lost a couple of marks.

However, it’s all over and done with, and I’m fairly pleased with what I’ve written. I can’t account for moderation and stupid errors on my part, but I guess that 70% isn’t an outrageous estimate. I’d be very pleased with that, considering I only actually went to one of the lectures for the entire course.

Next exam is on Friday morning, and it’s Introduction to Topology. I’m actually really looking forward to that since it’s one of my favourite lecture courses of all time. Unfortunately I also have Combinatorics immediately afterwards, so tonight will be spent completing the very last topology paper, and tomorrow I’ll mostly do Combinatorics.

So, we’ll see how the both of those go. Will hopefully update the blog afterwards, but that depends on how much time it’s going to take since the dreaded Measure Theory is on Monday morning. Eek.

8:31 pm | Posted in Exams, Linux, Maths, Warwick Blogs | No Comments »

I’ve recently become somewhat disillusioned with the annoyances of getting Qt and GStreamer to play nicely, so I thought I’d give Gnome a go. So, I spent a couple of hours emerging the source and getting to know it, and I have to say that I was quite impressed. I spent quite a lot of time trying out little bits here and there, and Nautilus is extremely nippy when compared to Konqueror.

However, after having used it for a few days I’m switching back to KDE. Why? There are certain things that I love doing in KDE that I simply can’t do in pretty much every other OS out there. For example, I can open Konqueror, browse the web for a bit and then open a new tab for browsing the local network, or perhaps transfer a couple of files by FTP. KDE seems a lot more tightly-knit, and for some reason I just couldn’t get along with Firefox.

Perhaps I’m just too picky, but I love the KDE interface. I think it’s one of the most elegant out there, and with a decent font (Bitstream Vera or, in my case, Luxi Mono) it just looks great.

So for now, back to KDE. I’ll probably have to give XFCE another go at some point though.

1:40 am | Posted in Computing, Linux, Warwick Blogs | 2 Comments » | Show comments » | Hide comments » |

Well, it’s been a whole load of time since I posted, mainly due to revision and lazyness on my part. So I thought I’d post a quick update on what I’m doing – not that anyone will be all too interested.

So I’m back in Walsall after spending a week down at Sarah’s. Unfortunately, things haven’t been brilliant on the revision front. Whilst I’m mildly optimistic about Combinatorics, Introduction to Topology and to a lesser extent Topics in Mathematical Biology, I know that Measure Theory is going to be awful. I’ve not done a lot of work for it, and to be honest, I just don’t understand a lot of the material.

However, on the upside, I’ve finished all of my Topology and Topics exercise sheets, which is something that has never been done before. I’m now at the stage where, a week before the dreaded exams, I’m getting my act together and doing three past exam papers a day. So maybe there’s hope yet; we shall see, I suppose. I figure that even if Measure Theory goes badly, I can still pull it up with some better marks from the other modules.

However, I didn’t really want to talk about exams so I’ll talk about something equally as boring – programming. Been playing around quite a bit with two things over the holidays. Firstly – and this has been on the agenda for some time now – coding an OpenGL GStreamer application to map video onto a cube or some other random object. Surprisingly, this was initially very easy, once I’d found out how to actually get the contents of the GStreamer buffer into my program.

Basically I use decodebin to get the video playback working, and then pipe it through ffmpegcolorspace to get a nice RGB texture. Then, I use fakesink and a handoff function to actually generate the texture to screen and map it onto a cube. The hardest bit was actually finding out about the handoff function – it’s buried deep inside the GStreamer Plugin Guide, which is quite annoying. However, it does look quite cool. You can even take a look at it for yourself if you’d like.

Even so, I wasn’t quite satisfied. The biggest speed hump is the ffmpegcolorspace conversion, as generically most video files use some kind of YUV format to store video frames. Typically, and somewhat ashamingly, there’s no general way of dealing with YUV textures in OpenGL – all of the extensions out there generally suck, and don’t work for different versions. Plus, none of them cope with 4:2:0 planar YUV, only 4:2:2 pixel-packed for the most part. So some form of conversion to RGB is required, and a significant amount of CPU overhead is incurred in the process.

For the last couple of weeks I’ve been looking at various ways of efficiently using the GPU to do this conversion for me. Colour matrices were the first option I looked at. Basically, given some texture, one can use a colour matrix to remap colours in any given format. However, the major problem is going from 4:2:0 planar to 4:4:4 pixel packed, and that itself incurs quite a significant overhead.

Blending textures is possibly the best way of doing this. Basically, one generates 5 different textures along with using a simple colour-lookup table. Firstly, we lay down the Y channel as a luminance channel. Then the additive U, then subtractive U, and the same for V. In effect, just layering one channel on top of another to simulate the YUV->RGB conversion in the GPU.

However, there are again problems with this. The naive approach is to redraw texture upon texture, which means that every pixel needs to be redrawn 5 times every frame – at 25fps, that’s 125 times/second! In order to get the most out of the GPU, I used the ARB_multitexture extension, which allows you to layer textures extremely quickly in one pass. I also delved into the world of shaders to try and work out whether it might even be worth usnig only 3 textures and doing the RGB conversion through a simple ARB fragment program. This is something I’m still looking into.

Unfortunately, although this actually works and I’ve tested it, it’s still very very slow. However, I’m using quite a crap Intel 915GM based GPU under Linux. The i915 driver is not particularly good at the moment, and upgrading to Mesa 6.5 seems to have made matters even worse. glxgears is showing only 600fps – I should be getting at least 1300 according to other guys with the same card. So I’m guessing – and hoping – that the problem lies with the drivers and not my card. When I get back, I’m going to check it out on my Radeon 9800 and we’ll see how that copes.

Anyway, that’s probably enough for now. I’m going to reboot and see if I can get this all working nicely.

10:00 pm | Posted in GStreamer, Linux, Maths, Revision, Warwick Blogs | 3 Comments » | Show comments » | Hide comments » |

Literally 20 seconds after I say that, something else has broken. There is a seriously major bug in this output code somewhere, and I can’t find it. I think I’m going to revert back to a much simpler way of handling the output than this concat() stuff and see if that helps. If not, I’m going to sit down and beat my computer.

2:27 am | Posted in Computing, Linux | No Comments »

Been having some wierd bugs turn up recently in my code. Stuff like simpling defining an integer array completely messes up an unrelated routine and causes my methods to diverge off to infinity. Very odd stuff indeed.

Switched the compiler back to 3.3, and am having absolutely no problems at all. Lucky that, because up until this afternoon I was seriously considering getting rid of the 3.3 branch. Very glad I didn’t now.

I suppose it could be my code that’s buggy, but considering it works with 3.3 flawlessly I’m wondering whether I’ve managed to somehow break my toolchain. I’ll just have to find another copy of 3.4 on another box and see if I run into the same problems.

All in all, a very odd day.

2:22 am | Posted in Computing, Linux | No Comments »