• Blogs
• Silverlight Mosaic Generator

Photo Mosaic Generator - Fun Adventures With Silverlight

30 January 2009 - 2:30 AM / by Dominic Pettifer. 2 Images for Photo Mosaic Generator - Fun Adventures With Silverlight. 3 Comments for Photo Mosaic Generator - Fun Adventures With Silverlight.

Personal Blog - I've been learning Silverlight and have decided to write a Photo Mosaic Generator with it. Check out some of the cool Mosaics I've made already.

Photo Mosaics with Silverlight

I've been learning to program Silverlight applications with a lot of help from Matthew MacDonald's excellent book, Pro Silverlight 2 in C# 2008. I highly recommend this book if you're getting into Silverlight development, not for those new to C# though. Matthew MacDonald is one of my favourite authors and has written many books on .NET development. Anyway I digress.

I thought of a cool idea for a SIlverlight application. A Photo Mosaic Generator application. A Photo Mosaic is a large picture made up of 100s of smaller pictures, where each individual pic has been selected and placed because it roughly matches the colour and shapes of the tile in the larger picture. When pieced together they all resemble the larger image, the effect is quite cool, especially when you zoom in. Check out the Wikipedia article for some examples, but I've included some examples I made myself here.

Algorithm

These come from a simple C#.NET console application I wrote. Actually it's two console apps. One app trawls the photo sharing website Flickr using it's API/Web services. As it downloads each image, it scans each pixels RGB colour value to work out the average colour for the entire image, and stores that colour value and the image's meta data in a database. The other app generates the Mosaics. It starts by accepting a photo as input, breaks up that photo into tiles in a grid fashion and works out the average colour value for each tile. Then it searches the database for the closest colour, and downloads that photo from Flickr. All the photos are pieced together to make up the final Mosaic.

I confess this was a rather simplified explanation, the algorithm is a little more complicated, each tile is further broken down into smaller sub tiles (5x5 = 25 sub tiles), same with photos indexed from Flickr. This is to allow better, more accurate matches for each individual photo in the Mosaic. It also flips the images allowing for double the potential matches. Here's some code:

public Photo FindMatchForTile(IList<Color> colorsFromTile)
{
    var photos = from p in Photos
                 orderby Math.Abs(CompareColors(p.Colors,
                     colorsFromTile)) ascending
                 select p;

    return photos.FirstOrDefault();
}

public int CompareColors(IList<Color> cols1, IList<Color> cols2)
{
    int total = 0;

    for (int i = 0; i < 25; i++)
    {
        Color color1 = cols1[i];
        Color color2 = cols2[i];

        total += ((color1.R - color2.R) * (color1.R - color2.R));
        total += ((color1.G - color2.G) * (color1.G - color2.G));
        total += ((color1.B - color2.B) * (color1.B - color2.B));
    }

    return total;
}

The algorithm works similarly to calculating the distance between 3D points, imagine that the RGB (Red, Green, Blue) values (which each can be between 0 to 255) are 3D coordinates, and you want to find the closest point to another point.

Problems

This works well, the console app version was more a proof of concept to see if it worked. Well it does, kind of! While the resulting photo Mosaics are great, it currently takes a very, very long time to generate them. I've indexed 120'000 photos from Flickr so far. If you generate a Mosaic consisting of 30x40 tiles, that's 1200 tiles. The code has to find a closest match among 120'000 images for each of the 1200 tiles in the Mosaic to be, and then each individual tile is further dividing up into 5x5 smaller tiles.

Well, lets say my Intel Core 2 Duo 2.2GHz takes a few minutes to perform this match, a few minutes is a long time on the web. This was executing on a single thread, multi-threading could reduce this time down to a quarter on a quad core processor potentially. However, this is still too long and under heavy load, even the fastest server will get hammered to a bloody pulp as a single user is essentially loading all four cores for a sizable duration of time. I believe I have an answer to this problem, but I don't want to tell you just yet for fear someone might steel my idea :-) Just watch this space.

Why Silverlight?

So why use Silverlight? Well, the photo supplied by the user needs to be divided up and the average colours calculated and the final image generated. This obviously can't be done with client-side Javascript, and it's too computationally intensive to be done on the server. There's also bandwidth considerations, the server would need to download all of the images from Flickr itself then send the final image to the client, I can't afford that much bandwidth. The beauty with Silverlight is that all this can be done on the client-side, using the client computer's CPU and memory resources, taking the load off the server.

Also Silverlight has an amazing technology called Deep Zoom that lets you zoom into a hi res photo, individual parts of the photo are seamlessly downloaded as you zoom in to reveal more detail. Perfect as Flickr offers all its photos in three different sizes, thumbnail, medium and original size. Small problem I have at the moment, I'm currently researching Silverlight's Deep Zoom and there doesn't seem to be a way to generate the Deep Zoom images dynamically on demand. You seem to need to use a Deep Zoom image generator tool and manually create the Deep Zoom images ahead of time. Anybody know a way around this?

I've registered www.photomogen.com and www.photomosaicgenerator.com (currently parked, sorry). PhotoMoGen (basically short for Photo Mosaic Generator) is the name I’ll use for the application. If you think up a better name, let me know :-)