Iteration: the repetition of a process or utterance.

I started a project a while back, neglectfully named Pixelate. It had a very loose spec. Namely, I wanted to make a nerdy background for my new portfolio site’s landing page. I wanted something that I could create a pixelated representation of an image I either took or stole from someone else on the Internet (sorry!). I wanted it to pack a little extra punch, though - something the discerning, curious developer would be able to dig in to and discover. At ClearSlide, I spent a fair amount of time researching and playing with different image analysis techniques. One of my favorite projects was creating animated images using overlaid transparent PNGs derived from video frame differentials. Say that ten times fast. With that in mind, I wanted to process an image, bit for bit, and output a matrix, rendered using various cross-browser compatible techniques. I decided to start with html divs.

The Pixelate Unicorn

Original Unicorn
Pixelate.js Unicorn!



<div class="row">
  <div class="block" style="background: rgb(123, 29, 212);"></div>
  <div class="block" style="background: rgb(123, 29, 212);"></div>
  <div class="block" style="background: rgb(123, 29, 212);"></div>
  <div class="block" style="background: rgb(123, 29, 212);"></div>
</div>
<div class="row">
  <div class="block" style="background: rgb(123, 29, 212);"></div>
  <div class="block" style="background: rgb(123, 29, 212);"></div>
  <div class="block" style="background: rgb(123, 29, 212);"></div>
  <div class="block" style="background: rgb(123, 29, 212);"></div>
</div>
<div class="row">
  <div class="block" style="background: rgb(123, 29, 212);"></div>
  <div class="block" style="background: rgb(123, 29, 212);"></div>
  <div class="block" style="background: rgb(123, 29, 212);"></div>
  <div class="block" style="background: rgb(123, 29, 212);"></div>
</div>
<div class="row">
  <div class="block" style="background: rgb(123, 29, 212);"></div>
  <div class="block" style="background: rgb(123, 29, 212);"></div>
  <div class="block" style="background: rgb(123, 29, 212);"></div>
  <div class="block" style="background: rgb(123, 29, 212);"></div>
</div>

Pretty straight forward, right? I got a working prototype together in about thirty minutes. It was slow, I could break it pretty easily, and it looked like crap, but the fact of the matter was it worked!

The prototype:

var imageSource = 'images/unicorn.jpg';

var img = new Image();
img.src = imageSource;
var canvas = $('<canvas/>')[0];
canvas.width = img.width;
canvas.height = img.height;
canvas.getContext('2d').drawImage(img, 0, 0, img.width, img.height);
var context = canvas.getContext('2d');

var pixelDensity = 150;

var timerStart = new Date();

for (var i = pixelDensity/2; i < img.height; i += (img.height/pixelDensity) ) {
    $('.container').append($('<div class="row">'));
    for(var j = pixelDensity/2; j < img.width; j += img.height/pixelDensity) {
        var value = context.getImageData(j, i, 1, 1).data;
        var colorValue = 'rgb(' + value[0] + ', ' + value[1] + ', ' + value[2] + ')';
        $('.row:last').append($('<div class="block">').css({'background-color' : colorValue}));
    }
}

var timerStop = new Date();

console.log(timerStop - timerStart + ' ms');

What’s happening here is, I’m taking an image from the document, creating a canvas and drawing the image to the canvas’s context. Then, I’m iterating, brutally, over the context, pulling pixel color data based on my pixelDensity value (pixel density is a poorly named representation of the distance between color samples from the original image). From there, I’m creating block divs, assigning them a background color value based on the sampling, loading those blocks in to rows, which I’m then stacking.

With an average image, using a pixel density of 150, the processing time was roughly eight to ten seconds. That was when it actually worked. Increasing the pixelDensity drastically reduces both the speed of the process and the likelihood of it actually completing. If you want to, you can melt your computer with the Pixelate.js prototype.

Commence head banging

From here, I was pretty much lost. I obviously needed something that worked all of the time, and it had to be fast. I found a similar technology called jscii which was able to transform a live video stream. I obviously had to do that! So, I had a new goal. Not just to make an image pixelator, but to make a real time video filter. Was this even possible? Are imageData arrays just too heavy to get the kind of performance I was after. I dug in to jscii’s algorithm and was surprised to find that it was very similar to mine, albeit pluginified (that’s a word, right?) and much more efficient. The author also avoided using jquery, which upon further investigation is generally a good idea with data heavy processing stuff.

So with that in mind, I had a long term goal and an immediate task to get one step closer to that goal. I refactored my code to not use jquery.

var imageSource = 'images/unicorn.jpg';

var img = new Image();
img.src = imageSource;
var canvas = document.createElement('canvas');
canvas.width = img.width;
canvas.height = img.height;
canvas.getContext('2d').drawImage(img, 0, 0, img.width, img.height);
var context = canvas.getContext('2d');

var pixelDensity = 150;

var timerStart = new Date();

var container = document.getElementById('output')

for (var i = pixelDensity/2; i < img.height; i += (img.height/pixelDensity) ) {
    var row = document.createElement('div');
    row.className = 'row';
    for(var j = pixelDensity/2; j < img.width; j += img.height/pixelDensity) {
        var value = context.getImageData(j, i, 1, 1).data;
        var colorValue = 'rgb(' + value[0] + ', ' + value[1] + ', ' + value[2] + ')';
        var block = document.createElement('div');
        block.className = 'block';
        block.style.background = colorValue;
        row.appendChild(block);
    }
    container.appendChild(row);
}

var timerStop = new Date();

console.log(timerStop - timerStart + ' ms');

Ok. So we’ve eliminated a huge dependency. Let’s run it. Down to 7ish seconds. Now we’re moving. Not quite as much as I had hoped, but we were getting places. I started breaking the process down to see where the actual time suck was happening. I broke the iterator down in to different version, but for the most part couldn’t really find the culprit. At this point I started getting frustrated, so I asked around and was offered a few different pieces of advice.

  • Why aren’t you using canvas, idiot?
  • Don’t touch the DOM until the very last minute
  • Cache your iterators
  • Don’t call getImageData in your iterator

So I made the changes:

var img = document.getElementById('image');
var canvas = document.createElement('canvas');
canvas.width = img.width;
canvas.height = img.height;
canvas.getContext('2d').drawImage(img, 0, 0, img.width, img.height);
var context = canvas.getContext('2d');
var pixelDensity = 150;

var timerStart = new Date();
var wrapper = document.createElement('div');
wrapper.className = 'image';
var imgData = context.getImageData(0, 0, img.width, img.height).data;
var getRGB = function(i) { return [imgData[i], imgData[i + 1], imgData[i + 2]]; };

var start = Math.round(pixelDensity / 2),
    inc = Math.round(img.height / pixelDensity);

for (var i = start, imax = img.height; i < imax; i += inc) {
    var row = document.createElement('div');
    row.className = 'row';
    for (var j = start, jmax = img.width; j < jmax; j += inc) {
        var colorValue = getRGB((i * (img.width * 4)) + (j * 4));
        var block = document.createElement('div');
        block.className = 'block';
        block.style.backgroundColor = 'rgb(' + (colorValue.join(',')) + ')';
        row.appendChild(block);
    }
    wrapper.appendChild(row);
}

var container = document.getElementsByClassName('container')[0];
container.innerHTML = '';
container.appendChild(wrapper);
console.log(new Date() - timerStart + ' ms');

The key differences in this iteration are the cached iterator maximums imax and jmax and the fancy new variable imgData and it’s companion method getRGB. These were the reason I almost fell out of my seat when I ran this version teh first time. The image that originally took nine seconds to process, now completed in 100 milliseconds. At lower pixelDensity, the process is capable of sub 20ms processing times, but the images aren’t really legible. My current goal is turn refactor the script in to an extensible object that takes arguments and can dynamically change state. I’d also like to experiment rendering the results in a canvas, which I think will allow me to get the processing time down for higher pixelDensities. The current iteration of the project looks like this:

var Pixelate = function (imagePath, output, options) {

  // Quit if no imagePath or output
  if (!imagePath || !output) return;
  options = options || {};

  // Set options
  var start = Math.round(pixelDensity / 2) // @TODO - Pretty sure this can be set to 1 every time.  Need testing.
    , inc = Math.round(image.height / pixelDensity)
    , pixelDensity = options.pixelDensity || 130 // @TODO - Should probably limit this to a reasonable range
    , pixelStyle = options.pixelStyle || 'square' // @TODO - Square or circle
    , pixelBorder = options.pixelBorder || 'none' // @TODO - add border options
    , image
    , canvas
    , context
    , imgData
    , getRGB
    , wrapper
    , container;

  function setup () {
    // Output container
    wrapper = document.createElement('div');
    wrapper.className = 'image';
    container = document.getElementById(output);
    // Load image context
    canvas = document.createElement('canvas');
    loadImage();
  }

  function loadImage () {
    image = new Image;
    // On the image load, we're going to write our data to the canvas
    image.onload = function () {
      canvas.width = image.width;
      canvas.height = image.height;
      canvas.getContext('2d').drawImage(image, 0, 0, image.width, image.height);
      imgData = context.getImageData(0, 0, image.width, image.height).data;
      // Get image pixel data and build an array with it
      getRGB = function(i) { return [imgData[i], imgData[i + 1], imgData[i + 2]]; };
    }
    image.src = imagePath;
  }

  function clearOutput () {
    container.innerHTML = '';
  }

  function changeBlockSize (blockSize) {
    blocks = document.getElementsByClassName('block');
    for (var i = 0, imax = blocks.length; i < imax; i++) {
      block = blocks[i]
      block.style.width = blockSize;
      block.style.height = blockSize;
    };
  }

  function drawOutput () {
    // body...
  }

  function renderFrame () {
    for (var i = start, imax = image.height; i < imax; i += inc) {
      var row = document.createElement('div');
      row.className = 'row';
      for (var j = start, jmax = image.width; j < jmax; j += inc) {
        var colorValue = getRGB((i * (image.width * 4)) + (j * 4));
        var block = document.createElement('div');
        block.className = 'block';
        block.style.backgroundColor = 'rgb(' + (colorValue.join(',')) + ')';
        row.appendChild(block);
      }  
    }
    wrapper.appendChild(row);
    clearOutput();
    container.appendChild(wrapper);
  }

  // Expose the Pixelate.js API
  return {

    setup : function () {
      setup();
    },

    clearOutput : function () {
      clearOutput();
    },

    changeBlockSize : function (size) {
      changeBlockSize(size);
    },

    renderFrame : function () {
      renderFrame();
    }
  }
}

If you’d like to contribute to pixelate.js, the project is hosted on Github!

StatusPage is hiring engineers in San Francisco, Denver and Durham. If you'd like to be part of our incredible team, follow the link, or shoot me an email at tyler at statuspage dot io. We'd would love to chat with you.