40win/FTW

a home for my bits

Web Gestures With getUserMedia: Part1

A few weeks ago I read a blog post by Tim Taubert where he talked about building a green screen with his webcam, a canvas and a green sheet of paper. His demo ran in Firefox, and it was pretty slick. Here his a video, check it out. <iframe src=”http://player.vimeo.com/video/51593914?badge=0” width=”500” height=”191” frameborder=”0” webkitAllowFullScreen mozallowfullscreen allowFullScreen></iframe>

A few days later I went to the October meeting of the Utah Google Developers Group where one of the presenters (Mike Heath) did a separate demo where he used the getUserMedia webcam feed and a canvas to attempt to detect movement. He was looking for hand movement, but wasn’t able to successfully reduce the amount of noise in the background where movement would appear to happen due to random lighting changes.

After watching his demo, I thought about combinin the two demos to build a motion detection screen using a green sheet of paper. Tonight I finished part 1 of that demo. It is pretty nifty.

What does it do?

So, what did I manage to get up and running this evening? As you will see, when I move the green card around the screen, there is a floating div that follows it’s movement on the page. As I move the card up on the webcam feed, the div will move up. As I move the div down/sideways/up/back the div follows the card on the page. Here is a video of what I got going. <iframe width=”420” height=”315” src=”http://www.youtube.com/embed/h13LgUV_ZZg” frameborder=”0” allowfullscreen></iframe>

There is a bit going on here. While most of my code is new, I did shamelessly take the rgb2hsl function from Tim’s live demo. The rest of it is mine. Let me walk you through it.

1. Get the webcam feed

You should note upfront that this is a demo that should be run in Chrome. It uses the webkitGetUserMedia, which will fail in IE, FF and Safari.

To get started we need some basic HTML. Here is what that will look like:

[Basic HTML]
1
2
3
4
5
6
7
8
9
10
<html>
	<head>
		<script type="text/javascript" src="main.js"></script>
	</head>
	<body style="background-color:red;">
		<video id="v" width="320" height="240"/>
		<canvas id="c" width="320" height="240"></canvas>
		<div id="highlight"></div>
	</body>
</html>

Now that we have that html, let’s lay some tracks to feed the webcam into our video element.

[getUserMedia] []
1
2
3
4
5
6
7
8
9
10
11
var v = document.querySelector("#v"),
	c = document.querySelector("#c"),
	x = c.getContext("2d"),
	hl = document.querySelector("#highlight"),
	pixels;

navigator.webkitGetUserMedia({video:true},function(stream){
	v.src = URL.createObjectURL(stream);
	v.play();
	setTimeout(draw,200);
});

On line ‘7’ you can see that we make our call to ‘webkitGetUserMedia’, passing ‘{video:true}’ as our first param, and a callback function as our second parameter. Once the method is invoked the user is asked if they would like to share their webcam feed. If the user accepts, then the callback is called. In our callback, we create an ObjectUrl from the video stream and then set that to the ‘src’ of the video. After which we tell the video to play, and then call a setTimeout, which will run our ‘draw’ method. You haven’t seen the draw method, but it contains all of our code. Let’s break it down a step at a time.

2. Draw the video feed onto our canvas

The first few lines of our ‘draw’ method will draw the video onto a canvas. Here is how we do that:

[Load Video Into Canvas] []
1
2
3
4
5
6
	var w = v.width,
		h = v.height;
	
	//Draw the current video frame to the canvas
	x.drawImage(v, 0, 0, (w), h);
	

3. Find Green Pixels in the Canvas

Now that we have the canvas loaded up with a snapshot from the video, we need to find the green pixels so that we can track them. We track them for two reasons. The first is so that we can turn them transparent, so that you can get the nice visual feedback on the page and see which pixels it detects as green, and second so that we can then go through and look through different neighborhoods of pixels and score the different parts of the screen to see which ones have the most concentration of green pixels. We will get to that in a second. For now, take a look at how we detect green pixels:

[Find Green Pixels] []
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
	var pixels = x.getImageData(0, 0, w, h);
	var pixLength = pixels.data.length / 4;
	
	//map: make two dimensional array to store which pixels detect green
	//scores: 2d array to store the 5x5 scores for each pixel. Each pixel
	//	gets a score of the summary of the green pixels around it. It looks
	//	at the 5 pixels to the left, right, above and below the pixel. The
	//	pixel gets the score of the sum of that total.
	var map = new Array(w);
	var scores = new Array(w);
	for(var i = 0; i < w; i++){
		map[i] = new Array(h);
		scores[i] = new Array(h);
	}
	//load the map with 1 and 0 for green and non-green pixels respectively
	for(var i = 0; i < pixLength; i++){
		var index = i*4;
		var r = pixels.data[index],
			g = pixels.data[index+1],
			b = pixels.data[index+2];
		var hsl = rgb2hsl(r, g, b),
			ha = hsl[0],
			s = hsl[1],
			l = hsl[2];
		
		var left = Math.floor(i%w);
		var top = Math.floor(i/w);
			
		if (ha >= 70 && ha <= 180 &&
			s >= 25 && s <= 90 &&
			l >= 20 && l <= 95) {
				
				pixels.data[i * 4 + 3] = 0;
			map[left][top] = 1;
		}else{
			map[left][top] = 0;
		}
	}

Let’s walk throuh these lines of code. We have over 30 lines of code here, and I don’t want to lose anyone.

First we pull the imageData out of the canvas. The ‘imageData’ and an object that holds all the RBG and Alpha values for each pixel in our canvas. We pull that out so that we can analyze it in a few lines.

Second, on line 9 and 10 we create two new 2-dimensional Arrays. These Arrays are as wide as the canvas and as tall as the canvas. They are the same dimensions as the canvas. The ‘map’ array[] will be used because I want to score each pixel with a 1 or a 0. If the pixel was green it gets a score of 1. If it was not green, it gets a score of zero. On lines 18-24, I turn the RGB into HSL values. Then on line 29, I verify if it was a green-ish pixel or not. If it was green-ish enough, the map gets a value of 1 for that pixel. If not, it gets a value of 0. I didn’t use the ‘scores’ array[], but I will in a minute.

4. Find the section with the most green-ish pixels

So at this point we have our map, with is a 2D array that has a 1 or a 0 for each pixel in our image. How do we use this to find the part of the screen with the most green-ish pixels? We are going to check out the entire neighborhood. That’s right, we are going to perform a Neighborhood Operation to find out what the surrounding pixels look like. If I have a pixel that is surrounded by all green-ish stuff, we will add up his 1 and the 1’s of all of his neighbors, and that will be his score. So, each pixel will get scored based on the summed values ofitself and it’s neighbors. Once we do that, we just have to go through the scores and find the first pixel with the highest score (if other later pixels tie, we ignore them). See the code for perhaps a better explanation:

[Sum Each Pixel by Neighborhood] []
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
	//sum the score for each pixel
	for(var j = 5; j < h-5; j++){
		for(var i = 5; i < w-5; i++){
			var l5 = map[i-5][j],
				l4 = map[i-4][j],
				l3 = map[i-3][j],
				l2 = map[i-2][j],
				l1 = map[i-1][j],
				r1 = map[i+1][j],
				r2 = map[i+2][j],
				r3 = map[i+3][j],
				r4 = map[i+4][j],
				r5 = map[i+5][j],
				u5 = map[i][j-5],
				u4 = map[i][j-4],
				u3 = map[i][j-3],
				u2 = map[i][j-2],
				u1 = map[i][j-1],
				d1 = map[i][j+1],
				d2 = map[i][j+1],
				d3 = map[i][j+1],
				d4 = map[i][j+1],
				d5 = map[i][j+1],
				self = map[i][j];
			//console.log(i,j);
			scores[i][j] = l5+l4+l3+l2+l1+r1+r2+r3+r4+r5+u5+u4+u3+u2+u1+d1+d2+d3+d4+d5+self;
		}
	}

Now the ‘scores’ array is populated with the highest scores. Now we just have to go find the highest scoring pixel and set move our ‘highlight’ div to a position relative to that, adjusted from the size of the canvas to the size of the entire screen.

[Find The Highest Score] []
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
	//Find the pixel closest to the top left that has the highest score. The
	//	pixel with the highest score is where the highlight box will appear.
	var targetx = 0;
	var targety = 0;
	var targetscore = 0;
	for(var i = 5; i < w-5; i++){
		for(var j = 5; j < h-5; j++){
			if(scores[i][j] > targetscore){
				targetx = i,
				targety = j;
				targetscore = scores[i][j];
			}
		}
	}
	hl.style.left = ""+Math.floor(document.width*(targetx/v.width))+"px";
	hl.style.top = ""+Math.floor(document.height*(targety/v.height))+"px";
	x.putImageData(pixels, 0, 0);


	setTimeout(draw,200);

Notice that on lines 169 and 170 I translate the location of the pixel inside the canvas into a location on the page.

At that point, I set the image on the canvas back to the updated images, and I call another setTimeout so that this will happen all over again.

Feel free to checkout the LIVE DEMO!. NOTE: You will need something very chartreuse green in order to get it to work correctly.

Comments