HTML5 Video
+ Canvas
= Woah!

Wes Bos

May 31, 2012


Hey there, I'm Wes Bos

Independent Front End Designer & Developer

From Toronto, Canada

I design and build the front end of applications and websites.

Huge JavaScript / CSS3 / HTML5 Fan.

I hang out on Twitter @wesbos

This Talk...

Will review the basics of HTML5 Video, it's API and then move into video manipulation.

Is heavy on code and demos so cross your fingers for no bugs!

The Basics

HTML5 video is a new element used to embed video into your site that you use along with the rest of your HTML tags.

It plays nice and interacts with CSS and other HTML5 features such as canvas.

Your browser ships basic video controls but we can use JavaScript to control the HTML5 video player.

Support or UGH IE

Support for HTML5 video is fantastic!

Chrome 3+, FF 3.5+, Opera 10.5+, Safari 3.1+, IE9+

Most mobile phones and tablets made in the past 3 years support HTML5 video. (to a point)

In other words, all these browsers have supported it for a long time.

The only browsers that don't support HTML5 Video are IE7 and 8 where we would reqire a flash fallback.

Lets take a look at the code

<video controls>
	<source src="Chrome_ImF.mp4" type="video/mp4;">
	<source src="Chrome_ImF.webm" type="video/webm;">		        
	<source src="Chrome_ImF.ogv" type="video/ogg;">
    <p>Your browser sucks, insert a flashfallback here.</p>

The browser vendors haven't agreed on a codec so we have to serve up three :(

Chrome, Opera and FF4+ = .webm
IE = .MP4 (and .webm if installed)
FF 3.5,36 = .ogg
safari/ios =.MP4

The order above is important for ios compatability.

Miro Video Converter is your friend!

Notable Attributes

The video element currently has 12 attributes and is constantly in flux as the spec is still changing.

autoplay - autplays the video (duh). Note: doesn't work on ios

preload - Tells the browser to preload nothing, the metadata(length etc..) or the entire video

controls - Set to false to hide the browsers native controls

poster - placeholder image to be displayed while the video loads.

Enough of the boring stuff, lets see some JS!

First, select your video

var vid = document.querySelector('#myVid'); 

Then use the methods to control it: play, pause, skip ahead, play, volume up.; vid.pause(); vid.currentTime+=10; vid.volume+=0.1

You can control the playback rate, volume and current time which is enough to make your own video player UI

You also have access to all the data/properties of the video element.

vid.duration; vid.currentTime;


Like any element, you can also subscribe to events. Thre are a number that are specific of HTML5 video/audio

These are handy when you are building an application that works with video

vid.addEventListener('seeking', function() {
	console.log("a scrub is a guy who thinks he's fly");
vid.addEventListener('seeked', function(e) {
	ga(e.type,this.currentTime); // send seek data to analytics

An awesome reference to all methods, events and properties

Creating a Custom Player

So, given that we have access to all the video's meta data, we can build an entire player with HTML and CSS and then use javascript to update the players scrub bar, time, volume and so on.

A few good players:

VideoJS -

JWPlayer -


All of the above have great flash fallbacks for when HTML5 video is not supported

Add a dash of CSS3

Because HTML5 Video is just another HTML element, we can use CSS to transform and style our player - with or without our HTML controls

HTML5 Video Manipulation!

HTML5 video plays nice with Canvas.

Canvas is another part or HTML5 that enables pixel based drawing.

Its kind of like MS paint, but for the browser.

We can take each frame from our video element and dump them into a canvas element!

Super Basic Example

Video Element

Canvas Element

On a set interval or requested animation frame we take a frame from the video and use the canvas's ctx drawImage() method to write part of all of the frame.

ctx.drawImage(videoEl, x, y, width, height);

Cool, cool cool cool.

But what can you do with that?

Slice and Dice

While the video is playing, we can cut it into tiles and then spit them into a canvas at a random order.

Kind of a silly example, but it gives you the idea that we have total control over how and where the pixels are displayed

Filters and Pixel Manipulation

Canvas has a method called getImageData() which returns an array of every single pixel available.

Better yet, we have access to the Red, Green, Blue and Alpha Channels of every pixel.

We can play aroud and change these values to whatever we want and then stick them back into the canvas.

Some Code

Lets step through a basic example. First setup your HTML.

<video id="example4" width="700" height="350">
      <source id="mp4" src="localVid.mp4" type="video/mp4">
<canvas id="example4Canvas" ></canvas> 

Now with JS, grab hold of your elements and get the canvas's context.

var example4 = document.querySelector('#example4'),
	canvas4 = document.querySelector("#example4Canvas"),
	ctx4 = canvas4.getContext("2d"),

We also created a variable called 'example4Pixels' to hold our pixels once we nab them from the canvas.

Now that we are set up, we have 4 steps that will repeat every 50ms

or at whatever framerate your choose.

  1. Take the frame from the video and write it to canvas
  2. Grab the pixels from the canvas and store in our example4Pixels variable
  3. Loop over each pixel and adjust your RGBA channels accordingly
  4. Spit those pixels back into the canvas

Note: For simplicity, I'm using a single canvas but its often a good idea to have two: One for dumping the pixels into and one for dumping the manipulated pixels.

1. Take the frame from the video and write it to canvas
ctx4.drawImage(example4, 0, 0, example4.width, example4.height);

The above code grabs the current frame from our 'example4' video and writes to canvas.

2. Grab the pixels from the canvas and store in our example4Pixels variable
example4Pixels = ctx4.getImageData(0,0,canvas4.width,canvas4.height);

ProTip: Don't console.log() this variable as its HUGE and will crash your browser.

3. Loop over each pixel and adjust your RGBA channels accordingly

This is where the fun parts comes in. Every 4 numbers represent 1 pixel.

100		235		54		0.35 	34		245		199		1
// Loop over RGBA
for (i = 0; i <; i=i+4) {[i + 0] =[i + 0] + 100 ; // RED[i + 1] =[i + 1] - 50; // GREEN[i + 2] =[i + 2] * 0.5; // Blue


In this example, I upped our RED by 50, decreased GREEN by 50 and multiplied BLUE by 0.5. Alpha is left at 1.

The end result is a red tint on the the video.


( or HTML5 Video & Audio Hardware Access! )


WebRTC? Real Time Communication

There is something new coming down the pipe called WebRTC which allows browsers to communicate directly.

Still heavily under development but we are able to create something called a PeerConnection which allows for a 1:1 channel between two peers.

We can stream audio, video and eventually data/files over this connection without relying on a server in the middle.

We would only need websockets/XMLHttpRequest to do the initial handshake

The real benefit here is that we will soon be seeing HTML5 Chatroulette!

Say Hello to getUserMedia()

PeerConnections aside, this new spec means that browsers are currently implementing native webcam and microphone access!

Thats right, no flash required!

getUserMedia is currently supported in Opera, Opera Mobile 12 and Google Chrome Canary (or dev channel)

Will hit boot2geko in August-ish

Cool, demo it!

Start Stop Hipster Threshold Blur Green HTML5 Glasses

A quick word on performance

Iterating on a large canvas takes more processing power

When doing pixel analysis, if you can, its often a good idea to run the process on a smaller version of the canvas.

I used to get 1-2 FPS on face detection.

Not even web workers would make it faster!

Simply running the face detection script on a 100px x 100px canvas improved performance by at least 10 fold

Large Canvas === slow === better accuracy

Smaller Canvas === fast === less accurate

Larger Canvas Demo

HTML5 Video + Canvas in the real world


Built by Bocoup / Mozilla

Allows you to create interactive HTMl5 web pages that sync up with a video/audio element

Think cuepoints on steroids

Huge Library of plugins to work with everything from twitter to Google Maps Street View


Build by the super awesome Rick Waldron

Uses WebRTC, Node.js and web sockets to create an online photobooth and save images to the server

100% JavaScript!

Source on Github and demo at

getUserMedia() Shim

So all this stuff is great, but its going to be forever until we can use it, right?!

Nope! We can use Shims and Polyfills to recreate the functionality for older browsers

Addy Osmani has written a shim that provides a flash fallback

So much cool, so little time

Check out a ton more video experiments at Chrome Experiments

Augmented Reality

In the same way we analyzed the pixels in the last examples, we can use an Augmented Reality lib to detect markers.

Here we use JSAR Toolkit which was ported from flash and a wrapper written by f1lt3r/bocoup



HTML5 Video is awesome

You can use Javascript to control every aspect of it

Canvas makes manipulating video fun and easy

Go make cool things!

Cool Stuff EH?
The future is awesome

@wesbos on Twitter