HTML5 Video
+ Canvas
= Woah!

Wes Bos
wesbos.com
@wesbos

March 24, 2012

FITC #SpotlightJS

Hey there, I'm Wes Bos

Independent Front End Designer & Developer

I design and build the front end of applications and websites.

Huge JavaScript / CSS3 / HTML5 Fan.

I hang out on Twitter @wesbos

This Talk...

Will review the basics of HTML5 Video, it's API and then move into video manipulation.

Is heavy on code and demos so cross your fingers for no bugs!

The Basics

HTML5 video is a new element used to embed video into your site that you use along with the rest of your HTML tags.

It plays nice and interacts with CSS and other HTML5 features such as canvas.

Your browser ships basic video controls but we can use JavaScript to control the HTML5 video player.

Support or UGH IE

Support for HTML5 video is fantastic!

Chrome 3+, FF 3.5+, Opera 10.5+, Safari 3.1+, IE9+

Most mobile phones and tablets made in the past 3 years support HTML5 video. (to a point)

In other words, all these browsers have supported it for a long time.

The only browsers that don't support HTML5 Video are IE7 and 8 where we would reqire a flash fallback.

The Code

<video controls>
	<source src="Chrome_ImF.mp4" type="video/mp4;">
	<source src="Chrome_ImF.webm" type="video/webm;">		        
	<source src="Chrome_ImF.ogv" type="video/ogg;">
    <p>Your browser sucks, insert a flashfallback here.</p>
</video>

The browser vendors haven't agreed on a codec so we have to serve up three :(

Chrome, Opera and FF4+ = .webm
IE = .MP4 (and .webm if installed)
FF 3.5,36 = .ogg
safari/ios =.MP4

The order above is important for ios compatability.

Miro Video Converter is your friend!

Notable Attributes

The video element currently has 12 attributes and is constantly in flux as the spec is still changing.

autoplay - autplays the video (duh). Note: doesn't work on ios

preload - Tells the browser to preload nothing, the metadata(length etc..) or the entire video

controls - Set to false to hide the browsers native controls

poster - placeholder image to be displayed while the video loads.

Enough of the boring stuff, lets see some JS!

First, select your video

var vid = document.querySelector('#myVid'); 

Then use the methods to control it: play, pause, skip ahead, play, volume up.

vid.play(); vid.pause(); vid.currentTime+=10; vid.volume+=0.1

You can control the playback rate, volume and current time which is enough to make your own video player UI

You also have access to all the data/properties of the video element.

vid.duration; vid.currentTime;

Events

Like any element, you can also subscribe to events. Thre are a number that are specific of HTML5 video/audio

These are handy when you are building an application that works with video

vid.addEventListener('seeking', function() {
	console.log("a scrub is a guy who thinks he's fly");
}); 
vid.addEventListener('seeked', function(e) {
	ga(e.type,this.currentTime); // send seek data to analytics
}); 

An awesome reference to all methods, events and properties

http://www.w3.org/2010/05/video/mediaevents.html

Creating a Custom Player

So, given that we have access to all the video's meta data, we can build an entire player with HTML and CSS and then use javascript to update the players scrub bar, time, volume and so on.

A few good players:

VideoJS - videojs.com/

JWPlayer - longtailvideo.com/players

MediaElement.js

All of the above have great flash fallbacks for when HTML5 video is not supported

Add a dash of CSS3

Because HTML5 Video is just another HTML element, we can use CSS to transform and style our player - with or without our HTML controls

HTML5 Video Manipulation!

HTML5 video plays nice with Canvas.

Canvas is another part of HTML5 that enables pixel based drawing.

Its kind of like MS paint, but for the browser.

We can take each frame from our video element and dump them into a canvas element!

Super Basic Example



Video Element

Canvas Element

On a set interval or requested animation frame we take a frame from the video and use the canvas's ctx drawImage() method to write part of all of the frame.

Cool, cool cool cool.

But what can you do with that?

Slice and Dice

While the video is playing, we can cut it into tiles and then spit them into a canvas at a random order.

Filters and Pixel Manipulation

Canvas has a method called getImageData() which returns an array of every single pixel available.

Better yet, we have access to the Red, Green, Blue and Alpha Channels of every pixel.

We can play aroud and change these values to whatever we want and then stick them back into the canvas.

Some Code

Lets step through a basic example. First setup your HTML.

<video id="example4" width="700" height="350">
      <source id="mp4" src="localVid.mp4" type="video/mp4">
</video>
<canvas id="example4Canvas" ></canvas> 

Now with JS, grab hold of your elements and get the canvas's context.

var example4 = document.querySelector('#example4'),
	canvas4 = document.querySelector("#example4Canvas"),
	ctx4 = canvas4.getContext("2d"),
	example4Pixels;

We also created a variable called 'example4Pixels' to hold our pixels once we nab them from the canvas.

Finally, a little trick to make your canvas and video elements the same size

(note: Different from CSS width/height)

canvas4.height  = example4.height;
canvas4.width   = example4.width;

Now that we are set up, we have 4 steps that will repeat every 50ms

or at whatever framerate your choose.

  1. Take the frame from the video and write it to canvas
  2. Grab the pixels from the canvas and store in our example4Pixels variable
  3. Loop over each pixel and adjust your RGBA channels accordingly
  4. Spit those pixels back into the canvas

Note: For simplicity, I'm using a single canvas but its often a good idea to have two: One for dumping the pixels into and one for dumping the manipulated pixels.

1. Take the frame from the video and write it to canvas
ctx4.drawImage(example4, 0, 0, example4.width, example4.height);

The above code grabs the current frame from our 'example4' video and writes to canvas.


2. Grab the pixels from the canvas and store in our example4Pixels variable
example4Pixels = ctx4.getImageData(0,0,canvas4.width,canvas4.height);

ProTip: Don't console.log() this variable as its HUGE and will crash your browser.

3. Loop over each pixel and adjust your RGBA channels accordingly

This is where the fun parts comes in. Every 4 numbers represent 1 pixel.

100		235		54		0.35 	34		245		199		1
RED 	GREEN 	BLUE 	ALPHA 	RED 	GREEN 	BLUE 	ALPHA	
// Loop over RGBA
for (i = 0; i < example4Pixels.data.length; i=i+4) {

example4Pixels.data[i + 0] = example4Pixels.data[i + 0] + 100 ; // RED
example4Pixels.data[i + 1] = example4Pixels.data[i + 1] - 50; // GREEN
example4Pixels.data[i + 2] = example4Pixels.data[i + 2] * 0.5; // Blue

}	

In this example, I upped our RED by 50, decreased GREEN by 50 and multiplied BLUe by 0.5. Alpha is left at 1.

The end result is a red tint on the the video.

Web RTC

( or HTML5 Video & Audio Hardware Access! )

( SO COOL )

WebRTC? Real Time Communication

There is something new coming down the pipe called WebRTC which allows browsers to communicate directly.

Still heavily under development but we are able to create something called a PeerConnection which allows for a 1:1 channel between two peers.

We can stream audio, video and eventually data/files over this connection without relying on a server in the middle.

We would only need websockets/xmlHTTPRequest to do the initial handshake

The real benefit here is that we will be seeing HTML5 Chatroulette clones!

Say Hello to getUserMedia()

PeerConnections aside, this new spec means that browsers are currently implementing native webcam and microphone access!

Thats right, no flash required!

getUserMedia is currently supported in Opera, Opera Mobile 12 and Google Chrome Canary (or dev channel)

Cool, demo it!


Start Stop Hipster Blur Green HTML5 Glasses

A quick word on performance

Iterating on a large canvas takes more processing power

When doing analysis, if you can, its often a good ideal to run the process on a smaller version of the canvas.

I used to get 1-2 FPS on face detection.

Not even web workers would make it faster!

Simply running the face detection script on a 100px x 100px canvas improved performance by at least 10 fold

Large Canvas === slow === better accuracy

Smaller Canvas === fast === less accurate

Larger Canvas Demo

HTML5 Video + Canvas in the real world

Popcorn JS

Rick's DMV

Platoon

Shim for getUserMedia()

Popcorn.js

Built by Bocoup / Mozilla

Bobby Richter had an awesome talk at FITC #SpotlightHTML5

Allows you to create interactive HTMl5 web pages that sync up with a video/audio element

Think cuepoints on steroids

Huge Library of plugins to work with everything from twitter to Google Maps Street View

DMV

Build by the super awesome Rick Waldron

Uses WebRTC, Node.js and web sockets to create an online photobooth and save images to the server

100% JavaScript!

Source on Github and demo at http://dmv.nodejitsu.com/

getUserMedia() Shim

So all this stuff is great, but its going to be forever until we can use it, right?!

Nope! We can use Shims and Polyfills to recreate the functionality for older browsers

Addy Osmani (past and future FITC speaker) has written a shim that provides a flash fallback

So much cool, so little time

Check out a ton more video experiments at Chrome Experiments

Cool Stuff EH?
The future is awesome
Questions?

@wesbos on Twitter