Become a Master
This course and others like it are available as part of our Frontend Masters video subscription.
HTML5 Media: Audio, Video & WebRTC
Mark Boas
Creator of jPlayer
HTML5 gives us power beyond what we've had before through native APIs in the browser delivering rich media, realtime effects and communication. In this course, you'll get hands-on examples of building your own media players on top of HTML5 APIs. You'll dive into camera support with getUserMedia, WebRTC for realtime communication, video effects with canvas, and get introduced to the Web Audio API for low-level audio support.
Published: April 8, 2014
Table of Contents
Introduction
Audio and Video Basics
Support
Before any HTML5 audio or video can be used, you need support. Mark summarizes the current browser support matrices for both audio and video.Containers and Implementation
There are a few different types of audio and video containers. These containers hold different codec that are supported in browsers. Mark demonstrates how to add audio and video to a webpage and specify different media containers.HTML5 Video with Fallback
Mark shows how to implement a simple HTML5 video player that includes a Flash-based fallback for browsers that don't support HTML5 media. He also discusses the different HTML5 media attributes available.Assignment 1 - Build a Player
Embed video and audio into a webpage with maximum cross-browser compatibility and a simple fallback. Create the minimal possible markup for an HTML5 web page that contains both audio and video.Assignment Media Files
The audio and video files Mark uses in this assignment were developed years ago. Mark goes off topic for a couple minutes to show some of his first experiments with web audio and video.Mark's Projects
Mark takes a moment to discuss a few of the media projects he works on. This includes jPlayer and Hyperaudio.Assignment 1 - Solution
Mark walks through the solution for Assignment 1: Building a Player - http://jsbin.com/AdiFeWo/4/
Tools and Configuration
Tools and Services
There are a lot of tools that will encode audio and video into the correct formats required by HTML5. Some tools are available for download. Others are hosted solutions.Assignment 2 - Configure and Encode Media
In this optional assignment, you should configure your server environment so support the HTML5 media types and encode some of your own media files.
Introduction to the Media API
play(), pause(), setMedia() and canPlayType()
Audio and video elements can be created dynamically and controlled completely with JavaScript. The media API even allows developers to determine the type of media supported.Audio and Video API Resources
Mark references a few articles he wrote on html5doctor.com about the Audio API. He also walks through a couple other resources for JavaScript media frameworks. - http://html5doctor.com/html5-audio-the-state-of-play/ - http://praegnanz.de/html5video/Creating your first Audio Player
Marks walks through the process of making your first audio player. He adds the base HTML as well as some controls and some JavaScript.Assignment 3 (with solution) - Create a Custom Video and Audio Player
The third assignment is to create a custom video and audio player with play and pause button. You cannot use the "controls" attribute for the play and pause buttons. - http://jsbin.com/oHAdude/2/
Buffering, Seeking, Preloading, and the User Experience
Preloading and Buffering
Preloading requests the media be downloaded (but not played) as soon as the page is loaded. It can be configured using the "preload" attribute. The buffered attribute will return the time ranges of the media that have been completely downloaded.Seeking, Seekable and Displaying Time
Seekable returns a TimeRange object of playable time ranges. This is often preferred over the buffered attribute. Combining seekable and buffered to display progress can be beneficial from a user experience point of view. - http://jplayer.org/HTML5.Media.Event.Inspector/Anatomy of a Progress Bar
Progress bars are an integral part of media players and come in many shapes and sizes. Mark walks through the HTML and JavaScript behind a typical progress bar.Playlists
Playlists are an import user experience element in media players. They are typically marked up using ordered or unordered lists.Assignment 4 - Adding a Functional Progress Bar
Building off the previous assignment, you should add a functional progress bar that includes displaying media progress and allowing the user to navigate by clicking on the bar.Assignment 4 - Solution
Mark walks through the solutions for Assignment 4. - http://jsbin.com/AqEZAfu/1/
Subtitles and Playback Rate
WebVTT
WebVTT is a standard that can be used for subtitles, captions, descriptions, chapters, and metadata.WebVTT Capabilities and Support
WebVTT allows for the ability to position text and add speaker semantics. Defining chapters can make navigation easier and more intuitive. Mark also talks a little about browser support for WebVTT.Subtitle Resources
Mark talks about a number of resources for creating and synchronizing subtitles with HTML5 media.Playback Rate
A new feature in HTML5 audio and video is the ability to change the playback rate. This API is supported across most modern browsers but still has some gotchas.Assignment 5.5 - Add playbackRate Capability to your Player
In this assignment, integrate the playbackRate API into your media player.Assignment 5.5 - Solution
Solution for assignment 5.5 - http://jsbin.com/omAtEM/1/
Canvas and WebGL
Canvas and Video
The canvas element in HTML5 introduces a new way to draw on a webpage. The canvas is pretty powerful and can by tightly coupled with HTML5 video. Mark demonstrates Seriously.js to show the possibilities. - http://seriouslyjs.orgCreating Simple Effects
While Seriously.js can do some amazing effects, the process of creating an effect is simple. Mark shows how to convert a video to black and white in real time.More Canvas Examples
Mark shares a few more examples of using the canvas element with HTML5 video and provides some additional resources. You can also use WebGL to paint video in a 3D environment. - http://html5doctor.com/video-canvas-magic - http://gige.org/demos/videodemo/Assignment 6 - Manipulating Color
Assignment 6 asks you to manipulate the color of your video using the canvas element. Mark provides some instruction on how to get started with this assignment.Assignment 6 - Solution
Mark discusses the solution for assignment 6 and talks a little with the audience about other uses of video effects. - http://jsbin.com/ImohifA/2/
WebRTC
WebRTC Introduction and Components
WebRTC enabled real-time communication of audio, video and data from browser to browser. To demonstrate this Mark invites attendees into a shared video web application. The three API's enabling this are MediaStream, PeerConnection, and DataChannel.MediaStream API
The MediaStream API allows you direct access to devices like the microphone and camera. Audio and video are split into synchronized MediaStreamTracks.PeerConnection and DataChannel API's
Getting access to camera and microphone information is nice, but you need the PeerConnect API to tie those streams together with another client. The DataChannel API makes it possible to stream any data you want. It handles issues like prioritization, security and congestion control. - http://webcamtoy.comWebRTC Support and Resources
Mark overviews the support matrix for WebRTC and gives a number of additional resources as well as a simple code example.Assignment 7 (and solution) - WebRTC
The 7th assignment is to capture the stream from your cameral and microphone and display them in real time. After a little discussion, Mark demonstrates the solution. - http://jsbin.com/eQudoCA/1/
Advanced Audio
Audio and Video Limitations
While there have been many advancements with HTML5 audio and video, there are a number of limitations. These include issues with autoplay, preloading, short audio, and latency. Mark talks about some of these limitations and how to work around them.Web Audio API
Mark starts by demonstrating a few things he worked on for the BBC using the Web Audio API. He then starts describing some of the benefits of using this API.Node-based Approach
The Web Audio API is very modular. It's node-based approach allows inputs and outputs to be connected by any number of intermediary nodes.Assignment 9 (and solution) - Using the Audio API
Assignment 9 is to use the Web Audio API to load and play audio. - http://jsbin.com/umldeHE/1/Web Audio API Examples
Mark ends this topic showing a number of examples that use the Web Audio API. These include audio examples like cross-fading or visual examples like equalizers.
Wrap-up
Creative Examples
Mark showcases a few creative examples that use many of the HTML5 media features including ways the BBC has recreated sounds from the 1960's by simulating radiophonic equipment. - http://webaudioapi.com/samples - http://acko.net/files/three-audio - http://labs.dinamoe.com/plinkPopcornJS
Popcorn.js is a JavaScript library for integrating the web and video production. Mark discusses how to use Popcorn.js and demonstrates some of the capabilities of the Sketchpad application. - http://popcornjs.comAudience Question
Mark answers an audience question about how he synchronizes and stores the transcripts that accompany the video/audio files.Wrap-up
Mark ends the course with a couple more questions from the audience regarding his Hyperaudio application.