Chapter 4

Working with Audio and Video

In this chapter, we’ll explore what you can do with two important HTML5 elements—audio and video—and we’ll show you how they can be used to create compelling applications. The audio and video elements add new media options to HTML5 applications that allow you to use audio and video without plugins while providing a common, integrated, and scriptable API.

First, we’ll discuss audio and video container files and codecs, and why we ended up with the codecs supported today. We’ll go on to describe lack of common codec support—the most important drawback for using the media elements—and we’ll discuss how we hope that this won’t be such a big issue in the future. We’ll also show you a mechanism for switching to the most appropriate type of content for the browser to display.

Next, we’ll show you how you can use control audio and video programmatically using the APIs and finally we’ll explore the use of the audio and video in your applications.

Overview of Audio and Video

In the following sections, we’ll discuss some of the key concepts related to Audio and video: containers and codecs.

Video Containers

An audio or audiocontainersvideocontainersvideo file is really just a container file, similar to a ZIP archive file that contains a number of files. Figure 4-1 shows how a video file (a video container) contains audio tracks, video tracks, and additional metadata. The audio and video tracks are combined at runtime to play the video. The metadata contains information about the video such as cover art, title and subtitle, captioning information, and so on.

Figure 4-1. Overview of the video container

Some of the popular video containeraudiocontainersvideocontainers formats include the following:

  • Audio Video Interleave (.avi)
  • Flash Video (.flv)
  • MPEG 4 (.mp4)
  • Matroska (.mkv)
  • Ogg (.ogv)

Audio and Video Codecs

Audio and video coders/decoders (codecs) areaudiocodecsvideocodecs algorithms used to encode and decode a particular audio or video stream so that they can be played back. Raw media files are enormous, so without encoding, a video or audio clip would consist of tremendous amounts of data that could be too large to transmit across the Internet in a reasonable amount of time. Without a decoder, the recipient would not be able to reconstitute the original media source from the encoded form. A codec is able to understand a specific container format and decodes the audio and video tracks that it contains.

Some example audio codecs are the following:

  • AAC
  • MPEG-3
  • Ogg Vorbis

Example video codecs are audiocodecsvideocodecsthe following:

  • H.264
  • VP8
  • Ogg Theora

The Codec Wars and the Tentative Truce

Some of the codecs are patent-encumbered, while others are freely available. For example, the Vorbis audio codecVorbis audio codec and the Theora video codecTheora video codec are freely available, while the use of the MPEG-4 and H.264 codecs are subject to license fees.

Originally, the HTML5 specification was going to require that certain codecs were supported. However, some vendors did not wish to include Ogg Theora as it was not part of their existing hardware and software stacks. Apple's iPhone, for example, includes hardware accelerated decoding for h264 video but not Theora. Free systems, on the other hand, cannot include proprietary for-pay codecs without hurting downstream distribution. On top of that, the performance that certain proprietary codecs provide is a factor in the browser uptake of free codecs. This situation has led to a stalemate; there does not appear to be a single codec that all browser vendors are willing to implement.

For now, the codec requirement has been dropped from the specification. However, this decision may be revisited in the future. For now, understand the current browser support and understand that you may need to re-encode your media for different environments. (You should probably be doing this already.)

We do expect that support for different codecs will increase and converge over time, making the choice of common media types easy and ubiquitous. It is also possible that one codec will grow to be the de facto standard codec for the Web. Additionally, the media tags have a built in mechanism for switching to the most appropriate type of content for the browser toaudiocodecsvideocodecs display to make supporting different environments easy.

Audio and Video Restrictions

There are a few thingsaudiounsupportedvideounsupported that are not supported in the Audio and video specification:

  • Streaming audio and video. Thatvideostreamingaudiostreaming media streamingis, there is currently no standard for bitrate switching in HTML5 video; only full media files are supported by current implementations. However, there are aspects of the spec that are designed to support streaming media in the future once the formats are supported.
  • Media is restricted by HTTP cross-origin resource sharingcross-origin resource sharing. See Chapter 6 DOI::10.1007/978-1-4302-3865-2_6 for more information about cross-origin resource sharing (CORS).
  • Full-screen video is not full-screen videovideofull-screenscriptable because it could be considered a security violation to let a scriptable element take over the full screen. However, browsers have the option of letting users choose to view videos in full screen through additional controls.

Browser Support for Audio and Video

Due to the fractured codec support, simply knowing which browsers support the new audioaudio elementbrowser support forvideo elementbrowser support for and video elements is not enough; you also need to know which codecs are supported. Table 4-1 shows which browsers support which codecs at the time of this writing.

Table 4-1. Audio and Video Codec and Container Support
Browser Codec and Container Support
Chrome Ogg (Theora and Vorbis)
WebM (VP8 and Vorbis)
MPEG 4 (H.264 and AAC)
Firefox Ogg (Theora and Vorbis)
WebM (VP8 and Vorbis)
Internet Explorer MPEG 4 (H.264 and AAC)
Opera Ogg (Theora and Vorbis)
WebM (VP8 and Vorbis)
Safari MPEG 4 (H.264 and AAC)

Note also that Google announced it will drop support for the MP4 format, but that has not happened yet. Also, there is a plugin that can be used to play WebM in Internet Explorer 9. It is always good idea to first test whether audio and video are supported. The section “Checking for Browser Support” later in this chapter will show you how you can programmatically check for browser support.

Using the Audio and Video API

In this section, we’ll exploreaudioAudio APIvideoVideo API the use of the audio and video in your applications. There are two main benefits to using the new media tags over previous video-embedding techniques—usually videos are embedded using the Flash, QuickTime, or Windows Media plugins—that aim to make life easier for users and developers:

Of course, there is one primary drawback to using the media tags: lack of common codec support, as discussed in the earlier sections of this chapter. However, we expect that support for codecs will increase and converge over time, making the choice of common media types easy and ubiquitous. Plus, the media tags have a built-in mechanism for switching to the most appropriate type of content for the browser to display, as you will soon see.

Checking for Browser Support

The easiest way to check for support of theaudio elementbrowser support forvideo elementbrowser support for audioAudio APIvideoVideo API video and audio tags is to dynamically create one or both with scripting and check for the existence of a function:

var hasVideo = !!(document.createElement('video').canPlayType);

This simple code line will dynamically create a video element and check for the existence of the canPlayType() function. By using the !! operator!! operator, the result is converted to a Boolean value, which indicates whether or not a video object could be created.

However, if video or audio support is not present, you may choose to use an enabling script that introduces media script tags into older browsers, allowing the same scriptability but using technologies such as Flash for playback.

Alternatively, you can choose to include alternate content between your audio or video tags, and the alternate content will display in place of the unsupported tag. This alternate content can be used for a Flash plugin to display the same video if the browser doesn’t support the HTML5 tags. If you merely wish to display a text message for nonsupporting browsers, it is quite easy to add content inside the video or audio elements as shown in Listing 4-1.

Listing 4-1. Simple Video Element
<video src="video.webm" controls>
  Your browser does not support HTML5 video.
</video>

However, if you choose to use an alternative method to render video for browsers without HTML5 media support, you can use the same element content section to provide a reference to an external plugin displaying the same media as shown in Listing 4-2.

Listing 4-2. Video Element with Flash Fallback
<video src="video.webm" controls>
  <object data="videoplayer.swf" type="application/x-shockwave-flash">
    <param name="movie" value="video.swf"/>
  </object>
  Your browser does not support HTML5 video.
</video>

By embedding an object element that displays a Flash videoFlash videovideoFlash inside the video element, the HTML5 video will be preferred if it is available, and the Flash video will be used as a fallback. Unfortunately, this requires multiple versions of the video to be served upaudio elementbrowser support forvideo elementbrowser support for until HTML5 support is ubiquitous.

Accessibility

audioAudio APIvideoVideo APIMaking your web applications accessible to everyone isn’t just the right thing to do; it’s good business, and, in some cases, it’s the law! Users with limited vision or hearing should be presented with alternative content that meets their needs. Keep in mind that the alternative content located between the video and audio elements is only displayed if the browser does not support those elements at all and, therefore, is not suitable for accessible displays where the browser may support HTML5 media but the user may not.

The emerging standard for video accessibilityvideoaccessibility is Web Video Text Tracks (WebVTT)Web Video Text Tracks (WebVTT), formerly known as Web SubRip Text (WebSRT) format. At the time of this writing, it is only just starting to appear in some early builds of browsers. WebVTT uses a simple text file (*.vtt) that starts with the word WEBVTT on the first line. The vtt file must be served up with the mime type text/vtt. Listing 4-3 shows the contents of an example vtt file.

Listing 4-3. WebVTT File
WEBVTT
  
1
00:00:01,000 --> 00:00:03,000
What do you think about HTML5 Video and WebVTT?…

2
00:00:04,000 --> 00:00:08,000
I think it’s great. I can’t wait for all the browsers to support it!

To use the vtt file in your video element, add the track element pointing to the vtt file as shown in the following example:

<video src="video.webm" controls>
  <track label="English" kind="subtitles" srclang="en" src="subtitles_en.vtt" default>
  Your browser does not support HTML5 video.
</video>

You can add multiple track elements. Listing 4-4 shows how you can support English and Dutch subtitles using track elements pointing to a vtt file.

Listing 4-4. Using WebVTT Tracks in a Video Element
<video src="video.ogg" controls>
  <track label="English" kind="subtitles" srclang="en" src="subtitles_en.vtt">
  <track label="Dutch" kind="subtitles" srclang="nl" src="subtitles_nl.vtt">
  Your browser does not support HTML5 video.
</video>

videoaccessibilityThe WebVTT standard Web Video Text Tracks (WebVTT)supports more than just subtitles. It also allows for captions and cue settings (instructions for how text is rendered). The full WebVTT syntax is beyond the scope of this book. See the WHATWG specification at www.whatwg.org/specs/web-apps/current-work/webvtt.html for more details.

Understanding Media Elements

media elements See audio element media elements See video element Due to a wise designaudio elementunderstandingvideo elementunderstanding decision, there is audioAudio APIvideoVideo APImuch commonality between the audio and video elements in HTML5. Both audio and video support many of the same operations—play, pause, mute/unmute, load, and so on—and therefore, the common behavior was separated out into the media element section of the specification. Let’s start examining the media elements by observing what they have in common.

The Basics: Declaring Your Media Element

For the sake of example, audio elementdeclaringvideo elementdeclaringwe will use an audio tag to try out the common behaviors of HTML5 media. The examples in this section will be very media-heavy (surprise!), and they are included in the code/av folder of the support files that come with this book.

For the very simplest example (the example file audio.html), let’s create a page that shows an audio player for a soothing, satisfying, and very public domain audio clip: Johann Sebastian Bach’s “Air” (shown in Listing 4-5).

Listing 4-5. HTML Page with an Audio Element
<!DOCTYPE html>
<html>
  <title> HTML5 Audio </title>
  <audio controls src="johann_sebastian_bach_air.ogg">
           An audio clip from Johann Sebastian Bach.
  </audio>
</html>

This clip assumes that the HTML document and the audio file—in this case, johann_sebastian_bach_air.ogg —are served from the same directory. As shown in Figure 4-2, viewing this in a browser supporting the audio tag will show a simple control and play bar representingaudio elementdeclaringvideo elementdeclaring the audio to play. When the user clicks the play button, the audio track starts as expected.

Figure 4-2. Simple audio controls audio controls

The controls attributecontrols attribute tells the browser to display common user controls for starting, stopping, and seeking in the media clip, as well as volume control. Leaving out the controls attribute hides them, and leaves the clip with no way for the user to start playing.

The content between the audio tags is audio elementunderstandingvideo elementunderstandingtext representation of what the browser will display if it does not support the media tag. This is what you and your users will see if they are running an older browser. It also gives the opportunity to include an alternate renderer for the media, such as a Flash player plugin or a direct link to the media file.

Using the Source

Finally, wemedia source files comeaudio elementsourcevideo elementsourcesrc attribute to the most importantaudioAudio APIvideoVideo API attribute: src. In the simplest setup, a single src attribute points to the file containing the media clip. But what if the browser in question does not support that container or codec (in this case, Ogg and Vorbis)? Then, an alternate declaration is shown in Listing 4-6; it includes multiple sources from which the browser can choose (see the example file audio_multisource.html).

Listing 4-6. An Audio Element with Multiple Source Elements
<audio controls>
  <source src="johann_sebastian_bach_air.ogg">
  <source src="johann_sebastian_bach_air.mp3">
  An audio clip from Johann Sebastian Bach.
</audio>

In this case, we include two new source elements instead of the src attribute on the audio tag. This allows the browser to choose which source best suits the playback capabilities it has and use the best fit as the actual media clip. Sources are processed in order, so a browser that can play multiple listed source types will use the first one it encounters.

Running this clip in a supported browser may not change what you see. But if a browser supports the MP3 format and not the Ogg Vorbis format, the media playback will now be supported. The beauty of this declaration model is that as you write code to interact with the media file, it doesn’t matter to you which container or codec was actually used. The browser provides a unified interface for you to manipulate the media, no matter which source was matched for playback.

However, there is another way to give the browser hints about which media source to use. Recall that a container for media can support many different codec types, and you will understand that a browser may be misled into which types it does or does not support based on the extension of the declared source file. If you specify a type attribute that does not match your source, the browser may refuse to play the media. It may be wise to include the type only if you know it with certainty. Otherwise, it is better to omit this attribute and let the browser detect the encoding as shown in Listing 4-7 (in the example file audio_type.html). Also note that the WebM format allows only one audio codec and one video codec. That means the .webm extension or the video/webm content-type tells you everything you need to know about the file. If a browser can play .webm, it should be able to play any valid .webm file.

Listing 4-7. Including Type and Codec Information in an Audio Element
<audio controls>
  <source src="johann_sebastian_bach_air.ogg" type="audio/ogg; codecs=vorbis">
  <source src="johann_sebastian_bach_air.mp3" type="audio/mpeg">
  An audio clip from Johann Sebastian Bach.
</audio>

As you can see, the type attribute can declare both the container and codec type. The values here represent Ogg Vorbis and MP3, respectively. Theaudio elementsourcevideo elementsourcesrc attribute full list is governed by RFC 4281, a document maintained by the Internet Engineering Task Force (IETF), but audio elementunderstandingvideo elementunderstandingsomeaudioAudio APIvideoVideo API common combinations are listed in Table 4-2.

Table 4-2. Media Types and Attribute Values
Type Attribute Value
Theora video and Vorbis audio in an Ogg container type='video/ogg; codecs="theora, vorbis"'
Vorbis audio in an Ogg container type='audio/ogg; codecs=vorbis'
WebM video in a Matroska container type='video/webm; codecs="vp8, vorbis"'
Simple baseline H.264 video and low complexity AAC audio in an MP4 container type='video/mp4; codecs="avc1.42E01E, mp4a.40.2"'
MPEG-4 visual simple profile and low media source filescomplexity AAC audio in an MP4 container type='video/mp4; codecs="mp4v.20.8, mp4a.40.2"'

Taking Control

You’ve already seen that the media controlsaudio elementplayback controlsvideo elementplayback controlsdefault playback controls can be displayed by using the controls attributecontrols attribute in the video or audio tag. As you might expect, leaving out this attribute will not display controls when the media is displayed, but it will also not show anything at all in the case of audio files, as the only visual representation of an audio element is its controls. (A video without controls still displays the video content.) Leaving out the controls attribute should not display any content that affects the normal rendering of the page. One way to cause the media to play is to set another attribute inautoplay attribute the tag: autoplay (see Listing 4-8 and the example file audio_no_control.html).

Listing 4-8. Using the Autoplay Attribute
<audio autoplay>
  <source src="johann_sebastian_bach_air.ogg" type="audio/ogg; codecs=vorbis">
  <source src="johann_sebastian_bach_air.mp3" type="audio/mpeg">
  An audio clip from Johann Sebastian Bach.
</audio>

By including the autoplay attribute, the media file will play as soon as it is loaded, without any user interaction. (Note that autoplay is not supported everywhere. For example, it is disabled on iOS.) However, most users will find this highly annoying, so use autoplay with caution. Playing audio without prompting may be intended to create an atmospheric effect or, worse, to force an advertisement on the user. But it also interferes with other audio playing on the user’s machine, and can be quite detrimental to users who rely on audible screen readers to navigate web content. Note also that some devices, like the iPad, prevent autoplay and even automatically playing a media file (triggered by a page load event, for example).

If the built-in controls do not suit theaudioAudio APIvideoVideo API layout of your user interface, or if you need to control the media element using calculations or behaviors that are not exposed in the default controls, there are manyaudio elementunderstandingvideo elementunderstanding built-in JavaScript functions and attributes to help you, too. Table 4-3 lists some of the most common functions.

Table 4-3. Common Control Functions
Function Behavior
load() Loads the media file and prepares it for playback. Normally does not need to be called unless the element itself is dynamically created. Useful for loading in advance of actual playback.
play() Loads (if necessary) and plays the media file. Plays from the beginning unless the media is already paused at another position.
pause() Pauses playback if currently active.
canPlayType(type) Tests to see whether the video element can play a hypothetical file of the given MIME type.

The canPlayType(type) method has a non-obvious use case: by passing in a MIME type of an arbitrary video clip to a dynamically created video element, you can use a simple script to determine whether the current browser supports that type. For example, the following code provides a quick way to determine whether the current browser can support playing videos with MIME type media controlsaudio elementplayback controlsvideo elementplayback controlsof fooType without displaying any visible content in the browser window:

var supportsFooVideo = !!(document.createElement('video').canPlayType(‘fooType’));
Note that this function returns the very non-binary “null,” “maybe,” or “probably,” with probably being the best possible scenario.

Table 4-4 shows a few of theread-only media attributes read-only attributes on media elements.

Table 4-4. Read-only Media Attributes
Read-only Attribute Value
duration The duration of the full media clip, in seconds. If the full duration is not known, NaN is returned.
paused Returns true if the media clip is currently paused. Defaults to true if the clip has not started playing.
ended Returns true if the media clip has finished playing.
startTime Returns the earliest possible value for playback start time. This will usually be 0.0 unless the media clip is streamed and earlier content has left the buffer.
error An error code, if an error has occurred.
currentSrc Returns the string representing the file that is currently being displayed or loaded. This will match the source element selected by the browser.

Table 4-5 shows some of the attributes on the media elements that allow scripts to modify them and affect the playback directly. As such, scriptable media attributesthey audio elementunderstandingvideo elementunderstandingbehaveaudioAudio APIvideoVideo API similar to functions.

Table 4-5. Scriptable Attribute Values
Attribute Value
autoplay Sets the media clip to play upon creation or query whether it is set to autoplay.
loop Returns true if the clip will restart upon ending or sets the clip to loop (or not loop).
currentTime Returns the current time in seconds that has elapsed since the beginning of the playback. Sets currentTime to seek to a specific position in the clip playback.
controls Shows or hides the user controls, or queries whether they are currently visible.
volume Sets the audio volume to a relative value between 0.0 and 1.0, or queries the value of the same.
muted Mutes or unmutes the audio, or determines the current mute state.
autobuffer Tells the player whether or not to attempt to load the media file before playback is initiated. If the media is set for auto-playback, this attribute is ignored.

Between the various functions and attributes, it is possible for a developer to create any media playback user interface and use it to controlmedia controlsaudio elementplayback controlsvideo elementplayback controls audio elementunderstandingvideo elementunderstandingany audio or video clip that is supported by the browser.

Working with Audio

If you understand theaudioworking with shared attributes for both audio and video media elements, you’ve basically seen all that the audio tag has to offer. So let’s look at a simple example that shows control scripting in action.

Audio Activation

If your user interfaceaudioactivation needs to play audioAudio APIvideoVideo APIan audio clip for users, but you don’t want to affect the display with a playback timeline or controls, you can create an invisible audio element—one with the controls attribute unset or set to false —and present your own controls for audio playback. Consider the simple code in Listing 4-9, also available in the sample code file audioCue.html.

Listing 4-9. Adding Your Own Play Button to Control Audio
<!DOCTYPE html>
<html>
  <link rel="stylesheet" href="styles.css">
  <title> Audio cue</title>
  
  <audio id="clickSound">
    <source src="johann_sebastian_bach_air.ogg">
    <source src="johann_sebastian_bach_air.mp3">
  </audio>
  
  <button id="toggle" onclick="toggleSound()"> Play</button>
  
  <script type="text/javascript">
    function toggleSound() {
      var music = document.getElementById("clickSound");
      var toggle = document.getElementById("toggle");
      if (music.paused) {
        music.play();
        toggle.innerHTML = "Pause";
      }
      else {
        music.pause();
        toggle.innerHTML = "Play";
      }
    }
  </script>
</html>

Once again, we are using an audio element to play our favorite Bach tune. However, in this example we hide user controls and don’t set the clip toaudioactivation autoplay on load. Instead, we have created a toggle button to control the audio playback with script:

<button id="toggle" onclick="toggleSound()"> Play</button>

Our simple button is initialized to inform the user that clicking it will start playback. And each time the button is pressed, the toggleSound() functiontoggleSound() function is triggered. Inside the toggleSound() function, we first gain access to the audio and button elements in the DOM:

if (music.paused) {
  music.play();
  toggle.innerHTML = "Pause";
}

By accessing the paused audioworking withattribute on the audio element, we can check to see whether the user has already paused playback. The attribute defaults to true if no playback is initiated, so this condition will be met on the first click. In that case, we call the play() functionplay() function on the clip and change the text of the button to indicate that the next click will pause the clip:

else {
  music.pause();
  toggle.innerHTML = "Play";
}

Conversely, if the music clip is not paused (if it is playing), we will actively pause() it and change the button text to indicate that the next click will restart play. Seems simple, doesn’t it? That’s the point of the media elements in HTML5: to create simple display andaudioAudio APIvideoVideo API control across media types where once a myriad of plugins existed. audioactivationSimplicityaudioworking with is its own reward.

Working with Video

Enough with video elementworking withsimplicity. Let’s try something more complicated. The HTML5 video element is very similar to the audio element, but with a few extra attributes thrown in. Table 4-6 shows some of these attributes.

Table 4-6. Additional Video Attributes
Attribute Value
poster The URL of an image file used to represent the video content before it has loaded. Think “movie poster.” This attribute can be read or altered to change the poster.
width, height Read or set the visual display size. This may cause centering, letterboxing, or pillaring if the set width does not match the size of the video itself.
videoWidth, videoHeight Return the intrinsic or natural width and height of the video. They cannot be set.

The video element has one other key feature that is not applicable to the audio element: it can be provided to many functions of the HTML5 Canvas (see Chapter 2 DOI::10.1007/978-1-4302-3865-2_2 ).

Creating a Video Timeline Browser

In this more complexvideo elementcreating video timeline browser example, we’ll show how a video element can have its frames grabbed and displayed in a dynamic canvas. To demonstrate this capability, we’ll build a simple video timeline viewer. While the video plays, periodic image frames from its display will be drawn onto a nearby canvas. When the user clicks any frame displayed in the canvas, we’ll jump the playback of the video to that precise moment in time. With only a few lines of code, we can create a timeline browser that users can use to jump around inside a lengthy video.

Our sample video clip is the tempting audioAudio APIvideoVideo APIconcession advert from the mid-20th century movie theaters, so let’s all go to the lobby to get ourselves a treat (see Figure 4-3).

Figure 4-3. The video timeline application

Adding the Video and the Canvas Element

We start with a simple declaration video elementaddingcanvas elementaddingvideo elementcreating video timeline browservideo elementworking withto display our video clip:

<video id="movies" autoplay oncanplay="startVideo()" onended="stopTimeline()" autobuffer="true" width="400px" height="300px">
  <source src="Intermission-Walk-in.ogv">
  <source src="Intermission-Walk-in_512kb.mp4">
</video>

As most of this markup will look familiar to you from the audio example, let’s focus on the differences. Obviously, the <audio> element has been replaced with <video>, and the <source> elements point to the Ogg and MPEG movies that will be selected by the browser.

The video has, in this case, been declared to have autoplay so that it starts as soon as the page loads. Two additional event handler functions have been registered. When the video is loaded and ready to begin play, the oncanplay function will trigger and start our routine. Similarly, when the video ends, the onended callback will allow us to stop creating video frames.

Next, we’ll add a canvas called timeline into which we will draw frames of our video at regular intervals.

<canvas id="timeline" width="400px" height="300px">

Adding Variables

In the next section of ourvideo elementadding variables demo, we begin audioAudio APIvideoVideo APIour script by declaring some values that will let us easily tweak the demo and make the code more readable:

// # of milliseconds between timeline frame updates
var updateInterval = 5000;
// size of the timeline frames
var frameWidth = 100;
var frameHeight = 75;

// number of timeline frames
var frameRows = 4;
var frameColumns = 4;
var frameGrid = frameRows * frameColumns;

updateIntervalupdateInterval variable controls how often we will capture frames of the video—in this case, every five seconds. The frameWidthframeWidth variable and frameHeightframeHeight variable set how large the small timeline video frames will be when displayed in the canvas. Similarly, the frameRows, frameColumns, and frameGrid determine how many frames we will display in our timeline:

// current frame
var frameCount = 0;

// to cancel the timer at end of play
var intervalId;

var videoStarted = false;

To keep track of which frame video elementworking withof video we are viewing, a frameCount is made accessible to all demo functions. (For the sake of our demo, a frame is one of our video samples taken every five seconds.) The intervalId is used to stop the timer we will use to grab frames. And finally, we add a videoStarted flag to make sure that we onlyvideo elementadding variables create one timer per demo.

Adding the updateFrame Function

The core function ofupdateFrame functionvideo elementupdateFrame function, adding our demo—where tvideo elementcreating video timeline browserhe video meets the canvas—is where we grab a video frame and draw it onto our canvas:

// paint a representation of the video frame into our canvas
function updateFrame() {
  var video = document.getElementById("movies");
  var timeline = document.getElementById("timeline");
  
  var ctx = timeline.getContext("2d");
  
  // calculate out the current position based on frame
  // count, then draw the image there using the video
  // as a source
  var framePosition = frameCount % frameGrid;
  var frameX = (framePosition % frameColumns) * frameWidth;
  var frameY = (Math.floor(framePosition / frameRows)) * frameHeight;
  ctx.drawImage(video, 0, 0, 400, 300, frameX, frameY, frameWidth, frameHeight);
  
  frameCount++;
}

As you’ve seen in Chapter 2 DOI::10.1007/978-1-4302-3865-2_2 , theaudioAudio APIvideoVideo API first thing to do with any canvas is to grab a two-dimensional drawing context from it:

var ctx = timeline.getContext("2d");

Because we want to populate our canvas grid with frames from left to right, top to bottom, we need to figure out exactly which of the grid slots will be used for our frame based on the number of the frame we are capturing. Based on the width and height of each frame, we can then determine exact X and Y coordinates at which to begin our drawing:

var framePosition = frameCount % frameGrid;
var frameX = (framePosition % frameColumns) * frameWidth;
var frameY = (Math.floor(framePosition / frameRows)) * frameHeight;

Finally, we reach the key call tovideo elementworking with draw an image onto the canvas. We’ve seen the position and scaling arguments before in our canvas demos, but instead of passing an image to the drawImage routine, we here pass the video object itself:

ctx.drawImage(video, 0, 0, 400, 300, frameX, frameY, frameWidth, frameHeight);

Canvas drawing routines canupdateFrame functionvideo elementupdateFrame function, adding take video sources as images or patterns, which gives you a handy way to modify the video and redisplay it in another location.

Adding the startVideo Function

Finally, we updatestartVideo functionvideo elementstartVideo function, adding frameCount tovideo elementcreating video timeline browser reflect that we’ve taken a new snapshot for our timeline. Now, all we need is a routine to regularly update our timeline frames:

function startVideo() {
  // only set up the timer the first time the
  // video is started
  if (videoStarted)
    return;
      
    videoStarted = true;
      
    // calculate an initial frame, then create
    // additional frames on a regular timer
    updateFrame();
    intervalId = setInterval(updateFrame, updateInterval);

Recall that the startVideo() function is triggered as soon as the video has loaded enough to begin playing. First, we make sure that we are going to handle the audioAudio APIvideoVideo APIvideo start only once per page load, just in case the video is restarted:

// only set up the timer the first time the
// video is started
if (videoStarted)
  return;
videoStarted = true;

When the video starts, we will capture our first frame. Then, we will start an interval timer—a timer that repeats continuously at the specifiedvideo elementworking with update interval—which will regularly call our updateFrame() function. The end startVideo functionvideo elementstartVideo function, addingresult is that a new frame will be captured every five seconds:

// calculate an initial frame, then create
// additional frames on a regular timer
updateFrame();
intervalId = setInterval(updateFrame, updateInterval);

Handling User Input

Now all we need video elementusing input, handlingto do is handle video elementcreating video timeline browseruser clicks for the individual timeline frames:

// set up a handler to seek the video when a frame
// is clicked
var timeline = document.getElementById("timeline");
timeline.onclick = function(evt) {
  var offX = evt.layerX - timeline.offsetLeft;
  var offY = evt.layerY - timeline.offsetTop;
  
  // calculate which frame in the grid was clicked
  // from a zero-based index
  var clickedFrame = Math.floor(offY / frameHeight) * frameRows;
  clickedFrame +=Math.floor(offX / frameWidth);
  
  // find the actual frame since the video started
  var seekedFrame = (((Math.floor(frameCount / frameGrid)) *
                              frameGrid) + clickedFrame);
                              
  // if the user clicked ahead of the current frame
  // then assume it was the last round of frames
  if (clickedFrame> (frameCount % 16))
    seekedFrame -=frameGrid;
    
    // can't seek before the video
    if (seekedFrame < 0)
      return;

Things get a little moreaudioAudio APIvideoVideo API complicated here. We retrieve the timeline canvas and set a click-handling function video elementusing input, handlingon it. The handler will use the event to determine which X and Y coordinates were clicked by the user:

var timeline = document.getElementById("timeline");
timeline.onclick=function(evt) {
  var offX = evt.layerX - timeline.offsetLeft;
  var offY = evt.layerY - timeline.offsetTop;

We then use the frame dimensions to figure out which of the 16 frames was clicked by the user:

// calculate which frame in the grid was clicked
// from a zero-based index
var clickedFrame = Math.floor(offY / frameHeight) * frameRows;
clickedFrame +=Math.floor(offX / frameWidth);

The clicked frame should be only one of the most recent video frames, so determine the most recent frame that corresponds to that grid index:

// find the actual frame since the video started
var seekedFrame = (((Math.floor(frameCount / frameGrid)) *
                                               frameGrid) + clickedFrame);

If the user clicks ahead of the video elementworking withcurrent frame, jump back one complete cycle of grid frames to find the actual time:

// if the user clicked ahead of the current frame
// then assume it was the last round of frames
if (clickedFrame> (frameCount % 16))
  seekedFrame -=frameGrid;

And finally, we have to video elementcreating video timeline browsersafeguard against any case in which the user clicks a frame that would bevideo elementusing input, handling before the start of the video clip:

// can't seek before the video
if (seekedFrame < 0)
  return;

Now that we know what point in time the user wants to seek out, we can use that knowledge to change the current playback time. Although this is the key demo function, the routine itself is quite simple:

// seek the video to that frame (in seconds)
var video = document.getElementById("movies");
video.currentTime = seekedFrame * updateInterval / 1000;

// then set the frame count to our destination
frameCount = seekedFrame;

By setting the currentTime attributecurrentTime attribute on audioAudio APIvideoVideo APIour video element, we cause the video to seek to the video elementusing input, handlingspecified time and reset our current frame count to the newly chosen frame.

Adding the stopTimeline Function

All that remains for stopTimeline functionvideo elementstopTimeline function, addingour videovideo elementworking with timeline demo is to stop capturing frames when the video finishes playing. Although not required, if we don’t take this step, the demo will continue capturing frames of the finished demo, blanking out the entire timeline after a while:

// stop gathering the timeline frames
function stopTimeline() {
  clearInterval(intervalId);
}

The stopTimeline handler will be called when another of our video handlers— onended —is triggered by the completion of video playback.

Our video timeline is probably not full-featured enough to satisfy power users, but it took only a short amount of code to accomplish. Now, onvideo elementcreating video timeline browser withvideo elementworking with the show.

The Final Code

Listing 4-10 shows the complete code for the video timeline pagevideo elementtimeline code.

Listing 4-10. The Complete Video Timeline Code
<!DOCTYPE html>
<html>
  <link rel="stylesheet" href="styles.css">
  <title> Video Timeline</title>
  
  <video id="movies" autoplay oncanplay="startVideo()"
onended="stopTimeline()" autobuffer="true"
    width="400px" height="300px">
    <source src="Intermission-Walk-in.ogv">
    <source src="Intermission-Walk-in_512kb.mp4">
  </video>
  
  <canvas id="timeline" width="400px" height="300px">
  
  <script type="text/javascript">
  
    // # of milliseconds between timeline frame updates
    var updateInterval = 5000;
    
    // size of the timeline frames
    var frameWidth = 100;
    var frameHeight = 75;
    
    // number of timeline frames
    var frameRows = 4;
    var frameColumns = 4;
    var frameGrid = frameRows * frameColumns;
    
    // current frame
    var frameCount = 0;
    
    // to cancel the timer at end of play
    var intervalId;
    var videoStarted = false;
    function startVideo() {
      
      // only set up the timer the first time the
      // video is started
      if (videoStarted)
      return;
      
      videoStarted = true;
      
      // calculate an initial frame, then create
      // additional frames on a regular timer
      updateFrame();
      intervalId = setInterval(updateFrame, updateInterval);
      
      // set up a handler to seek the video when a frame
      // is clicked
      var timeline = document.getElementById("timeline");
      timeline.onclick = function(evt) {
        var offX = evt.layerX - timeline.offsetLeft;
        var offY = evt.layerY - timeline.offsetTop;
        
        // calculate which frame in the grid was clicked
        // from a zero-based index
        var clickedFrame = Math.floor(offY / frameHeight) * frameRows;
        clickedFrame +=Math.floor(offX / frameWidth);
        
        // find the actual frame since the video started
        var seekedFrame = (((Math.floor(frameCount / frameGrid)) *
                            frameGrid) + clickedFrame);
                            
        // if the user clicked ahead of the current frame
        // then assume it was the last round of frames
        if (clickedFrame> (frameCount % 16))
          seekedFrame -=frameGrid;
            
        // can't seek before the video
        if (seekedFrame < 0)
          return;
          
        // seek the video to that frame (in seconds)
        var video = document.getElementById("movies");
        video.currentTime = seekedFrame * updateInterval / 1000;
        
        // then set the frame count to our destination
        frameCount = seekedFrame;
      }
    }
    
    // paint a representation of the video frame into our canvas
    function updateFrame() {
      var video = document.getElementById("movies");
      var timeline = document.getElementById("timeline");
      
      var ctx = timeline.getContext("2d");
      
      // calculate out the current position based on frame
      // count, then draw the image there using the video
      // as a source
      var framePosition = frameCount % frameGrid;
      var frameX = (framePosition % frameColumns) * frameWidth;
      var frameY = (Math.floor(framePosition / frameRows)) * frameHeight;
      ctx.drawImage(video, 0, 0, 400, 300, frameX, frameY, frameWidth, frameHeight);
      
      frameCount++;
    }
    
    // stop gathering the timeline frames
    function stopTimeline() {
      clearInterval(intervalId);
    }
  </script>
</html>

Practical Extras

Sometimes there are techniques that don’t fit into our regular examples, but which nonetheless apply to many types of HTML5 applications. We present to you some short, but common, practical extras here.

Background Noise in a Page

Many a web site has audio elementbackground noise in a pageattempted to entertain its viewers by playing audio by default for any visitors. While we don’t condone this practice, Audio support makes it quite easy to achieve this, as shown in Listing 4-11.

Listing 4-11. Using the Loop and Autoplay Attributes
<!DOCTYPE html>
<html>
  <link rel="stylesheet" href="styles.css">
  <title> Background Music</title>
  <audio autoplay loop>
    <source src="johann_sebastian_bach_air.ogg">
    <source src="johann_sebastian_bach_air.mp3">
  </audio
  <h1> You're hooked on Bach!</h1>
</html>

As you can see, playing a looping audioAudio APIvideoVideo APIbackground sound is as easy as declaring a single audio tag with the autoplay audio elementbackground noise in a pageautoplay attributeloop attributeand loop attributes set (see Figure 4-4).

Figure 4-4. Using autoplay to play music when a page loads

Mouseover Video Playback

Another way to usemouseover video playbackvideo elementmouseover video playback simple scripting effectively with video clips is to trigger the play and pause routines, based on mouse movement over the video. This could be useful in a site that needs to display many video clips and let the user choose which ones to play. The video gallery can display short preview clips on when a user moves the mouse over them and a full video display when theaudioAudio APIvideoVideo API user clicks. It is quite easy to achieve this affect using a code sample similar to Listing 4-12 (see the example file mouseoverVideo.html).

Listing 4-12. Mouse Detection on a Video Element
<!DOCTYPE html>
<html>
  <link rel="stylesheet" href="styles.css">
  <title> Mouseover Video</title>
  <video id="movies" onmouseover ="this.play()" onmouseout="this.pause()"
        autobuffer="true"
    width="400px" height="300px">
    <source src="Intermission-Walk-in.ogv" type='video/ogg; codecs="theora, vorbis"'>
    <source src="Intermission-Walk-in_512kb.mp4" type='video/mp4; codecs="avc1.42E01E,
                 mp4a.40.2"'>
  </video>
</html>

By simply setting a few extra attributes, the preview playback can trigger when a user points at the video, as shown in Figure 4-5.

Figure 4-5. Mouseover mouseover video playback video element mouseover video playback video playback

Summary

In this chapter, we have explored what you canaudioAudio APIvideoVideo API do with the two important HTML5 elements audio and video. We have shown you how they can be used to create compelling web applications. The audio and video elements add new media options to HTML5 applications that allow you to use audio and video without plugins, while at the same time providing a common, integrated, and scriptable API.

First, we discussed the audio and video container files and codecs and why we ended up with the codecs supported today. We then showed you a mechanism for switching to the most appropriate type of content for the browser to display, and we showed you how to make video accessible using WebVTT.

Next, we showed you how you can use control audio and video programmatically using the APIs and finally we looked at how you can use of the HTML5 audio and video in your applications.

In the next chapter, we'll show how you can use Geolocation to tailor your application's output to the whereabouts of your users with a minimal amount of code.