Maximiliano Firtman
Independent Consultant
Course Description
Explore APIs you may not realize exist on the web! Get an overview of new Web APIs and their maturities, such as speech recognition, Permissions Policy and security, sensors, geolocation, OS integration (managing windows, file, and URL protocol handlers), and hardware-related capabilities. You’ll also see advanced features like face detection, augmented reality, and working with external hardware and devices. Along the way, you’ll build a “Cooking Masters” app to demonstrate the practical application of capabilities in real-life scenarios.
This course and others like it are available as part of our Frontend Masters video subscription.
Preview
CloseCourse Details
Published: July 11, 2023
Learning Paths
Topics
Learn Straight from the Experts Who Shape the Modern Web
Your Path to Senior Developer and Beyond
- 200+ In-depth courses
- 18 Learning Paths
- Industry Leading Experts
- Live Interactive Workshops
Table of Contents
Introduction
Section Duration: 5 minutes
Maximiliano Firtman reviews the topics discussed throughout the course. The code and demonstrations will cover permissions & security, sensors, speech, OS integration, and other hardware-related capabilities. The demo application and source code can be found on the course website.
Web Capabilities
Section Duration: 56 minutes
Max defines a capability and shares how they are categorized by maturity. Green APIs are available in every browser on every platform. Light-Green APIs are only supported in some browsers but are mature enough to be used in production.
Max introduces origin trials, which allows developers to provide experimental APIs to a wider range of users. Rather than requiring users to enable a flag in their browsers, developers embed a token in a header or meta tag, automatically enabling the feature in Chromium-based browsers.
Max concludes the summary of capability maturity levels by discussing the dangers of using red-labeled capabilities. These capabilities have little browser support and may not be added for months or years.
Max shares several resources for understanding how to use capabilities and researching their status or browser support.
Max shares several basic web capabilities that this course won't cover. These include PWA APIs, web authentication, web storage, and background APIs. These APIs can be found in other Frontend Masters courses.
Max describes how browsers will require and prompt the user for permission to use specific capabilities. These security measures are in place to protect users from malicious use of capabilities that can potentially exploit or expose a user or their data.
Max introduces the Permissions-Policy spec, which allows a header to be set to turn on or off a capability for a specific origin or across multiple origins. This is useful for turning off capabilities and eliminating a malicious actor's ability to execute code in a browser console.
Sensors, Geolocation, & Input Devices
Section Duration: 1 hour, 20 minutes
Max introduces sensors and explains how the APIs will vary based on a device's capabilities. Available sensors may include an accelerometer, gyroscope, magnetometer, proximity, and light sensors. Global DOM APIs still exist for accessing sensor data. The Sensor API is the new promised-based API which is still gaining browser support.
Max demonstrates how to use the new Sensor API. Each API has a constructor for creating a sensor object and events for reading and handling errors. Some APIs are behind a flag because the API is still experimental.
Max introduces the Geolocation API, which uses several metrics to determine a user's location. The metrics may include data from GPS satellites, WIFI networks, and other location-enabled devices.
Max uses the GPS API to display the latitude and longitude information. A condition first checks to see if the API is available. Functions for receiving the position value and handling errors are created, and configuration options are reviewed.
Max discusses some caveats with location accuracy on iOS and iPadOS. Users specify whether they want web applications to use precise or imprecise location data. This will apply to all web applications, and there is no API for determining which mode the user selected.
Max shares examples of APIs for device orientation, touch events, and working with virtual keyboards. Some APIs are JavaScript-based. Others, like the virtual keyboard, also provide CSS APIs for adjusting styles when the virtual keyboard is present.
Max introduces the Gamepad API. It works with both USB and Bluetooth controllers. Since the buttons and controls vary greatly across controllers, a game loop must be created using requestAnimationFrame to watch for input and determine which button was pressed.
Max walks through the Cooking Masters application. The application allows users to follow recipes without using traditional computer input controls. Users can use gamepads, audio commands, and other capabilities for navigating the interface.
Max demonstrates how to use the Web Human Interface Device API, which is built on top of the widely adopted HID protocol. Permission is granted through a browser dialog and the API provides low-level control with the selected device.
Speech, Voice, & Camera
Section Duration: 32 minutes
Max introduces the Speech Recognition API. Once microphone permissions are granted, an application can use the API to detect words or phrases in a specified language. A confidence value is also returned as the speech is processed.
Max demonstrates the Speech Synthesis API, which speaks a string of text through the computer speakers using the available voices in the operating system.
Max demonstrates the Shape Detection API, which provides methods for detecting barcodes, text, or faces using a device's camera. The Advanced Control Camera API gives the application control over pan, tilt, and zoom and can be used to adjust the input stream when live streaming or taking photos.
Max explains that augmented reality can be experienced in web applications using the WebXR API. Since this API is a low-level API, developers often prefer to use a higher-level solution like Model Viewer
Max discusses the Screen Wake Lock API, which allows applications to prevent the operating system from locking the screen.
External Hardware & Devices
Section Duration: 39 minutes
Max introduces the Web Bluetooth API, which provides low-level communication support with connected Bluetooth devices. The API can be difficult to use because instructions must be sent to devices in binary. Developers can find documentation with individual vendors or through third-party libraries built on top of the low-level APIs.
Max demonstrates how to use the Web Bluetooth API. Bluetooth lights are connected to the Cooking Masters application. When a time finishes running, the lights flash and the gamepad rumbles.
Max explains that the Web Audio API can generate dynamic or 3D audio. It's another low-level API that third-party libraries like SonicSocket simplify. The Web MIDI API is also discussed in this lesson.
Max discusses the Web Serial API, which communicates with serial devices connected to a user's computer via USB, Bluetooth, or another serial connection. Another low-level API, Web USB, is also discussed in this lesson.
Max reviews the APIs for vibrating a device, reacting to changes in status with the battery, and determining the idle or lock state of the user and screen.
Max demonstrates the Web NFC API by reading data from an NFC card and an Apple Airtag. The data transmitted depends on the tag/device. Developers process the data manually using the "reading" event.
OS Integration
Section Duration: 33 minutes
Max demonstrates the features and how to install progressive web applications. PWAs behave like native applications by running outside the browser in a native window. The history of PWAs is also discussed in this lesson
Max explores APIs for moving and managing windows on a screen. Older JavaScript APis for moving and resizing a window still work, but only with PWAs. Newer APIs can receive details from the screen and respond to changes in size or location.
Max walks through the metadata, which can be added to a PWAs manifest file allowing the application to handle specific file extensions. The differences between PWAs and technologies like Electron or PhoneGap are also discussed in this lesson.
Max looks at the Web Share API, which triggers the native sharing dialog in an operating system. An application can also be configured as a share target when a specific file format is shared. This adds the application to the operating systems share menu as a possible destination.
Max covers the Contact Picker API, Fullscreen API, and Payment Requests API. The Fullscreen API is used by the Cooking Masters application to launch into fullscreen mode when viewing a recipe.
Max shares use cases for a few additional APIs. The Installed Related Apps API is useful for detecting if a websites related application is installed so users can continue using the site in the application. The App Banner API is also useful for prompting users to use the installed application.
Wrapping Up
Section Duration: 4 minutes
Max wraps up the course by sharing some experimental capabilities, including the View Transition API, Popover API, and Picture in Picture API.
Learn Straight from the Experts Who Shape the Modern Web
- In-depth Courses
- Industry Leading Experts
- Learning Paths
- Live Interactive Workshops