Media Session & PiP Practice

[00:00:00]
>> Let's try to see this in action. Okay, so in background.js, now I think I have another one media.js, okay? So, here I have already a background audio playing. So if we look at the HTML, in the HTML I have a video element and an audio element. This is HTML.

[00:00:26]
Okay, nothing really fancy, but using JavaScript and these APIs, I can actually add all the metadata. So that's the change. For example, when I'm playing, I hear there is an open source royalty free song, so when I'm playing so in the bottom play, I want to add the metadata.

[00:00:52]
So that metadata will let me using the media session API, will let me in the background interact with OS, okay? So from here for example, in my Btnplay, before or after the play doesn't matter in the event handler, so I'm going to talk with the navigator.media session.metadata option.

[00:01:19]
And I'm going to create a new media metadata. There is a constructor for that, that receives an object with all the properties that I have to set. There are three mandatory properties title, artist and artwork. You need to provide those three at least. There are more properties that might be used or not or might be ignored.

[00:01:43]
So for example in this case the title is too much funk. That's the name of the song. Then we have the artist, this is the real artist, Steve Oaks. The album we could say, well, I don't need the quotes here, that this is Fronted Masters greater hits maybe.

[00:02:07]
And for the artwork, it's actually an array of images that is following the same format as in the web app manifest for PWAs. I'm not sure if you have that object in mind, but actually it has source with the URL of the image, type And sizes. Oops, so that's the structure, okay?

[00:02:38]
So I do have an E mash. It's media thumb dot png, okay? So I'm going to use that one. So it's media, from png. The type is image png, and the size is, if I remember correctly, it's 800 by 800. So you can provide many, and each OS will pick the one that it prefers, okay?

[00:03:06]
So there are no standard icon sizes here, so every OS will just take wherever you provide. So again, this is just metadata, and over the same object I could also listen for JavaScript actions. So then while you are in the background from the watch you can do something in JavaScript.

[00:03:36]
Makes sense? You have a question face, I'm not sure.
>> And then thinking about like the other ways that you can interact with that, so then like I think also like Siri and other voice assistants can then interact with that media session.
>> Yeah, yeah, that's correct, so both assistants they are connected to those as well.

[00:03:56]
>> That's very interesting that if you're like streaming it to one of those speakers you can then interact with your voice to have. You can navigate through part of your JavaScript.
>> Yeah.
>> I'm just thinking of the weird things you could probably do with that, like if you go backwards, it actually moves down the page.

[00:04:15]
>> Yeah, also let me increase the weirdness there, you could stream that if it's video it's only also you can stream that to Google cast device or Apple TV and also you can use your remote control. So from your remote control you could be actually executing JavaScript.
>> And the homepods can hook up to an Apple TV if you use the Siri on the homepod is gonna go to your Apple TV, which goes to your phone which controls a website.

[00:04:48]
>> Yeah.
>> Which is very very strange.
>> I actually could be someone on a television program that you're watching that says, remote execution attack,
>> Exactly, so now you have a remote, boys, child scraping checking attacks.
>> You could like use the seek function to scroll through your page and do percentage of the way through your song is how much of your website you're scrolled down to, and have the worst website possible.

[00:05:15]
>> That's what I want.
>> Well, let's see this in action. Let's see if that works. So I did that on the audio plane, is this one, so if I hit play, we have the song. And on the OS you can see here, too much funk, okay? With the front and master logo, so that works.

[00:05:40]
And if we run this on the phone I can show you that later if you want, it works also on the watch shaft by sending metadata, and that also works. Even if I minimize this, I'm not actually seeing that I can still play in that. I remember that plays not just playing the audio element, it executing the JavaScript event handler, and you can do wherever you want there, not just working with the audio that you're hearing, okay?

[00:06:12]
Make sense? So again, this is new in the JavaScript in the background space. This is actually compatible now with every browser, and it happened within the last year. It works on Safari also, so it works on Android, on IOS, on iPad OS, and desktop. Make sense? About Picture-in-Picture, we do have a button here that toggle Picture-in-Picture, but it's doing nothing right now.

[00:06:46]
So I'm going to add the code. If I wanna directly start Picture-in-Picture, I can get the element by ID or by tag name or directly. Our new friend queryselector. I can take the video element, I have only one. It just request Picture-in-Picture. It's just that fancy, so it's pretty simple.

[00:07:14]
And of course I can do the toggle thing, so it gets in or out. Well, I think it's minimized, it's. So if I hit Toggle immediately, even if it's not playing, it goes to a window. It's as simple as that, and if I want this to be toggled, you can see my window is there, okay?

[00:07:41]
I should check if in the document, we have the Picture-in-Picture element. And if it's there, it's because we are already in Picture-in-Picture. So we are going to exit Picture-in-Picture. And if not, we are going to request Picture-in-Picture. This is going to toggle. Make sense? It's as simple as that.

[00:08:09]
Any question? So toggle, toggle, gets in, get out. Of course the user is in charge also, so you can actually hit the exits Picture-in-Picture.
>> So this Picture-in-Picture will be available even if you move from one tab to another tab.
>> Yeah. So it's even if I minimize, let me toggle this.

[00:08:34]
Even if I minimize the browser or if I go to another app, it's always there, running in the background. In this case is, is it running JavaScript? Not in my case, but it could be running JavaScript as well. In fact, the code that generates each frame of the video can be JavaScript.

[00:08:56]
>> Is this still having the throttle issues or it will run?
>> No, because when you are rendering media playing in the background, your timers are not being throttled in the same way. So you have the same power, the same abilities as if you're in the foreground, because you need to provide enough data to keep the frame rate.

[00:09:19]
So the browser is giving you more power. Of course can you abuse that? Well, the user is seen the Picture-in-Picture, so it's difficult. Can you cryptomine? Can you do cryptomining with this? You can, but the user will see something that means you need to entertain the user while you're doing cryptomining with something, because if not the user will close it.

[00:09:43]
And if it closes it goes back to normal. So we request Picture-in-Picture to the video element and we're requests exit in Picture-in-Picture to the document.
>> If you said this is added u all videos automatically?
>> It depends on the browser but, yeah. In that case, it depends on the browser but if the browser like Chrome for example is adding that if you're in probably macOS.

[00:10:12]
So if I go back to the tab, where is my tab? It's minimized. There is a menu here, so it's always here in the drop down menu. Maybe Safari. So there is one. Yes, Safari. Safari is the one that is giving you the icon there no matter what, even if you're not adding any JavaScript.

[00:10:38]
So Safari works in the same way. Okay, so this is the one of the biggest background execution rules that has nothing to do with service workers.