Transcript from the "Accessibility" Lesson
[00:00:00]
>> So the question was how do you do accessibility in WebXR? It is a tough one. We are trying to figure this out ourselves. But with XR itself Is only a way for the BPI to give you the sensor information for you to create a immersive web experience.
[00:00:24] So basically, WebXR is not in itself rendering anything on your screen. You are doing that with WebGR or you're doing that with Babylon or Three.js. But you are having access to it through the WebXR APIs. So, since we have nothing to do with the content, it is hard for us to have any enforced way of having accessibility.
[00:00:53] But one of the things that is coming for WebXR is layers which is a way of adding a layer of HTML on top of the Canvas, and maybe that layer can follow you everywhere. Having layers allows you to have some alt text as well. So you could connect those layers to maybe the objects and make it accessible that way.
[00:01:24] But unfortunately, that all depends on the developer itself, which is a hard situation to be in because, It is already, you have to cover so many situations with any AR or VR experiences, especially for AR, you don't know the space, you don't know any of the needs of the user.
[00:01:50] So that's a huge problem. But having said that, there are really cool things that you could do for accessibility, if that's your purpose. One of them is, I'm going to search for is seeing AI from one of our coworkers. And thank you for asking this. I love showing this video.
[00:02:17]
>> I'm Saqib Shaikh. I lost my sight when I was seven, and shortly after that, I went to a school for the blind. And that's where it was introduced to talking computers. And that really opened up a whole new world of opportunities. I joined Microsoft ten years ago as a software engineer.
[00:02:38] I love making things which improve people's lives. And one of the things I've always dreamt of since I was at university was this idea of something that could tell you at any moment what's going on around you.
[00:02:50] [MUSIC]
[00:02:53]
>> I think it's a man jumping in the air doing a trick on a skateboard.
[00:02:56] [MUSIC]
[00:02:59]
>> I teamed up with like minded engineers to make an app which lets you know who and what is around you. It's based on top of the Microsoft intelligence APIs, which makes it so much easier to make this kind of thing. The app runs on smartphones but also on the Pivothead smart glasses.
[00:03:17] When you're talking to a bigger group, sometimes you can talk and talk and there's no response and you think, is everyone listening really well, or are they half asleep? And you never know.
>> I see two faces, 40 year old man with a beard looking surprised 20 year old woman looking happy.
[00:03:36]
>> The app can describe the general age and gender of the people around me and what their emotions are, which is incredible. One of the things that's most useful about the app is the ability to read our text.
>> Hello, good afternoon, here's your menu.
>> Okay, thank you.
[00:03:54] I can use the app on my phone to take a picture of the menu and it's gonna guide me on how to take that correct photo.
>> Move camera to the bottom right and away from the document.
>> And then it'll recognize the text. Read me the headings.
[00:04:08]
>> I see Appetizers, Salads, Paninis, Pizzas, Pastas.
>> Years ago, this was science fiction. I never thought it would be something that you could actually do. But artificial intelligence is improving at an ever faster rate and I'm really excited to see where we can take this. As engineers, we're always standing on the shoulders of giants, building on top of what went before.
[00:04:32] And in this case, we've taken years of research from Microsoft Research to pull this off.
>> I think it's a young girl throwing an orange frisbee in the park.
>> For me, it's about taking that far off dream and building it one step at a time. I think this is just the beginning.
[00:04:47] [MUSIC]
[00:04:49]
>> This is one of the applications that shows the potential of AR and how useful it could be. And anytime you build something that is accessible and for the purpose of accessibility, you are making your application much more usable for everybody else. Interestingly, for example, the typewriter was invented for a blind person.
[00:05:14] And there are all kinds of things that we have in our life today that we use daily was created for making somebody's life much better, someone who needs the accessibility feature. So that's something that I'm very passionate about, so thank you for asking. There's so much opportunity, but in terms of tooling, we need a lot.
[00:05:41] One of the things as the community group at W3C, we want to incubate these kind of ideas, these kind of resources to make it much easier for people. And if you join our group, you would end up getting the emails for our meetings and then maybe one day, we'll just all come together and build great WebXR accessibility tools.
[00:06:14] One of the things that we have here is a group that that we do is for example, the examples, Immersive Web Weekly is our newsletter that we send out every week. And you can submit your blog posts or your applications to be part of the Immersive Web Weekly as well.
[00:06:35] And we also have tools like WebXR input profiles. This is one of the cool things that we asked the manufacturers to have the input devices like the hand controls. And so, depending on your user's device, you can send some hand input controller model, 3D model that matches their device.
[00:07:10] Patrick here said if you are someone who is not able to move, it is really hard to have VR experiences, which is true. But at the same time if you give your user the opportunity to have the same experience, for example, standing up, moving around, plus sitting down and not moving around and be able to, for example, transfer from one location to the other without having to actually move those, maybe not all, but some will benefit from that.
[00:07:44] And for that, one thing I always want to say is have as many options as possible for your users. For example, with eye tracking, you can help people to write or communicate a little bit better. And instead of having to walk a distance, you can create movements with your controllers that will help them to move along.
[00:08:15] One other thing that is possible is with spatial audio. Saqib here created that app, Seeing AI, and it's the most popular app for visually impaired in US. And last time I saw him, he was adding some spatial audio. So with spatial anchors, you are able to leave kind of like breadcrumbs along the way.
[00:08:44] And people can listen to it and say, okay, this is coming from this site and if I follow it, I end up going to the location that I want to. And imagine for a person who is not able to see going into a new building. Saqib himself moved to Seattle recently from London.
[00:09:04] There's just so much to learn. And every time you go into a building, where's the bathroom? How are you going to figure it out in a big place? With little things like that, we can create a much more accessible world, hopefully. But the challenge is we have no text.
[00:09:27] And we don't know the objects that are going to be there. So it's really hard to create some spec around it to say, these are the rules that you're going to follow. So, [LAUGH] if you're interested in the subject, there's also this really great organisation called XR Access.
[00:09:54] They do a lot of research and they have a lot of resources and also they are working group. They come together and work on different parts of the accessibility. Any other app ideas that is inspired from today hopefully? And please do let me know through social media or Twitter, wherever you would like, what did you think about Babylon versus Aframe or Three.js, and what kind of Resources that you are interested in.
[00:10:44] Cool. It looks like that's end of our questions. Thank you everyone for your attention. Here is my Slides, WebXR slides, and you can find all the repositories and all of the resources that we covered today, and hope to see you next time.