Video Upload Entities & Workflow

[00:00:00]
>> Jem Young: OK. Now what do we want to define? The entities in this, OK. Move down a space. So what would the entities be? A user, that person doing the uploading, yeah. What else? Source video file. Yeah, the video. OK, you already said source video, very fancy, but accurate. If we say video, what do you mean talking about further down the road? Anything else? Wouldn't the entity be video and then you might have different versions of it, source and processed?

[00:00:55]
Oh yeah. If you think of something like iMovie, there is a project, and then there's like raw video in the project and then there's like output video in the project or something. I mean, this is what we say it is. We'll stick with video, and we'll give it different flavors down the road. You need thumbnails though. Thumbnails? Yeah. We'll decide if that's a core entity or not, but it's worth mentioning.

[00:01:34]
Anything else? We said the audio tracks were separate audio. Hm. There is one thing we're missing, and this is something like endemic to a lot of systems design problems that you have to already know to think of this. We need metadata. Something really easy to forget. It's not fair because if you don't know about it, then you wouldn't think of it, but videos inherently are just, you know, I'll use the non-technical, it's just a bunch of pictures stitched together with an algorithm, but it doesn't have a title inherently, it doesn't have any sort of idea of what the resolution is and all that.

[00:02:21]
You have to attach that with the metadata, attach a video and all videos have metadata. But I'm sure the user uploaded it and gave it a title. How do we persist that title throughout? Are we putting metadata? There's one other one here. Again, it's not fair, you have to know this already about audio and video. In a lot of systems design, they expect you to know this.

[00:02:45]
You need what's called a manifest. Not as important for this one, but the manifest determines the bit rate that the client plays at along with all the metadata. And every client's going to fetch something. What resolution does it play at? That's determined by the manifest. It says some calculation on the back end based on your client and other things like that.

[00:03:03]
You just have to know that already. Is the manifest related to the user's client or is it related? OK, yeah, and it's computed on the back end, so it's different for everybody. But the reason why you need something like that is a phone showing a 4K video, waste, complete waste. You can't tell 4K from HD on your phone. It's just too small of a screen. So you want a specialized manifest.

[00:03:30]
So it's really tempting and naively to go into any sort of video playing scenario and say I want to show the client the best possible resolution. When in reality you don't want to show, you want to show them the best possible resolution that makes sense for their device. That's where you need to manifest versus just sending everybody's going to get 4K all the time.

[00:03:57]
Doesn't make as much sense. OK, we're grooving. A lot to think about. All right. What should we do now? Right now we're making an assumption that we're going to upload a video and the user is just going to wait for it to process and they're going to want, you know, and then they're going to see it. We didn't talk about like the fact that we are going to be queuing something up and processing it asynchronously and that might need like a status of some kind, but we could always get to that later too.

[00:04:33]
Let's do it now. Again, I mean, I have the diagram here just by default, but we haven't even gotten to the diagramming yet because we'd be wasting our time because we don't know what we're building. So we have the entities. Let's walk through, let's just talk it out. What would a workflow look like? So it starts with a user uploads a video. You know, and we're not overcomplicating it with like, oh, you need a UI and then the UI loads and it does all these things.

[00:05:18]
Not in scope right now. So the user uploads a video. The server has to store it for sure. Possibly create the metadata or probably create metadata and link the audio as well. Further down, thinking too soon. So user uploads a video, do we assume that that upload was successful? We can't. This is, don't, just because I ask you doesn't mean like, oh, it's Jem asking you a rhetorical question.

[00:05:54]
Should we assume that in our system or no? Especially not with big uploads. OK, so user starts. We'll say a user starts uploading a video and they're sending the metadata at that same time, right? Yeah, maybe. So if the client errors, you probably need a way to recover from that or you're going to have to handle a lot of redoing the same data. It might take forever or their internet connection might not handle it.

[00:06:45]
Yeah, let's simplify that down and we'll just say user is notified when the upload is successful. Two C's or one? I always forget. Successful. Two S's, right? I think it's two C's. OK. Yeah, we'll assume the retrying and all that, we'll leave that out of scope for now. We can come back to it at some point, but it won't affect our design too much. OK, so at this stage, the video is somewhere in some sort of blob storage.

[00:07:35]
Then what? What do we need next? Or I begin to process it, yeah. Process video. So our goal is to take this giant video and then we want to break it down into individual formats so they can display it on mobile, TV, so we want to break it down to 4K and whatever, all that. We're not going to get to video codecs and all that, but assume we want to break a video down to multiple resolutions that can be displayed on clients or on mobile devices, TVs, etc.

[00:08:23]
So that's our goal, one video into many. Users notified when the processing is complete, yeah. Pretty good workflow. Seems like a normal video uploading workflow. What is the purpose of uploading the video? Like is this site meant to like, it's a service to clean up videos and then at the end you get to download your video or something and then that's the output.

[00:08:47]
That's the final step. You get, they'll get notified when they're notified that the processing is complete, then they have like an option to download it or what's the, I guess I don't know what the end game is so far. That's fair. Let's say we're designing YouTube or social media, Twitter. I upload a video. We're not going to send, we're not going to share out the exact video you uploaded.

[00:09:15]
It has to be processed in some way. Maybe it has to be more than likely compressed into a different format like WebP or something that actually you can stream. But that's the end, that's the end goal is we want to share our video on something. Good question though, Nick. Yeah, we didn't ask why are we doing this? Just because.