Video Upload System Design

[00:00:00]
>> Jem Young: So let's diagram this out. A video upload workflow. I have not done this before. We're going to do it in real time. We said we're going to do more diagramming, let's do some more diagramming. But I think we know, given the proximity to the previous topic that there's probably going to be some sort of queue involved in some of this.

[00:00:20]
All right, let's do it. They open up a new design. Started my base design here. But before they do that, let's talk about what we need to do. I want to enable, I want to have a service and I enable people to upload busy videos, user generated content. What are things I need to think of first, what?

[00:00:48]
Where, where should we start? Max resolution for the videos that we support. It's a good one. Supported resolutions. Expected size of the videos in megabytes or whatever. The size limit. Yeah, size limit. Is there a size limit? We want thumbnails generated. Yeah, great call out. Do we need thumbnails?

[00:01:45]
Uh huh. Is this service supporting lots of reads as well, like, is it just an upload service are we streaming out as well. We'll scope it down, just uploading. Don't worry about streaming part. Yeah, keep it, we'll be easy to ourselves this time. What type of files, what type of video? What? Like video uploads we support like what like is it an MP?

[00:02:14]
I don't know what's a video format type of MP4 versus something else, or it can be anything, or does it have to be something specific? How many users or clients are going to be able to upload? Yeah and users are uploading it at once something like that. On thumbnails too. Thank you. Do we need to do any preprocessing like a trim, dead air, in the video?

[00:03:13]
These are pretty good. I'll throw in one. One more good call on the thumbnails, that's really easy to forget when we're generating videos is that comes with most videos these days. Well, maybe I'll throw in 2 more, but. What do videos also have? Transcriptions. So captions. Translations maybe? Say translation or subtitles.

[00:03:58]
And audio. Easy to audio. Do we need? Caption subtitles. Do we need to. OK, this is pretty good. Just an observation about all of you. You've all gotten much faster at this. Remember when we first started? And I was like, what questions do we ask? And it was always like, uh, now you all are just like, I'm there, I'm in the domain right now, I'm thinking.

[00:04:37]
What do I need, not from a system standpoint, just from a feature standpoint. So, nice job all of you. All right, it's important to recognize, you know, what you're doing, when you're growing. So because this is our own service, I'm going to design this to be easy for ourselves, OK, lazy. Why are you in the middle?

[00:05:12]
Oh well, like a new one. OK. Supportive formats, resolutions. I want to say something like that's an easy way of putting it. Up to 4K. I don't know, video resolutions off the top of my head. There's like HD super HD. I don't know if that's the same thing, but we'll say up to 4K and everything in between, just to make it easy.

[00:05:40]
It doesn't have to be too precise on this one. Is there a size limit? To the degree that matters, we'll say max file size, 4 gigabytes. Is that a reasonable file size? I don't know. Does it matter? Not really. We just need to be directionally correct and have some sort of limits. What's the other question?

[00:06:06]
How many users are uploading at the same time? I don't know, how would you figure out the ballpark on number of people uploading a video at the same time? Remember, we're not looking for precision, just what, what's your intuition here? How many users do we have? Yeah, how many users do we have? How often do they upload on average?

[00:06:32]
Give me some numbers. Maybe one a day per user. OK, yeah. You have 1000 users, that's 1000 uploads a day. So 1 user is making 1 upload a day and we have 1000 users, OK. Fine by me. Do we need thumbnails? Yes, yes, to thumbnails. Do we need to trim edit the video? I'm going to say no. Do we need captions?

[00:07:21]
Not today. Yeah, not today. Yeah. Let's make it easy on ourselves. Do we need to process audio? I would assume that based on the format, the audio has to be dealt with differently. Yes, on audio. Yeah, it's kind of, it's kind of hard to justify getting like I can get away with like not doing subtitles, but if you, if you're modifying a video, converting the format, you're going to have to mess with the audio a little bit.

[00:07:49]
Don't some formats embed the audio so you can make a choice on what formats you choose to accept. Yeah, I think I don't know KV files support multiple audio tracks out of the box. So let's, you know, if we make an assumption, let's say it out loud and let's write it down. Let's assume that the audio track has to be treated separately when we're processing the video, OK.

[00:08:20]
We're getting somewhere. All right, let's move on. What's next? Nonfunctional requirements. Normally, yes. Let's, we'll be easy on ourselves today. We'll let it go. We actually captured some of that with 1000. 1000 users, everything else beyond that. But we'll, we'll note it. Keep my, keep my own advice.

[00:08:50]
No perf metrics on upload speed. Might change this later, but we're not trying to hit all the uploads need to be done and processed and available by some time frame. Just be easy for now. Maybe we'll change that.