Wrapping Up

[00:00:00]
>> Male 1: If I want to build an open source library composed of multiple packages like Lib, Course, Lib, Utils, libreact, all inside of one monorepo, using Lerna or nx, what's the best way to structure it for long term maintainability? Should it just be each feature in its own package or is it better to keep everything in one package with an internal structure?

[00:00:23]
>> Mike: Yeah, that's a great question. There is a trap of trying to split things apart too much too early. And what I would say is look for, I would rather have a fat core package that I later understand I need to factor some things out than start in a world where every single feature is its own package and then have to worry about.

[00:00:52]
All right now people are using these things sort of very difficult to claw that back, right? Once people have installed these things into their apps, you sort of, I mean you can just stop publishing packages. But that loss of continuity of like a particular piece of functionality being available in a certain package, it's much harder to correct the problem where you fragmented too much and need to consolidate than like, all right, you've not fragmented enough and so you factor something out into its own package, but then you can always re export it from the original package and it still feels okay.

[00:01:30]
One exception, and that would be if whatever you're building is intended to be something that involves plugins and in that case maybe your features are plugins. That's where I would say, all right, well invest some of your mental energy in defining a good plugin interface and then you will want to have some world where maybe your internal features, some of them are modeled as plugins and then you can also let other people write libraries that also act as plugins.

[00:02:05]
That would be the case where maybe you end up with a lot of packages earlier rather than waiting for a need to introduce more packages to arise before factoring things out.
>> Male 2: Have you ever had to override the NX cloud to store the task caches in your company, whatever company, cloud provider, cloud storage use, whether it's like cloud storage or S3 or anything like that.

[00:02:37]
>> Mike: Sorry, have I ever done that?
>> Male 2: Yeah. Have you ever had to override instead of using NX cloud, their cloud service, storing the cache in your S3 bucket.
>> Mike: Or something like that, not while using NX. So I kind of these days do two kinds of development.

[00:02:56]
I do open source stuff, in which case I am happy to use NX and I'm fine with everyone being able to see the build artifacts. They would be able to see them in GitHub Action Logs anyway, and then on the other side, monorepos that aren't just JavaScript and that's where we're using Bazel and we have a similar concept of being able to take advantage of distributed build artifacts, but it's not something we'd use NX for and that would be like at Stripe.

[00:03:32]
Does that make sense? And in that case, absolutely. We don't want to be tossing our not yet released build artifacts onto another person's infrastructure. I mean like Amazon, sure, but a cloud provider is who we'd be trusting there. Not that the NX team is not trustworthy, but I think many enterprises would make a similar decision.

[00:03:59]
>> Male 2: Yeah, that's why I looked into it because my company was like. So I was curious if you've had to use that any other corporate side.
>> Male 1: Not on the corporate side, but yeah, sorry, I can't really provide a better path forward for you there.
>> Male 3: This isn't necessarily related to monorepos, but I was curious if you've heard of Archetype before.

[00:04:24]
>> Mike: No, I have not. No, it's similar to Zod.
>> Male 3: I'm basically trying to get into some data validation stuff for TypeScript since we don't really do any of that right now. I know Zod is the one you covered on last one, but I also looked into Archetype as well.

[00:04:49]
>> Mike: I mean Zod is used extensively and I think like I haven't seen Archetype, but when evaluating libraries like that, I would look at, you know, how likely is it that you can hire someone and they already know how it works, or how likely is it that when you run into a problem, you can go look on Stack Overflow or let's be real, ask an LLM for help and like, how many examples do they have to pull from?

[00:05:20]
And don't forget this as a TypeScript or JavaScript developer, there is an enormous amount of TypeScript and JavaScript code that LLMs have ingested. But if you try to get an LLM to help you with a more esoteric programming language like Elixir or something like that, it's a lot less on target because it just has a lot fewer examples to pull from.

[00:05:44]
And similarly with libraries like there is, if you're using AI coding help, there is an advantage to going with an option when you're making choices. But there's just a ton of usage out there and a ton of stuff in GitHub and GitLab open source that all helps Build, I'd say an understanding, but really a database of autocomplete results that will be more likely what you want to implement.

[00:06:20]
A more helpful answer, if that makes sense.
>> Male 3: Archetype seems cool because it literally uses TypeScript syntax at runtime.
>> Mike: Interesting.
>> Male 3: Can take TypeScript syntax and it'll take that to runtime and then do data validation. You don't have to learn any new APIs, which might make it an easier cell.

[00:06:39]
But also Zod is more proven, I guess.
>> Mike: Well, and in that case maybe what I just said is less relevant because if the way you express these things is in typescript types, then there is plenty of information out there on that and it makes sense. Like there are lots of libraries where you can extract a JSON schema or a protobuf or whatever you want from a TypeScript type, and it stands to reason you could use that to compile a validation of some sort.

[00:07:06]
>> Male 2: I'm gonna piggyback off of Pepe's question here in the chat. Do you have any best practices for building NX generators? Or do you have any favorite ones that you or your company have built and use?
>> Mike: Bluntly, I use the off the shelf stuff. Sorry. I do have a best practice.

[00:07:26]
It's not a best practice. I can offer my practice for something like our dev script. I would just let you know about this more, don't overlook this as a possibility thing. Not that it's the answer to everything, but a lot of people don't. Seems obvious when you think about it, but you can create a package that NX sees as a package and it has a Project JSON file.

[00:08:00]
You could just create a folder with a project JSON file in it, and it's kind of a fake package. Like it has nothing to do with software that's being imported into other packages in your monorepo. But that lets you. It has the effect of letting you almost like create commands that are of the shape that you want to create.

[00:08:19]
Like if I wanted to create something that was like NX dev serve, well, all I need to do is create. Really? That's following the pattern of NX UI test. Oops, sorry, I got it backwards. NXTest UI, yes, terminal, too small, gotcha. So really what you're saying is I want to create a target that is called dev and a project that is called serve.

[00:08:54]
And so this lets you, if you lean into that, these are just two words. And so, you can create a combination of projects and targets that let you make a very semantically relevant command. If you're just like, you're willing to break this sort of implicit pattern, but it doesn't have to stay that way of like, well, the project JSON goes in my models folder and there are absolutely no other places I can put this project JSON.

[00:09:24]
Like, you can make NX aware of projects that have nothing to do with your typescript monorepos here it could just be like another folder in your repo, like another top level folder that's not even packages. Does that make sense? This could become like noun, verb, or whatever you want it to be, because these are really just task definitions, like targets.

[00:09:51]
And then this last word is a project. A project is just a project JSON file. It does not have to be mapped one to one with a package JSON. Usually NX commands are sometimes not incredibly ergonomic. And so I will reach for that when I have something that's really common that needs to be run and I want it to be stupid simple like NX format manifests or NX lint css.

[00:10:31]
And you could do that if you just set it up this way. And you're not abusing the tool, you're just cleverly naming the targets and the projects while benefiting from the target dependencies and all of that, right? Like, of course you could do PNPM whatever you want. Like, you could use NPM scripts, but NX can do that too.

[00:10:54]
It just requires some more JSON.
>> Male 1: So this one's tangentially related, but you seem to be a big advocate for pnpm.
>> Mike: If I were to go back to my company, how do I sell PNPM to them to replace npm? So the most obvious benefit is really a developer productivity thing where every install is faster, which makes CI faster, local build startups are faster.

[00:11:29]
It means less compute happening on your dev machine. And so that means, I mean, I'm sure Stripe is not the only company where this happens, where developers get issued at the time they feel like these really powerful laptops and then you get to a point a couple years later where you're like, I only have 16 gigs of RAM on this thing.

[00:11:58]
God, I've got two dozen Chrome tabs open and I've got an NPM install going in the background and it. It sucks. So, yeah, I would say mostly the developer productivity angle is the main thing there. Alternatively, if the company is paying for storage of build artifacts, just size on disk with PNPM is so much smaller.

[00:12:26]
I mean, sorry, for a sizable project it's so much smaller because of just the way the linking works and the flat structure of that cached set of dependencies. Gotcha, interesting.
>> Male 1: Thank you.
>> Male 2: How much a lift do you think it is switching at scale?
>> Mike: What dimension of scale?

[00:12:52]
>> Male 2: I don't know, 100 packages or something like that.
>> Mike: 100, I mean that would be a dimension of scale where I would say doesn't matter. That should be fine. You'd start out with pnpm. You can point it at your package lock JSON or your yarn lock file.

[00:13:10]
It'll generate the right PNPM lock file. That doesn't drift any of your dependencies. That's fine. Here's the other dimensions of scale. What are developers used to like? How you kind of have to train them up a little bit. You're gonna get people who are a little curmudgeonly about like, there was this one feature of this thing that I really liked, where there is a little bit of a trap is if you're using some custom tooling that relies on a different layout of node modules.

[00:13:52]
That's the thing I would look at first. If you have wacky scripts that kind of depend on certain things being in the node modules at the workspace level and certain things at the package level, those are the things that are likely to break. But the good news is that's sort of a fail fast task there.

[00:14:11]
You try it out and it'll be obvious if it breaks. You just need to exercise all those tools and see if something produces a wacky result of some sort. But really the scale that I would say is what to worry about. It's not the 100 dependencies, it's the 100 engineers and kinda helping them understand.

[00:14:36]
All right, here's how to think about these things. Workspace colon is what you do. It's more about any tool requires this kind of thing. Sort of the cultural rollout. I mean, these things, their implementation is not simple, but they all read a package JSON, they all have a lock file, they have differing articulations of how workspace works, but that's a relatively small surface for them to be entangled with your code base.

[00:15:12]
And so, it depends. But granted you could have a bunch of custom stuff that's doing a bunch of NPM link everywhere and that's not going to be friendly with pnpm. So maybe the scale dimension that makes this really hurt is how much custom software has been built on top of the conventional use of these commands that more tightly integrates you with one package manager.

[00:15:38]
So I hope you enjoyed learning about TypeScript monorepos and some of the great tools you can use to make your builds fast, to make developing on a big project feel small in a good way, and how even in a world where you have many packages inside the same git repo, you have great ways to trim down your dependencies, to enforce consistency standards across your code base, and generally to make sure that you get to reap a lot of the benefits that the concept of a mon repo promises.