Introduction

[00:00:00]
>> Steve Kinney: My name is Steve, we're gonna talk a little bit about some of the tools around building your own language. We'll talk about it from a few different angles today. The repository we're gonna be using today is right here, we're gonna make a little language called dropbear. I'll explain what a dropbear is when it's time, not beforehand.

[00:00:19]
As I mentioned, my name is Steve, and I work at Twilio SendGrid, Senior Principal Engineer which is really hard to say multiple times, quickly but you know. Basically if you ever get a text that was sent by a software program or an email, we probably sent it to you through now our new combined business.

[00:00:41]
I work on the marketing campaigns product, which is kind of like a in-browser, HTML editor built in react. And as I'll mention a few times during this, a lot, this came out of having to do a lot of parsing, and compiling in my day to day, right? And so seeing how some of these things, we're gonna kinda look at it through playing with a language, but these are things that I have solved actual problems with.

[00:01:12]
Not that building a language isn't solving the problem of scratching your niche or anything like that, but you can also do it at work, which is kind of cool. The good news is as long as you rein in your ambitions, right? Many of the really popular languages that we use are someone's full-time job.

[00:01:31]
And usually not just someone's full-time job, a team of someones' full-time job, right? Usually with doctorates, and stuff along those lines. The good news is that you don't need a doctorate to use a lot of these techniques and tools, you don't need it to build your own little language as a hobby and you don't need it to solve very real problems in your day to day lives.

[00:01:50]
A language or any of these parsers or compilers are just programs in and of themselves. They're no different than the software that we write on a day to day basis. So why would we want to do any of this? You may not want to write your own programming language, it's not like, I'm gonna quit my job.

[00:02:08]
I'm gonna sit down and I'm going to write the new language that's gonna change everything versus the one that changed everything was written by some dude in ten days, and two plus two equals two. So there's some very kinda practical use cases. In fact, you probably use some of these tools on a day to day basis, whether it's something like Terraform.

[00:02:31]
We do all of our deployment with this idea of code is infrastructure. So we can write basically what databases we need, how many Amazon EC2 instances. So on and so forth, and that gets parsed, and commands just have to configure all of our infrastructure. Even figuring your dependencies, in npm, it's a JSON file and it's pretty standard, but a gemfile, or a TUML.

[00:02:57]
Or anything along those lines need parsers to figure out what's going on, they need to be evaluated to do their thing. Templating languages are near and dear to my heart. And a lot of the problems that I have needed to solve and templating languages are really great because they like to flex that muscle a lot.

[00:03:15]
And also our very confined space of what you know you need to do. They're a really great place to start. Also, it's kind of fun. I think it's fun. It's one of those kind of fun that hurts sometimes. [LAUGH] It gives you a headache, but it's a lot of fun.

[00:03:36]
The kind of meta point is that there's a bunch of different techniques that we're gonna kind of explore throughout our brief time together today. Any one of those kinda individual pieces can even be used in isolation to solve practical problems for yourself. So what do we do with some of these techniques?

[00:03:58]
I didn't know that I'd be writing so many parsers when I started to work on an email editor. Our original version of the editor, years ago, was jQuery and CoffeeScript and when you made changes to the email, we literally manipulated the DOM, and that's how we stored everything as one big HTML file.

[00:04:19]
And like most things, you can't really change an API. So as we built newer and newer versions of the product, basically the API was giving us back a full HTML file. But the new versions didn't react, in the Redux store, that's not a lot of DOM manipulation going on there.

[00:04:36]
We basically need to take that HTML file and turn it into like a large kind of tree like structure. Yeah, we can pass into React components. The first time around that we implemented it, a few years ago, we're like, okay, what we will do is we'll use jQuery and we'll just start navigating the DOM to find all the stuff that we need and then we'll put it into the React components.

[00:04:57]
And that sounded really great. Except for the fact that jQuery makes real DOM nodes and the browser tries to be helpful and fix HTML for you. Now, that seems great, good job jQuery. The problem with that is that we are trying to make emails for Outlook 2013, which intentionally needed broken DOM, [LAUGH] in order to actually render correctly or do anything along those lines.

[00:05:22]
So it was actually really problematic. So when we do the new version, we actually used a parser, broke into a large AST, an abstract syntax tree, started navigating and traversing this data structure to do all the manipulation that we need. So we break into this data structure and as the user starts tweaking things in the sidebar, moving stuff around, we're basically manipulating that tree and moving stuff around with some of the same techniques that we're gonna play around with a little bit today.

[00:05:47]
We are also building a templating language that started out kind of like Handlebars, but it's kind of morphing into its own thing. And so we're able to even use a lightweight version of some of these techniques to put a layer over Handlebars to kind of pre-compile it, so that we could bring it to customers and do our initial research before having to build an entire thing.

[00:06:07]
And we just kind of used some very basic parsing and moving stuff around in a tree like structure again to ship that really quickly. We have a code editor where you can click on the code and it brings you to that place in the preview, you click on the preview and it highlights that DOM node.

[00:06:21]
For that, we basically have to pull apart the HTML, annotate every DOM node with where it was in the text file. Again, very simple features that would have been a lot more complicated if we didn't have some of these tools at our disposal. On top of that, we have a custom query language that looks like SQL for making segments of your customers to trigger automated messages to along those lines.

[00:06:43]
[LAUGH] First, just take an arbitrary SQL off the Internet and executing it, not okay. But also, it is a custom language that we have to actually parse in JavaScript and zeros in Java as well go on the back end as well. And so there's tons of places in just day to day features for us that we end up using these things, and so, yeah, whether or not it's your own full language, there's still also a lot of just practical day to day places.

[00:07:11]
So my goal is to take the pieces of this workshop as inspiration either to build an entire language or to solve problems easily and probably in a more sustainable way. We haven't had to go back and change a lot of that code like we used to when we're either trying to RegEx it to death, or whatever other fun techniques we were trying at the time.

[00:07:33]
It ended up being some of our more stable code. And there's also a lot of tools that we all use, we use PostCSS or Babel JSX. That's all using the same kind of techniques. So we're surrounded by it all the time. And it's good to know how the tools that we're using every day work.