Temporal
Building Your Own Programming Language
3 hours, 9 minutes
CC
Course Description
Learn the essential parts of programming language design! Through creating a new programming language, you’ll learn how to break down a program into its necessary elements: first into tokens through lexical analysis, and then into a traversable abstract syntax tree composed of those tokens. You’ll also write an interpreter, and learn how to transpile your language to JavaScript to make it runnable.
This course and others like it are available as part of our Frontend Masters video subscription.
Preview
CloseCourse Details
Published: October 8, 2019
Learning Paths
Topics
Learn Straight from the Experts Who Shape the Modern Web
Your Path to Senior Developer and Beyond
- 200+ In-depth courses
- 18 Learning Paths
- Industry Leading Experts
- Live Interactive Workshops
Table of Contents
Introduction
Section Duration: 15 minutes
Steve Kinney introduces the course, gives different use cases, and covers day to day examples of needing to build a programming language.
Steve defines what a compiler is, what the different stages of a compiler are, and what an interpreter is.
Steve introduces the course project, which is to create a scheme-based language called dropbear.
Parsing: Lexical Analysis
Section Duration: 29 minutes
Steve defines both lexical analysis, or lexing, and syntactic analysis, which are both a form of parsing, and then dives into how lexing works.
Steve explains why helpers are useful when thinking about building a programming language, and demonstrates how to loop over an array of tokens with a helper.
Steve gives a preview of the first exercise, of the repository, and dives into creating a token that breaks a string into an array of tokens.
Steve goes over the first exercise, live codes a test that looks for numbers and letters in order to tokenize them, and explains how lexers tokenize numbers.
Steve moves on to the second exercise, and lives code a test that looks for characters and strings and tockenizes them.
Parsing: Syntactic Analysis
Section Duration: 25 minutes
Steve explains that once types are broken into tokens, the abstract syntax tree (AST) breaks the code into a meaningful structure that gives the intent of what was just tokenized. AST makes it easier to transpile the language created into JavaScript.
Steve demonstrates how to write an AST to make it readable for a babel-like compiler.
After seeing Steve live code the parsing of numbers, the students are instructed to add functionalities to their parser so that it is able to parse both identifiers and strings.
Steve live codes the solution to the exercise.
REPL and CLI
Section Duration: 45 minutes
Steve defines what a REPL is, gives different examples of existing REPLs, and introduces the next sections.
Steve explains what are the basic functions put in place in the standard library, and mentions a helper included in the library.
Steve starts live coding evaluating expressions and error messages that would give more explicit information to the developer when looking for an error.
Steve explains how to write identifiers, and encourages the audience to write a max identifier of a method that finds the greatest value.
Steve goes over the solution of the basic evaluation, and succinctly reviews the different elements necessary to create a REPL.
Steve demonstrates how to build a REPL, and tries a few commands, such as add and subtract in the new created dropbear shell.
Steve demonstrates how to build a CLI, create a command that allows developers to access the dropbear shell, and create a dropbear file extension.
Traversing and Transforming the AST
Section Duration: 44 minutes
Steve explains what generation, or parsing in reverse is, and gives an overview of the babel generator.
Steve explains what transformation is, namely breaking apart the source code and being able to manipulate the AST, and explains what the Visitor Pattern is by giving the example of the babel transformation. The next step is to build a Visitor Pattern for the dropbear language.
Steve explains how to traverse and transform a tree, live codes the traverse function containing node, parent, and visitor arguments.
Steve writes the traverse test and gives a brief overview of the traverse and transform exercise.
Steve live codes the traversing and transforming, and builds a new visitor function that can traverse the AST. Visitors add different ways to process items, in the instance of dropbear, a visitor modifies a node after traversing the tree.
Steve live codes a babel visitor that transpiled code to JavaScript, and explains that generally transformation happens when entering a tree.
Language Features
Section Duration: 26 minutes
Steve adds the ability to create variables to dropbear. Namely, dropbear needs to have the ability to understand variable declaration, which means creating a variable node to the AST tree, add logic to the evaluator, and test everything in the REPL.
Now that variable declarations are defined, Steve explains that the next step is to be able to evaluate them, and test both variable declaration and evaluation in the REPL.
Steve live codes tests checking that variable declaration and evaluation are transpiled into JavaScript.
Steve explains that parser generators are domain specific languages for generating domain specific languages, and gives an overview of a few parser generators such as Antler and Chevrotain.
Wrapping Up
Section Duration: 3 minutes
Steve answers questions from the audience about Lisp, gives references to further learn about the topic, and talks about his path to understanding and mastering how to build a new programming language.
Learn Straight from the Experts Who Shape the Modern Web
- In-depth Courses
- Industry Leading Experts
- Learning Paths
- Live Interactive Workshops