Building Your Own Programming Language
Learning Paths:
Topics:
Table of Contents
Introduction
Introduction
Steve Kinney introduces the course, gives different use cases, and covers day to day examples of needing to build a programming language.
Compilers & Interpreters Overview
Steve defines what a compiler is, what the different stages of a compiler are, and what an interpreter is.
Course Project
Steve introduces the course project, which is to create a scheme-based language called dropbear.
Parsing: Lexical Analysis
Lexing: Lexical Analysis
Steve defines both lexical analysis, or lexing, and syntactic analysis, which are both a form of parsing, and then dives into how lexing works.
Identity Helpers & Lexer
Steve explains why helpers are useful when thinking about building a programming language, and demonstrates how to loop over an array of tokens with a helper.
Lexing White Space & Parenthesis
Steve gives a preview of the first exercise, of the repository, and dives into creating a token that breaks a string into an array of tokens.
Lexing Numbers & Letters
Steve goes over the first exercise, live codes a test that looks for numbers and letters in order to tokenize them, and explains how lexers tokenize numbers.
Lexing Characters & Strings
Steve moves on to the second exercise, and lives code a test that looks for characters and strings and tockenizes them.
Parsing: Syntactic Analysis
Internal Representations & ASTs
Steve explains that once types are broken into tokens, the abstract syntax tree (AST) breaks the code into a meaningful structure that gives the intent of what was just tokenized. AST makes it easier to transpile the language created into JavaScript.
Steps to Build an AST
Steve demonstrates how to write an AST to make it readable for a babel-like compiler.
Parsing Exercise
After seeing Steve live code the parsing of numbers, the students are instructed to add functionalities to their parser so that it is able to parse both identifiers and strings.
Parsing Solution
Steve live codes the solution to the exercise.
REPL and CLI
REPLs
Steve defines what a REPL is, gives different examples of existing REPLs, and introduces the next sections.
Adding the Standard Library
Steve explains what are the basic functions put in place in the standard library, and mentions a helper included in the library.
Evaluating Expressions
Steve starts live coding evaluating expressions and error messages that would give more explicit information to the developer when looking for an error.
Looking Up Identifiers
Steve explains how to write identifiers, and encourages the audience to write a max identifier of a method that finds the greatest value.
Adding to the Standard Library
Steve goes over the solution of the basic evaluation, and succinctly reviews the different elements necessary to create a REPL.
Building the REPL CLI
Steve demonstrates how to build a REPL, and tries a few commands, such as add and subtract in the new created dropbear shell.
Running Files from the CLI
Steve demonstrates how to build a CLI, create a command that allows developers to access the dropbear shell, and create a dropbear file extension.
Traversing and Transforming the AST
Generation & Babel generate
Steve explains what generation, or parsing in reverse is, and gives an overview of the babel generator.
Transformation: Visitor Pattern
Steve explains what transformation is, namely breaking apart the source code and being able to manipulate the AST, and explains what the Visitor Pattern is by giving the example of the babel transformation. The next step is to build a Visitor Pattern for the dropbear language.
Implementing Traverse
Steve explains how to traverse and transform a tree, live codes the traverse function containing node, parent, and visitor arguments.
Testing Traverse
Steve writes the traverse test and gives a brief overview of the traverse and transform exercise.
Transform Practice
Steve live codes the traversing and transforming, and builds a new visitor function that can traverse the AST. Visitors add different ways to process items, in the instance of dropbear, a visitor modifies a node after traversing the tree.
Transpiling to JavaScript
Steve live codes a babel visitor that transpiled code to JavaScript, and explains that generally transformation happens when entering a tree.
Language Features
Transforming to Variable Declarations
Steve adds the ability to create variables to dropbear. Namely, dropbear needs to have the ability to understand variable declaration, which means creating a variable node to the AST tree, add logic to the evaluator, and test everything in the REPL.
Variable Evaluation
Now that variable declarations are defined, Steve explains that the next step is to be able to evaluate them, and test both variable declaration and evaluation in the REPL.
Transpiling Variable Declarations to JavaScript
Steve live codes tests checking that variable declaration and evaluation are transpiled into JavaScript.
Parser Generators
Steve explains that parser generators are domain specific languages for generating domain specific languages, and gives an overview of a few parser generators such as Antler and Chevrotain.
