Building Your Own Programming Language
Table of Contents
IntroductionSteve Kinney introduces the course, gives different use cases, and covers day to day examples of needing to build a programming language.
Compilers & Interpreters OverviewSteve defines what a compiler is, what the different stages of a compiler are, and what an interpreter is.
Course ProjectSteve introduces the course project, which is to create a scheme-based language called dropbear.
Parsing: Lexical Analysis
Lexing: Lexical AnalysisSteve defines both lexical analysis, or lexing, and syntactic analysis, which are both a form of parsing, and then dives into how lexing works.
Identity Helpers & LexerSteve explains why helpers are useful when thinking about building a programming language, and demonstrates how to loop over an array of tokens with a helper.
Lexing White Space & ParenthesisSteve gives a preview of the first exercise, of the repository, and dives into creating a token that breaks a string into an array of tokens.
Lexing Numbers & LettersSteve goes over the first exercise, live codes a test that looks for numbers and letters in order to tokenize them, and explains how lexers tokenize numbers.
Lexing Characters & StringsSteve moves on to the second exercise, and lives code a test that looks for characters and strings and tockenizes them.
Parsing: Syntactic Analysis
Steps to Build an ASTSteve demonstrates how to write an AST to make it readable for a babel-like compiler.
Parsing ExerciseAfter seeing Steve live code the parsing of numbers, the students are instructed to add functionalities to their parser so that it is able to parse both identifiers and strings.
Parsing SolutionSteve live codes the solution to the exercise.
REPL and CLI
REPLsSteve defines what a REPL is, gives different examples of existing REPLs, and introduces the next sections.
Adding the Standard LibrarySteve explains what are the basic functions put in place in the standard library, and mentions a helper included in the library.
Evaluating ExpressionsSteve starts live coding evaluating expressions and error messages that would give more explicit information to the developer when looking for an error.
Looking Up IdentifiersSteve explains how to write identifiers, and encourages the audience to write a max identifier of a method that finds the greatest value.
Adding to the Standard LibrarySteve goes over the solution of the basic evaluation, and succinctly reviews the different elements necessary to create a REPL.
Building the REPL CLISteve demonstrates how to build a REPL, and tries a few commands, such as add and subtract in the new created dropbear shell.
Running Files from the CLISteve demonstrates how to build a CLI, create a command that allows developers to access the dropbear shell, and create a dropbear file extension.
Traversing and Transforming the AST
Generation & Babel generateSteve explains what generation, or parsing in reverse is, and gives an overview of the babel generator.
Transformation: Visitor PatternSteve explains what transformation is, namely breaking apart the source code and being able to manipulate the AST, and explains what the Visitor Pattern is by giving the example of the babel transformation. The next step is to build a Visitor Pattern for the dropbear language.
Implementing TraverseSteve explains how to traverse and transform a tree, live codes the traverse function containing node, parent, and visitor arguments.
Testing TraverseSteve writes the traverse test and gives a brief overview of the traverse and transform exercise.
Transform PracticeSteve live codes the traversing and transforming, and builds a new visitor function that can traverse the AST. Visitors add different ways to process items, in the instance of dropbear, a visitor modifies a node after traversing the tree.
Transforming to Variable DeclarationsSteve adds the ability to create variables to dropbear. Namely, dropbear needs to have the ability to understand variable declaration, which means creating a variable node to the AST tree, add logic to the evaluator, and test everything in the REPL.
Variable EvaluationNow that variable declarations are defined, Steve explains that the next step is to be able to evaluate them, and test both variable declaration and evaluation in the REPL.
Parser GeneratorsSteve explains that parser generators are domain specific languages for generating domain specific languages, and gives an overview of a few parser generators such as Antler and Chevrotain.