Scott Moss
Superfilter AI
Build an AI Agent from Scratch
4 hours, 7 minutes CC
Course Description
Create your own AI agent! Build a command line messaging interface and an agent loop to continuously interact with the user. Code tools from scratch for retrieving information and generating images from Dahl-E. Manage the chat history to provide context with each prompt. Learn the bleeding-edge design patterns and resources for building modern agent-based API apps.
This course and others like it are available as part of our Frontend Masters video subscription.
Preview
CloseLearn Straight from the Experts Who Shape the Modern Web
Your Path to Senior Developer and Beyond
- 200+ In-depth courses
- 18 Learning Paths
- Industry Leading Experts
- Live Interactive Workshops
Table of Contents
Introduction
Section Duration: 15 minutes
Scott Moss introduces the course by discussing the prerequisites, which include knowledge of JavaScript and familiarity with APIs. He also provides technical setup instructions, such as installing Git and Node, and signing up for an AI API account. - Course Website: https://clumsy-humor-894.notion.site/Agent-from-scratch-13554fed51a380749554c44aa8989406 - Course Repo: https://github.com/Hendrixer/agent-from-scratch
Scott provides an overview of the course project which is an agent that can read from Reddit, generate images, and provide random dad jokes. He also discusses different types of agents, such as conversational agents and transactional agents, and provides examples of their use cases.
LLM Overview
Section Duration: 30 minutes
Scott explains what a large language model (LLM) is and how it differs from other AI models. He also introduces the idea of transformers and tokenization and token limits in LLMs.
Scott discusses different types of LLMs such as GPT, Claude, and Llama. He explains the concept of weights in AI models and how they are essential for the model's functionality. Scott also mentions the limitations of LLMs, including hallucinations and context window constraints.
Scott walks through setting up the environment for using OpenAI's API and demonstrates how to make a simple API call to generate a response. He also explains the parameters used in the API call, such as the model and temperature, and shows how to handle the response.
Memory
Section Duration: 42 minutes
Scott explains that one-off LLM calls are suitable for standalone tasks, while chat-based interactions are used for maintaining conversation history and context. Scott also mentions the importance of managing history and tokens in chat-based LLM interactions.
Scott discusses memory in a chat application, which is a collection of previous messages in a conversation. He also covers the different types of messages, including system messages, user messages, assistant messages, and tool messages.
Scott discusses the importance of memory in AI systems and how it enables follow-up capabilities and task continuity. He also explains the limitations of memory in language models and different strategies for managing memory constraints. Scott also mentions the concept of retrieval-augmented generation (RAG) and its potential to improve memory and context in AI systems.
Scott explains how to implement a memory feature in a chat application using a file-based database. He demonstrates how to add and remove metadata from messages and create functions to add and get messages from the database. He also shows how to integrate the memory feature into the chat interface by retrieving previous messages and passing them to the LLM.
AI Agents
Section Duration: 49 minutes
Scott provides examples of different types of agents, including chat-based agents and task-based agents. He discusses the characteristics of agents, such as autonomous decision-making, task persistence, and goal orientation. He also addresses questions from students regarding validating LLM responses and the potential impact of larger AI models on specialized agents.
Scott demonstrates how to add a user message to the database, show a loader, get all the messages, run the LLM with the messages and tools, and add the response to the chat messages. Scott also discusses how to handle tool calls and provides an example of creating a fake tool called "get weather" and passing it to the agent.
Scott discusses the importance of sending messages in a certain order to prevent the AI from erroring. He also explains how to use the Zod function to describe the purpose of a tool and how to add reasoning to improve the AI's decision-making process. Scott also mentions the use of token counting to manage token limits and the option to set a maximum number of tokens for the response.
Function Calling
Section Duration: 30 minutes
Scott discusses how to handle security when a user is using a tool that may access other users' data. Scott also provides examples of clear and ambiguous matches in tool calling and emphasizes the importance of clear descriptions for tools.
Scott demonstrates how to create a tool runner function that matches the tool call with the corresponding function and returns the result. He also shows how to save the tool response in the chat history and handle looping through tool calls.
Agent Loops
Section Duration: 20 minutes
Scott discusses agent loops and how they allow the agent to feed the answer back to the LLM so it can answer the original question and provide a response. Scott also provides examples of multi-step actions and how to handle dependent actions within the loop.
Scott demonstrates how to use a while loop to continuously send messages to the model and receive responses until a specific condition is met. Scott also discusses the importance of saving and retrieving the conversation history within the loop.
Real Tools
Section Duration: 35 minutes
Scott explains how to create a tool for generating a random dad joke. He demonstrates how to define the tool functions and their respective tool definitions using TypeScript and Zod.
Scott walks through how to create a tool that retrieves the latest posts from the NBA subreddit.
Scott demonstrates how to define the image generation tool, set up the prompts parameter, and guide the AI in generating the prompt. He also explains how to export the tool and integrate it into the tool runner.
Scott demonstrates how to combine the tools to generate a meme image from a random dad joke. He shows the process of inputting the prompt, retrieving the dad joke, generating the image, and receiving the final result. He also discuss the limitations of the tool and how it can be improved.
Scott demonstrates how to add instructions and context to the system prompt, such as providing rules for generating images and replacing celebrity names with generic characteristics. Scott also mentions the importance of including the current date in the system prompt and how to handle errors and optimize the prompts.
Next Level
Section Duration: 21 minutes
Scott discusses the concept of RAG (Retrieval-Augmented Generation) and embeddings, which are numerical representations of text. Scott also touches on topics such as human-in-a-loop, evaluation strategies, safety mechanisms, integration patterns, and specialized agents.
Scott answers student questions regarding recommended tools and resources, including Hugging Face and Brain Trust, and discusses the challenges and considerations when using AI services from providers like OpenAI, Azure, and AWS. The lesson concludes with a brief discussion on fine-tuning models and updating them as needed.
Wrapping Up
Section Duration: 1 minute
Scott wraps up the course by discussing how the advanced AI apps course builds on this course.
Learn Straight from the Experts Who Shape the Modern Web
- In-depth Courses
- Industry Leading Experts
- Learning Paths
- Live Interactive Workshops