Build an AI Agent from Scratch

Table of Contents

• 

  Introduction

  00:00:00 - 00:06:33 View Transcript

  Scott Moss introduces the course by discussing the prerequisites, which include knowledge of JavaScript and familiarity with APIs. He also provides technical setup instructions, such as installing Git and Node, and signing up for an AI API account.
• 

  AI Agents Overview

  00:06:34 - 00:15:17 View Transcript

  Scott provides an overview of the course project which is an agent that can read from Reddit, generate images, and provide random dad jokes. He also discusses different types of agents, such as conversational agents and transactional agents, and provides examples of their use cases.

• 

  LLMs Overview

  00:15:18 - 00:25:19 View Transcript

  Scott explains what a large language model (LLM) is and how it differs from other AI models. He also introduces the idea of transformers and tokenization and token limits in LLMs.
• 

  Types of LLMs Overview

  00:25:20 - 00:33:02 View Transcript

  Scott discusses different types of LLMs such as GPT, Claude, and Llama. He explains the concept of weights in AI models and how they are essential for the model's functionality. Scott also mentions the limitations of LLMs, including hallucinations and context window constraints.
• 

  OpenAI Hello World

  00:33:03 - 00:46:00 View Transcript

  Scott walks through setting up the environment for using OpenAI's API and demonstrates how to make a simple API call to generate a response. He also explains the parameters used in the API call, such as the model and temperature, and shows how to handle the response.

• 

  AI Chat with Memory Overview

  00:46:01 - 00:54:42 View Transcript

  Scott explains that one-off LLM calls are suitable for standalone tasks, while chat-based interactions are used for maintaining conversation history and context. Scott also mentions the importance of managing history and tokens in chat-based LLM interactions.
• 

  Chat Memory Message Types

  00:54:43 - 01:02:36 View Transcript

  Scott discusses memory in a chat application, which is a collection of previous messages in a conversation. He also covers the different types of messages, including system messages, user messages, assistant messages, and tool messages.
• 

  Memory & Token Limitations

  01:02:37 - 01:08:57 View Transcript

  Scott discusses the importance of memory in AI systems and how it enables follow-up capabilities and task continuity. He also explains the limitations of memory in language models and different strategies for managing memory constraints. Scott also mentions the concept of retrieval-augmented generation (RAG) and its potential to improve memory and context in AI systems.
• 

  Building an AI Chat with Memory

  01:08:58 - 01:28:47 View Transcript

  Scott explains how to implement a memory feature in a chat application using a file-based database. He demonstrates how to add and remove metadata from messages and create functions to add and get messages from the database. He also shows how to integrate the memory feature into the chat interface by retrieving previous messages and passing them to the LLM.

• 

  Types of AI Agents

  01:28:48 - 01:44:29 View Transcript

  Scott provides examples of different types of agents, including chat-based agents and task-based agents. He discusses the characteristics of agents, such as autonomous decision-making, task persistence, and goal orientation. He also addresses questions from students regarding validating LLM responses and the potential impact of larger AI models on specialized agents.
• 

  Building an AI Agent

  01:44:30 - 02:05:40 View Transcript

  Scott demonstrates how to add a user message to the database, show a loader, get all the messages, run the LLM with the messages and tools, and add the response to the chat messages. Scott also discusses how to handle tool calls and provides an example of creating a fake tool called "get weather" and passing it to the agent.
• 

  Building an AI Agent Q&A

  02:05:41 - 02:18:34 View Transcript

  Scott discusses the importance of sending messages in a certain order to prevent the AI from erroring. He also explains how to use the Zod function to describe the purpose of a tool and how to add reasoning to improve the AI's decision-making process. Scott also mentions the use of token counting to manage token limits and the option to set a maximum number of tokens for the response.

• 

  Function Calling Overview

  02:18:35 - 02:28:13 View Transcript

  Scott discusses how to handle security when a user is using a tool that may access other users' data. Scott also provides examples of clear and ambiguous matches in tool calling and emphasizes the importance of clear descriptions for tools.
• 

  Creating a Tool Runner

  02:28:14 - 02:49:23 View Transcript

  Scott demonstrates how to create a tool runner function that matches the tool call with the corresponding function and returns the result. He also shows how to save the tool response in the chat history and handle looping through tool calls.

• 

  Agent Loops Overview

  02:49:24 - 02:59:20 View Transcript

  Scott discusses agent loops and how they allow the agent to feed the answer back to the LLM so it can answer the original question and provide a response. Scott also provides examples of multi-step actions and how to handle dependent actions within the loop.
• 

  Coding the AI Agent Loop

  02:59:21 - 03:09:51 View Transcript

  Scott demonstrates how to use a while loop to continuously send messages to the model and receive responses until a specific condition is met. Scott also discusses the importance of saving and retrieving the conversation history within the loop.

• 

  Fetching Dad Jokes

  03:09:52 - 03:16:43 View Transcript

  Scott explains how to create a tool for generating a random dad joke. He demonstrates how to define the tool functions and their respective tool definitions using TypeScript and Zod.
• 

  Integrate Reddit API

  03:16:44 - 03:21:30 View Transcript

  Scott walks through how to create a tool that retrieves the latest posts from the NBA subreddit.
• 

  Generate Dahl-E Image

  03:21:31 - 03:29:28 View Transcript

  Scott demonstrates how to define the image generation tool, set up the prompts parameter, and guide the AI in generating the prompt. He also explains how to export the tool and integrate it into the tool runner.
• 

  Combining Tools into an Agent

  03:29:29 - 03:36:01 View Transcript

  Scott demonstrates how to combine the tools to generate a meme image from a random dad joke. He shows the process of inputting the prompt, retrieving the dad joke, generating the image, and receiving the final result. He also discuss the limitations of the tool and how it can be improved.
• 

  Add System Prompt to AI Agent

  03:36:02 - 03:45:23 View Transcript

  Scott demonstrates how to add instructions and context to the system prompt, such as providing rules for generating images and replacing celebrity names with generic characteristics. Scott also mentions the importance of including the current date in the system prompt and how to handle errors and optimize the prompts.

• 

  Improving Agents

  03:45:24 - 03:55:36 View Transcript

  Scott discusses the concept of RAG (Retrieval-Augmented Generation) and embeddings, which are numerical representations of text. Scott also touches on topics such as human-in-a-loop, evaluation strategies, safety mechanisms, integration patterns, and specialized agents.
• 

  Improving Agents Q&A

  03:55:37 - 04:06:49 View Transcript

  Scott answers student questions regarding recommended tools and resources, including Hugging Face and Brain Trust, and discusses the challenges and considerations when using AI services from providers like OpenAI, Azure, and AWS. The lesson concludes with a brief discussion on fine-tuning models and updating them as needed.

• 

  Wrapping Up

  04:06:50 - 04:07:53 View Transcript

  Scott wraps up the course by discussing how the advanced AI apps course builds on this course.