week1/notes/day4.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Revealing the Leadership Winner: A Fun LLM Challenge\n",
    "\n",
    "### Summary\n",
    "\n",
    "This section provides an overview of the rapid advancements in LLMs, tracing their development from the invention of the Transformer architecture to the latest models. It also discusses the ongoing debate about the nature of LLM intelligence, highlighting the concept of emergent intelligence.\n",
    "\n",
    "### Highlights\n",
    "\n",
    "- 🧠 **Transformer Architecture:**\n",
    "    - The Transformer architecture, introduced in 2017, revolutionized LLM development with its self-attention layers.\n",
    "- 📈 **Rapid LLM Evolution:**\n",
    "    - The progression from GPT-1 to GPT-4o, including the transformative impact of ChatGPT, demonstrates the rapid pace of LLM development.\n",
    "- 🗣️ **Stochastic Parrot Debate:**\n",
    "    - Early skepticism questioned LLMs' understanding, likening their responses to statistical pattern matching.\n",
    "- 💡 **Emergent Intelligence:**\n",
    "    - The current perspective acknowledges LLMs' statistical prediction but recognizes the emergence of apparent intelligence at massive scales.\n",
    "    - The concept of emergent intelligence is that through massive scale, the models appear to be intelligent, even though they are just predicting the next token.\n",
    "- ❓ **Ongoing Debate:**\n",
    "    - The nature of LLM intelligence remains a topic of discussion, with varying viewpoints on whether they truly understand or merely imitate understanding.\n",
    "- 🎮 **Outsmart Game:**\n",
    "    - The speaker has created a game called \"Outsmart\" that pits various models against each other.\n",
    "- 📰 **Attention is All You Need:**\n",
    "    - The paper that introduced the transformer model is mentioned.\n",
    "\n",
    "# Exploring the Journey of AI: From Early Models to Transformers\n",
    "\n",
    "### Summary\n",
    "\n",
    "This section discusses several phenomena in the AI field, including the rise and fall of prompt engineering, the popularity of custom GPTs, the emergence of co-pilots, and the current trend of agentic AI.\n",
    "\n",
    "### Highlights\n",
    "\n",
    "- 📉 **Prompt Engineering's Fluctuations:**\n",
    "    - The demand for specialized prompt engineers has decreased due to widespread knowledge and automated tools.\n",
    "- 🛍️ **Custom GPTs' Saturation:**\n",
    "    - The GPT store, while initially popular, has become somewhat saturated, although it remains a resource for experimenting with tuned GPTs.\n",
    "- 🤝 **Co-Pilots' Emergence:**\n",
    "    - Co-pilots, like Microsoft Copilot and GitHub Copilot, have become integral to collaborative human-AI workflows.\n",
    "- 🤖 **Agentic AI's Rise:**\n",
    "    - Agentic AI, involving multiple collaborating LLMs, is the current trend, focusing on task decomposition, memory, and autonomy.\n",
    "    - Agentic AI breaks down complex problems into smaller tasks.\n",
    "    - Agentic AI has a form of memory.\n",
    "    - Agentic AI has a form of autonomy.\n",
    "- 🛠️ **Future Agentic AI Project:**\n",
    "    - The course will culminate in building a seven-agent AI solution, demonstrating the practical application of agentic AI.\n",
    "\n",
    "# Understanding LLM Parameters: From GPT-1 to Trillion-Weight Models\n",
    "\n",
    "### Summary\n",
    "\n",
    "This section focuses on the concept of parameters (weights) in LLMs, highlighting their crucial role in controlling model outputs and the exponential growth in their numbers over successive LLM generations.\n",
    "\n",
    "### Highlights\n",
    "\n",
    "- ⚖️ **Parameters/Weights Defined:**\n",
    "    - Parameters, also known as weights, are the adjustable levers within an LLM that determine its output based on input.\n",
    "    - These weights are set during training as the model learns to predict the next token.\n",
    "- 📈 **Exponential Growth:**\n",
    "    - The number of parameters in LLMs has grown exponentially, from GPT-1's 117 million to potentially 10 trillion in the latest frontier models.\n",
    "    - This growth is illustrated using a logarithmic scale, emphasizing the massive increase.\n",
    "- 🤯 **Scale Comparison:**\n",
    "    - Traditional machine learning models, like linear regression, typically have far fewer parameters (tens to hundreds).\n",
    "    - The sheer scale of LLM parameters is almost incomprehensible, highlighting the complexity of these models.\n",
    "- 📊 **Model Examples:**\n",
    "    - GPT series: GPT-1 (117M), GPT-2 (1.5B), GPT-3 (175B), GPT-4 (1.76T).\n",
    "    - Open-source models: Gemma (2B), Llama 3 (8B, 70B, 405B), Mixtral (mixture of experts).\n",
    "- 🧠 **Impact of Parameters:**\n",
    "    - The vast number of parameters enables LLMs to capture complex patterns and generate sophisticated outputs.\n",
    "    - The more parameters a model has, the more complex it generally is.\n",
    "\n",
    "# GPT Tokenization Explained: How Large Language Models Process Text Input\n",
    "\n",
    "### Summary\n",
    "\n",
    "This section explains the concept of tokens in LLMs, detailing how text is broken down into these units for processing. It contrasts tokenization with earlier methods, such as character-by-character or word-by-word approaches, and provides examples using the OpenAI tokenizer to illustrate the process.\n",
    "\n",
    "### Highlights\n",
    "\n",
    "- **Token Definition:** Tokens are the individual units of text that are processed by LLMs.\n",
    "- **Evolution of Tokenization:**\n",
    "    - Early neural networks used character-by-character processing.\n",
    "    - Later models used word-by-word processing.\n",
    "    - Current LLMs use a hybrid approach, breaking text into chunks that can be whole words, parts of words, or sub-word units.\n",
    "- **Benefits of Tokenization:**\n",
    "    - Handles proper nouns and place names effectively.\n",
    "    - Captures word stems and the underlying meaning of words.\n",
    "    - Balances vocab size and model complexity.\n",
    "- **OpenAI Tokenizer:**\n",
    "    - The OpenAI tokenizer tool ([platform.openai.com/tokenizer](https://www.google.com/url?sa=E&source=gmail&q=https://platform.openai.com/tokenizer&authuser=7)) allows users to visualize how text is tokenized.\n",
    "    - Common words are often mapped to single tokens.\n",
    "    - Rare words are broken into multiple tokens.\n",
    "- **Tokenization Examples:**\n",
    "    - Examples demonstrate how words, phrases, and numbers are tokenized.\n",
    "    - The examples illustrate how word stems and sub-word units are captured.\n",
    "- **Token-to-Word Ratio:**\n",
    "    - A general rule of thumb is that one token is roughly equivalent to 0.75 words.\n",
    "    - Approximately 1,000 tokens correspond to about 750 words.\n",
    "- **Example: Shakespeare's Works:**\n",
    "    - The complete works of Shakespeare contain approximately 900,000 words, which translates to about 1.2 million tokens.\n",
    "- **Context Dependence:**\n",
    "    - Token counts can vary depending on the type of text (e.g., math formulas, code) and the specific tokenizer used.\n",
    "    - Different LLMs may employ different tokenization strategies.\n",
    "- **Key Takeaway:**\n",
    "    - Tokenization is a crucial process in LLM input processing, balancing vocabulary size and semantic representation.\n",
    "\n",
    "# How Context Windows Impact AI Language Models: Token Limits Explained\n",
    "\n",
    "### Summary\n",
    "\n",
    "This section explains the concept of the context window in LLMs, clarifying its role in determining how much information an LLM can consider when generating the next token. It emphasizes that the context window includes not just the current input but also the entire conversation history.\n",
    "\n",
    "### Highlights\n",
    "\n",
    "- **Context Window Definition:** The context window refers to the total number of tokens an LLM can consider when generating the next token.\n",
    "- **LLM's Task:** An LLM's primary task is to predict the most likely next token based on the input it receives.\n",
    "- **Conversation Illusion:** LLMs like ChatGPT appear to maintain context in conversations, but this is achieved by passing the entire conversation history as input with each new prompt.\n",
    "- **Context Window Components:** The context window includes the system prompt, user prompts (inputs), and the LLM's responses.\n",
    "- **Context Window Dynamics:**\n",
    "    - At the start of a conversation, the context window only needs to fit the initial prompt.\n",
    "    - As the conversation progresses, the context window needs to accommodate the growing history of inputs and outputs.\n",
    "- **Importance for Long Inputs:** To ask a question about a large text, like the complete works of Shakespeare, the entire text must fit within the context window.\n",
    "- **In Essence:** The context window is the total amount of information the LLM can see at once.\n",
    "\n",
    "# Navigating AI Model Costs: API Pricing vs. Chat Interface Subscriptions\n",
    "\n",
    "### Summary\n",
    "\n",
    "This section explains the cost structure of LLM APIs, contrasting it with subscription-based chat interfaces. It also addresses the initial credit requirement for using these APIs and provides options for using local models to avoid API costs.\n",
    "\n",
    "### Highlights\n",
    "\n",
    "- 💰 **API Cost Structure:**\n",
    "    - LLM APIs charge per call, based on the model used, input tokens, and output tokens.\n",
    "    - Input tokens have a lower cost than output tokens.\n",
    "    - While individual call costs are low, they can accumulate in high-volume applications.\n",
    "- 💳 **Initial Credit Requirement:**\n",
    "    - Platforms like OpenAI and Claude require an initial credit deposit (e.g., $5) to use their APIs.\n",
    "    - The speaker assures that the course activities will not exhaust this initial credit.\n",
    "    - This initial credit is a great investment for learning and experimentation.\n",
    "- 💻 **Local Model Alternative:**\n",
    "    - Users can use local models like Llama to avoid API costs.\n",
    "    - The course provides exercises to familiarize users with local model usage.\n",
    "- 📈 **Scalability Considerations:**\n",
    "    - For building scalable systems, API costs must be carefully monitored.\n",
    "    - The speaker is going to show example cost and context window information.\n",
    "\n",
    "# Comparing LLM Context Windows: GPT-4 vs Claude vs Gemini 1.5 Flash\n",
    "\n",
    "### Summary\n",
    "\n",
    "This section discusses the context window sizes and API costs of various frontier LLMs, using data from Vellum's LLM leaderboard. It emphasizes the importance of understanding these costs for practical applications and provides context for how to interpret the provided figures.\n",
    "\n",
    "### Highlights\n",
    "\n",
    "- 📊 **Vellum's LLM Leaderboard:**\n",
    "    - Vellum provides a valuable resource for comparing LLM capabilities, including context window sizes and API costs.\n",
    "    - It is a good resource to bookmark for future LLM comparisons.\n",
    "- 📏 **Context Window Sizes:**\n",
    "    - Gemini 1.5 Flash has the largest context window, at 1 million tokens.\n",
    "    - Claude models have a 200,000-token context window.\n",
    "    - GPT models typically have a 128,000-token context window.\n",
    "    - The context window must contain all of the previous conversation, and the current prompt.\n",
    "- 💰 **API Costs:**\n",
    "    - Costs are presented per million tokens, not per token.\n",
    "    - Claude 3.5 Sonnet costs $3 per million input tokens and $15 per million output tokens.\n",
    "    - GPT-4 Mini costs $0.15 per million input tokens and $0.60 per million output tokens.\n",
    "    - For typical short queries, costs are usually less than a cent.\n",
    "    - The total cost is the sum of input token cost, and output token cost.\n",
    "- 📈 **Cost Considerations:**\n",
    "    - While costs are generally low for individual queries, they can accumulate in high-volume applications.\n",
    "    - Users can specify a maximum number of output tokens to control costs.\n",
    "- 📌 **Practical Implications:**\n",
    "    - Understanding context window sizes and API costs is crucial for choosing the right LLM for specific tasks.\n",
    "    - The speaker stresses that these costs are very reasonable considering the computing power required to run the models.\n",
    "\n",
    "# Wrapping Up Day 4: Key Takeaways and Practical Insights\n",
    "\n",
    "### Summary\n",
    "\n",
    "This section recaps the key learnings from day four, emphasizing the understanding of tokens, context windows, and API costs. It also previews the next lecture, which will involve practical coding exercises to build a business solution using the OpenAI API.\n",
    "\n",
    "### Highlights\n",
    "\n",
    "- **Recap of Day Four:**\n",
    "    - The day covered essential concepts like tokens, tokenization, context windows, and API costs.\n",
    "    - Participants should now be able to confidently use these concepts in practical applications.\n",
    "    - The difference between the chat interface cost and the API cost was made clear.\n",
    "- **Understanding LLM Limitations:**\n",
    "    - The challenge of counting letters in a tokenized text was discussed, highlighting the importance of understanding how LLMs process information.\n",
    "    - The reason why some models were able to answer the \"how many A's\" question was explained.\n",
    "- **Practical Skills:**\n",
    "    - Participants should now be proficient in writing code to call the OpenAI API and local models like Llama.\n",
    "    - They can effectively compare and contrast different frontier LLMs.\n",
    "- **Preview of Next Lecture:**\n",
    "    - The next lecture will focus on hands-on coding exercises to implement a business solution using the OpenAI API.\n",
    "    - The lab will involve multiple calls to LLMs and will conclude with an exercise for participants.\n",
    "    - The next lecture will build coding confidence."
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}