13.1 LLMs

Q&A

Can a logistic regression model predict more than two classes?

• Multinomial logistic regression: predict a score for each category, exponentiate all scores, and normalize them to sum to 1.
• Same as binary logistic regression if there are only two categories. (The score of the negative category is 0.)

What was the purpose of us trying to predict iPhone vs Android?

• A classification model can make a perfect prediction on training data by using irrelevant features, but that won’t generalize.
• We can measure generalization by leaving out different parts of the data.

What is the difference between a linear regression model and a logistic regression model?

• Linear regression: predict a continuous value.
• Logistic regression: predict a probability.

LLMs

Objectives

• Explain what problem a language model is designed to solve.
• Explain how instruction tuning and human feedback alter how language models work.
• Use an API to generate text from a language model.

Language Modeling

• Input: a sequence of words.
• Output: a probability distribution over the next word.

How to generate

• Start with a prompt.
• Generate a probability distribution over the next word.
• Sample a word from that distribution.
• Append the word to the prompt.
• Repeat.

Example

Prompt:

def square(x):
    """Return the square of x."""

Probabilities for next word:

• return: 98.55%
• #: 0.57%
• squared: 0.16%
• y: 0.14%
• print: 0.11%

Training

• Trained to minimize surprise on Internet text. (i.e., maximize the probability given to the actual next word.)
• Hundreds of GB of text: huge diversity of human expression
  • Facts (Wikipedia, WikiHow, “how to clean tomato sauce stains” discussions, etc.)
  • Conversations (discussion forums, Reddit, Twitter, StackOverflow, etc.)
  • News and opinions from a wide range of perspectives
  • Fiction (fan fiction, etc.)
  • Code (GitHub, tech blogs, etc.)
  • and every kind of stereotyping and prejudice you can imagine

Tuning and Human Feedback

Problem: not everything on the Internet is helpful.

Example: when I generated from the prompt above, the model generated the return statement, and kept going:

f = square
print(square)
print(f)

§ Output

> stdout : ['<function square at 0x7fcb879b52f0>\n', '<function square at 0x7fcb879b52f0>\n']


§ Code

def fahrenheit(T):
    return (9.0/5)*T + 32

def celsius(T):
    return (5.0/9)*(T-32)

temp = [0, 22.5, 40, 100]

F_temps = list(map(fahrenheit, temp))

C_temps = list(map(celsius, F_temps))

print(F_temps)
print(C_temps)

Tuning and Human Feedback

• That’s a reasonable guess of what follows on the Internet after a function definiton like that
  • a function like that probably often occurs in an example where someone is demonstrating functions-as-objects or soomething.
• But in a code-assistant context, it’s not helpful.

Fine Tuning

• Start with a model trained on Internet text.
• Continue training on a specialized dataset
  • Instruction Fine-Tuning: curated examples of the behavior we want the model to have
  • Learning from Feedback: let the model generate several options, ask people which one is best, and train the model to prefer the one people like best

Challenges with LLMs

Hallucination and Plausibility

Shortcuts human thinking and empathy.

Reasoning

Fairness, Bias and Stereotyping

Gratitude

Prompting

Speed