All you need is logistic regression

Logistic regression is a machine learning algorithm that predicts the probability of a categorical outcome. It passes a linear prediction through a sigmoid, producing a probability between 0 and 1. It takes one or more features in and produces a single output value.

Where

• $\sigma$ is the logistic function usually taking the form $\sigma(x) = (1 + e^{-x})^{-1}$
• $a$ and $b$ are learned coefficients
• $x$ is our input

This can be extended to any number of inputs features, each with a learned coefficient

It can also be generalized to multiple classes.

Note that the weights become a matrix $W$ and our output becomes a vector of probabilities for each class. Softmax is the generalization of the logistic function to multiple dimensions and ensures that the total probability for all classes sums to 1.

Building an LLM#

Logistic regression predicts the probability of an output. Text generation is just a categorical outcome prediction.

Let's use logistic regression to build an Logistic Language Model - because why not? (A more appropriate name as you'll see later might be a Lobotomized Language Model)

First we'll need a corpus of text. We'll use some Hemingway excerpts.

We can tokenize it into words and then we've got our vocabulary.

def tokenize(text):
    return re.findall(r'\w+', text.lower())
 
with open("hemingway.txt", "r") as file:
    content = file.read()
 
tokenized = tokenize(content)
vocabulary = sorted(set(tokenized))

We can turn these into ngrams - sequences of tokens. We'll create 3-grams and use the first 2 tokens to predict the next one.

def create_ngrams(input: list[str], n: int) -> list[list[str]]:
    output = []
    for i in range(len(input)-n+1):
        output.append(input[i:i+n])
    return output
 
ngrams = create_ngrams(tokenized, 3)

We'll create a dataset of token IDs for our n-grams.

def encode(tokens: list[str], vocabulary: list[str]) -> list[int]:
    return [vocabulary.index(token) for token in tokens]
    
ngrams_encoded = [encode(ngram, vocabulary) for ngram in ngrams]

Now we can produce our input data and target data

def one_hot(index: int, size: int) -> list[int]:
    return [1 if i == index else 0 for i in range(size)]
 
# Our input features are the first 2 encoded ngrams converted to one hot and concatenated together
X = np.array([
    np.concatenate([one_hot(index, len(vocabulary)) for index in ngram[:2]])
    for ngram in ngrams_encoded
])
 
# Our output is the last encoded ngram
y = np.array(ngrams_encoded)[:, -1]

And train a model

model = LogisticRegression(solver='lbfgs', C=0.3)
 
model.fit(X, y)

We can then predict text autoregressively.

def predict(text: str) -> str:
    X = tokens_to_features(tokenize(text)[-2:], vocabulary)
 
    logits = model.decision_function(csr_matrix([X]))[0]
 
    # Temperature - lower is more coherent, higher is increasingly drunk-sounding
    p = softmax(logits / 0.5)
 
    # Now we just sample randomly from the probabilities
    token = np.random.choice(len(p), p=p)
 
    return vocabulary[token]
 
sentence = "There is"
 
print(sentence, end=" ")
 
for i in range(100):
    prediction = predict(sentence)
    print(prediction, end=" ")
    sentence = sentence + " " + prediction

Now we’ve successfully created a language model. Go and put "LLM engineer" on your CV without elaborating, sit back, and enjoy its beautiful prose.

There is no one. I dont it. He had never written them. The ground. His father himself of the world on the bed. His wife. On the men and the other was. The wouldnt of the world. It was a good. 5 she said. I do you feel the other. He was a coward. The lion was not to go. You have a look at the table for the money. He was a very on the bed of his life and francis macomber now.

Epilogue#

I'm obviously not suggesting doing this for real. What's interesting is that if you look at the very last layer of a real LLM, there is what looks very similar to a logistic regression hiding there.

The head of a trillion parameter frontier transformer model is still just a multiclass logistic regression.

In fact, we've built a reasonable chunk of a traditional LLM scaffold and prediction pipeline.

You might think the biggest bottleneck we have is that we're only looking at 3 tokens at a time. That by itself isn’t actually the problem here. We could just turn that number up - we could push it to 16, 32, 128 tokens or further.

Notice how we built our $X$ in the model. We concatenated together the encodings (one-hot) of each token. This means that our model is completely linear right up until the softmax. There's no interaction between the terms. The model has no way to understand that New and York together mean something.

The part of an LLM that makes it powerful is the part we’ve omitted. The power of a transformer is in building up an internal representation of the text.

All that power exists just to feed a really, really, good feature vector to a logistic regression.