How BERT Learned to Read Your Inbox

This is part one of a two-part series. It shows the public model and code that Klar runs today. Part two will show how a BERT stack can change once it meets real mail and real hardware.

ChatGPT made one branch of the Transformer family familiar: give it a prompt and it writes. BERT took the reader branch. It sees a complete text, finds how each part relates to the rest, and turns that view into a vector. A small head uses the vector to pick a label such as spam.

BERT is the encoder half of a Transformer

The first Transformer had two parts. Its encoder read the input; its decoder wrote the output. BERT, short for Bidirectional Encoder Representations from Transformers, kept the reader stack. It was built to make sense of text, not to write the next token.

In each layer, every token can weigh every other token. “Transfer” means one thing in a bank note and another in a match report. Its vector shifts with the words around it. BERT reads the words on both the left and the right at once. A text generator sees only the text it has written so far.

Pre-training learns language. Task training gives it a job.

BERT first learned from text with no labels. Its task was to hide some tokens, then guess them from the words on both sides. It also tried to tell if one sentence came after another. This long first pass built a model that could read. A smaller set of tagged text could then teach it a job such as search, mood, or question answering.

Fine-tuning BERT can change more than the last layer. The first BERT recipe updates both the old weights and the new head. You can instead lock the base model and train just the head. That is cheap and safe for small updates, but it cannot change how the base model reads the text.

The public model Klar runs today

The model in use today is icosha/spam-xlmr-v1. It is built on XLM-RoBERTa-large and sorts a full piece of text. It has 24 layers and 16 attention heads. Each token ends up as a list of 1,024 values. Klar turns the reader into a quantized GGUF file, pulls out the four-way head, and runs both on the same device with llama.cpp and ggml. The code is under AGPLv3. The model has its own CC-BY-NC-4.0 licence.

RFC 822 email
  -> MIME parse and HTML-to-text
  -> normalized subject, sender and body
  -> canonical "User (email)" input
  -> XLM-RoBERTa-large encoder
  -> 1024-value CLS vector
  -> four-class head
  -> gibberish, marketing, regular or spam

Each arrow matters. If train time and run time shape the text in two ways, the head sees vectors it has never learned. The public engine avoids that split with one path for all input. It wraps the clean mail as User (email): ..., then adds the sender fields when they exist. Both scoring and local learning use this same form.

for (const auto& exchange : transcript) {
  std::string from_type = exchange.from_type;
  if (!from_type.empty()) {
    from_type[0] = static_cast<char>(std::toupper(from_type[0]));
  }
  input_text += from_type + " (" + exchange.origin + "): "
              + exchange.text + "\n";
}

if (has_any_customer_signal) {
  input_text += "Customer Info:\n";
  if (!customer.name.empty()) input_text += "Name: " + customer.name + "\n";
  if (!customer.email.empty()) input_text += "Email: " + customer.email + "\n";
}

Step 1: tokens become one sequence vector

The public GgmlEncoder uses the token rules stored in the GGUF file. It caps the text, keeps the end token, and makes sure XLM-RoBERTa’s <s> token is first. This is the token often called [CLS]. Once the reader has run, CLS pooling gives us one vector for the whole mail.

int n = llama_tokenize(vocab_, text.c_str(), text.size(),
                        token_buf_.data(), token_buf_.size(),
                        /* add_special = */ true,
                        /* parse_special = */ false);

if (n > max_tokens_) {
  n = max_tokens_;
  token_buf_[n - 1] = llama_vocab_eos(vocab_);
}

const llama_token bos = llama_vocab_bos(vocab_);
if (token_buf_.empty() || token_buf_.front() != bos) {
  token_buf_.insert(token_buf_.begin(), bos);
}

auto batch = llama_batch_get_one(token_buf_.data(), token_buf_.size());
llama_encode(ctx_, batch);

const float* emb = llama_get_embeddings_seq(ctx_, 0);
return std::vector<float>(emb, emb + n_embd_);

That first-token check guards against a real bug. One model tool wrote GGUF data that hid the token even when the code asked for it. Position zero then held the first plain word, not the whole-mail vector. The code still ran, but each vector meant the wrong thing. The model and the code that runs it must agree.

Step 2: the vector becomes class probabilities

The pooled vector goes through a small head: one dense layer, tanh, one output layer, then softmax. The head returns four scores: gibberish, marketing, regular mail and spam. This is where a broad text model learns the labels used for mail.

const auto logits = impl_->trainable_head->forward(
    embedding, cache_for_training);
const auto probabilities = impl_->trainable_head->softmax(logits);

return ClassScores{
    probabilities[0],  // gibberish
    probabilities[1],  // marketing
    probabilities[2],  // regular
    probabilities[3],  // spam
};

The token limit, the encoder and the classifier head each shape a different part of the model, so a change to one buys something the others cannot.

Local learning does not retrain all of BERT

A full tune of the base model needs a large, well-tested mail set and an offline run. A mark from one user is small and local. Klar first gets the vector from the locked base model. It saves the head’s first pass, sends the error back through that head, and takes one guarded step. Clipped updates, a pull toward the first weights and a hard range stop a run of one-sided marks from moving the head too far.

float SpamEngine::train_embedding(
    const std::vector<float>& embedding,
    int correct_label) {
  (void)classify_embedding(embedding, true);
  const float loss = impl_->trainable_head->backward(correct_label);
  impl_->trainable_head->step(1);
  return loss;
}

This split gives each part a clear job. The large frozen encoder provides general meaning across many languages. The small head learns where this owner draws the line. Both run on the device, but only the head changes after a local correction.

What BERT is good at, and what it is not

• Good: sorting, search and fact-finding when the whole input is there from the start.
• Good: small, fixed outputs with no need for a text bot or a trip to the cloud.
• Not good: free-form writing. A BERT reader has no part that can write a reply.
• Not enough on its own: proof of who sent the mail, trust and each user’s taste. Those facts live outside the words.

Not every AI task is a chat. Some of the best tools read once, give one small answer, and fade into the app around them. That is what BERT does in Klar. The full AGPLv3 code, including each sample above, is open at github.com/klar-im/engine.

Part two will ask what changes when we swap the base model, reshape the input and face new limits on the device.

All posts