Throughline · machine learning notes
Teaching a small model how I file my thoughts
Part 1: the ML basics, the decisions, the bugs, and a first map, all before training anything. (Which turned out to be most of the work.)
Throughline is a voice-first app I built for my own brain. You talk, it turns the mess into threads, and it files them into projects so you don't have to. After a few months I have 218 of these little threads, and I wanted to see them. Not as a list. As a map. The kind of second-brain graph people build by hand in Obsidian and Notion, except one that grows by itself while you're busy living.
Maps like that run on an embedding model. Off-the-shelf ones are good at general meaning, but they don't know that, for me, "call about the soil test" and "order more mulch" belong together and "call about the invoice" doesn't. So I wanted to fine-tune one: take a small open-source model and teach it how I group my thoughts.
I didn't just want the result, though. I wanted to actually learn how this works: the vocabulary, the decisions, the ways it quietly goes wrong. So I paired with Claude as a tutor and a pair programmer. It wrote a lot of the Python and explained every concept as we hit it, and I made the calls (which models, which data, which loss, what counts as "better") and had to be able to defend each one. Every decision below has the options we weighed and why I chose what I chose.
The image I kept coming back to is a trellis. A pretrained model is a healthy plant that already grows; it learned language from millions of sentences. Fine-tuning doesn't grow a new plant. It's tying the vine to a frame so it grows where you need it.
ML basics in two minutes
- embedding
- A list of numbers (a vector; ours have 384) that a model produces for a piece of text. Texts with similar meanings get vectors that point in similar directions. It's a map with 384 dimensions instead of 2.
- cosine similarity
- How we measure "close" on that map: the angle between two vectors. 1 means same direction (same meaning), 0 means unrelated. Every similarity score in this post is a cosine.
- pretrained vs. fine-tuned
- A pretrained model learned general language from a huge public dataset. Fine-tuning keeps training it on a small, specific dataset (here: which thoughts belong in the same project) so its map bends toward your notion of similar.
- train / test split
- You teach the model on one pile of examples and grade it on a separate pile it has never seen. The grade on the unseen pile is the only number that tells you whether it learned the pattern or just memorised the answers.
- overfitting
- Memorising instead of learning. Training scores go up while test scores go down. With a small dataset like mine this is the main danger.
- baseline
- The score to beat. Without one, "it got 70%!" means nothing: 70% compared to what?
The stack
Everything runs on my laptop (an M4 Pro MacBook). No cloud GPUs, and none of the data, vectors or trained weights leave the machine.
| tool | what it does here |
|---|---|
| Python 3.12 + uv | The language most ML work happens in. uv manages the environment and pins exact package versions so results are reproducible. (I first tried Python 3.14; PyTorch didn't have builds for it yet. Lesson: ML moves fast but its dependencies don't.) |
| PyTorch | The deep-learning engine that does the actual maths. It runs on Apple Silicon's GPU through MPS (Metal Performance Shaders), so training a small model takes minutes, not hours. |
| sentence-transformers | The standard library for embedding models: load one, encode text, fine-tune with a chosen loss. |
| Hugging Face Hub | The "GitHub of models". The open models I'm comparing download from here. Some, like Google's EmbeddingGemma, are gated: you accept the licence before you can download. |
| scikit-learn | Classic ML toolkit. Here it does the grouped train/test splits. |
| UMAP + HDBSCAN | Squashing 384 dimensions down to something drawable, and finding the topic clusters. Both are in the map section below. |
1. Decide what "better" means first
If you fine-tune without a number to beat, you're just vibing. So before touching a model I picked a metric: kNN project accuracy. kNN stands for k-nearest neighbours. Hide a thread's project, find the 5 most similar threads I've already filed, let them vote (closer neighbours get a bigger vote), and check whether the winner matches what I actually chose.
Then I sabotaged my own metric on purpose: shuffle the project labels randomly and run it again. A healthy metric should collapse to the score of a dumb guess, because there's no longer any real pattern to find. It did (9 to 11% on my threads, against a 16.5% dumb-guess floor). If it hadn't, the next section is the reason.
Leakage, the quiet killer
Leakage is when information from the test pile sneaks into what the model learns from, so the grade measures memory instead of understanding. It's the most common way ML results turn out to be fake, and it almost never throws an error. It just makes your numbers look great.
Here's the version that hit me. When I talk to Throughline, one rambling voice note (a dump) gets split into several threads. All of those threads carry the same original transcript. Say one dump about the garden became three threads:
If "plant garlic" is in the test pile and its two siblings are in the training pile, they're unusually easy neighbours: same moment, same train of thought, and usually (75 to 87% of the time) the same project. But that situation never happens in the real app. When a dump gets filed, all of its threads are filed at the same moment, so none of them has a project yet to vote with. The test has to look like real life, so the fix is a grouped split: every thread from one dump goes to the same side, always.
My first explanation for this was wrong, and I only found out because I pushed back on it. I'd been told the siblings were easy because they share transcript text, so the model was "recognising a copy". But a dump isn't always one topic (I ramble), and if shared text were the cause, feeding in titles only (no shared text at all) would show no inflation. So we measured it. Splitting by thread inflated title-only scores too, by 4 to 11 points. The rule survived; the reason changed. It's about independence, not copying: siblings are correlated, and real filing never gets to see them. That challenge ended up shaping the map more than anything else in this post.
The leaks I checked for (and one I haven't closed yet)
- Duplicates. The same item appears twice, once in each pile. My server data had up to 25 copies of one capture (more on that below). Fix: deduplicate before splitting.
- Groups. Different items that share something the model can latch onto: siblings from one dump, or the habits of one person. Fix: split by dump when testing my own data, and by whole person when training on everyone's, so the test always measures a stranger.
- Time. In real life, when a new thought arrives, the app only has your earlier thoughts to compare against. My current test lets a thread vote with neighbours from the future. That's a small leak, and it's on the list for part 2: an "only the past" version of the metric.
- Not leakage: using the transcript as input. A thought's transcript genuinely exists at the moment it's filed, so it's fair game. The test for "is this leakage?" is always would the model have this information in real use?
2. My data alone is too small
134 of my 218 threads are filed into a project. For training, that's tiny. Throughline also keeps a de-identified training set from people who haven't opted out: names, places and organisations are replaced with tokens like <PERSON_1>, so a thought reads "call <PERSON_1> about the deck".
The raw count lied to me, though:
The same capture gets logged several times as it moves through the app (captured, then corrected, then accepted), up to 25 times for one dump. Without deduplicating, the model would over-learn a handful of thoughts, and copies would land on both sides of the test. That's the duplicate leak from above, caught before it cost anything.
One more thing the data taught me: labels are per person. My "Home" is not your "Home". So a thread is only ever judged by votes from its own person's other threads, which is also exactly how the real app would work: a general model, plus your own history as the labels.
3. Measure the stock models
You can only say fine-tuning helped if you measured where you started. I picked three small open models, plus one big one for curiosity:
Each model was fed either just the thread title or the title plus the original transcript (more context, longer input):
| model · input | my threads | 41 other people |
|---|---|---|
| always guess your biggest project | 16.5% | 59.0% |
| MiniLM · title | 48.1% | 70.0% |
| MiniLM · title + transcript | 54.9% | 73.6% |
| bge-small · title | 45.1% | 74.0% |
| bge-small · title + transcript | 51.9% | 75.3% |
| gte-small · title | 46.6% | 74.9% |
| gte-small · title + transcript | 51.1% | 77.1% |
| EmbeddingGemma (300M) · title | 42.1% | 67.4% |
| EmbeddingGemma (300M) · title + transcript | 39.8% | 70.5% |
How I read it:
- The three small models are tied. Every gap between them is inside the noise. I'm not putting "gte beat bge" on anything.
- The transcript helps me more than everyone else. On my threads it adds 5 to 7 points for every model. On other people's it adds only 1 to 4, which is inside the noise. So: real for me, "maybe" for everyone else. (And not leakage either way, because we split by dump.) I almost wrote "it helps everywhere" before checking the numbers against the noise, which is exactly the habit this section is about.
- Judge the lift, not the raw score. Most people have one dominant project, so guessing it already gets 59%. The models add about 15 to 18 points over that for other people and about 35 for me (I have 15 projects, apparently I like folders).
The big model lost
EmbeddingGemma is ten times bigger than the others, newer, and near the top of the public leaderboards. On my thoughts it came last, on both datasets.
It also taught me something about instruction-tuned models. EmbeddingGemma expects a task label in front of the text (it ships fourteen: clustering, classification, similarity, search and so on), and the vectors change depending on which one you pick. I picked "clustering", because a map is clustering. Then, out of curiosity, I tried three others:
| EmbeddingGemma prompt | my threads | 41 other people |
|---|---|---|
| Clustering (my pick) | 39.8% | 70.5% |
| no prompt | 45.9% | 71.4% |
| Classification | 45.9% | 70.5% |
| STS (sentence similarity) | 51.1% | 70.0% |
The label alone moved my score by 11 points, more than the whole gap between the small models. With these models, the instruction is a hyperparameter (a setting you tune and test like any other), not a detail.
And then I caught myself. I'd just picked a winner by looking at the test scores. If I now report "EmbeddingGemma gets 51%", that number is flattering, because I used the exam to choose the answer.
The 0% bug
My favourite moment so far. The "always guess your biggest project" floor came back as 0.0% on my data, which can't be right. Even a broken clock guesses something.
My two biggest projects are tied at 22 threads each. To score a thread fairly, the code hid it and took the biggest project among everything else. Hide a Throughline thread, and the remaining vote is Home 22, Throughline 21, so it guesses Home. Wrong. Hide a Home thread and it flips. Wrong again, every single time.
Small bug, but it's the kind that quietly makes every result look better or worse than it is. Every number in this post is only as honest as its baseline. I'm glad this one showed up as an impossible number and not a plausible one.
4. More context isn't free
Back in step 3, the transcript helped my filing by 5 to 7 points. The obvious next move was to feed it into the map too. My dump challenge made me check something first: what does the transcript do to a dump that holds several topics? A quarter of my multi-thread dumps mix projects ("pay the invoice, oh and plant the garlic").
With the whole transcript attached, the paragraph of shared words drowns out the five-word title, and every thread from a dump becomes a near-copy of its siblings, whatever it's about. On filing that still looked like a win, because siblings are usually in the same project anyway. The model wasn't learning topics. It was learning "same voice note", which happens to line up with "same project" most of the time.
Two fixes to the scoring
Then a guardrail: a check that can veto a result no matter how good its main score is. For every mixed-project sibling pair, look at how similar they are compared with all of that person's other pairs, and average the percentile. Above 80 means mixed dumps are collapsing, and the setting fails. I wrote the threshold down before running anything that would be judged by it, because a line drawn after you've seen where the points fall isn't a line.
Late fusion
If the transcript is too loud, turn it down. Late fusion embeds the title and the transcript separately and blends the two vectors, with α setting the mix: α = 1 is title only, α = 0 is transcript only. (Early fusion would be gluing the texts together before embedding, which is what drowned the title in the first place.)
v = normalise(α · v_title + (1 − α) · v_transcript)
Here's gte-small across the grid:
| α | others: score | others: guard | me: map | me: guard |
|---|---|---|---|---|
| 1.0 (title only) | 62.8 | 48 | 30.0 | 67 ✓ |
| 0.9 | 63.9 | 58 | 30.9 | 87 ✗ |
| 0.8 | 64.2 | 67 | 32.4 | 95 ✗ |
| 0.7 (chosen) | 64.7 | 73 | 33.3 | 97 ✗ |
| 0.5 | 64.6 | 80 | 36.4 | 98 ✗ |
| 0.0 (transcript only) | 61.8 | 73 | 38.1 | 98 ✗ |
Three things in that table I'll remember:
- The win was noise. The chosen blend beat title-only by 1.9 points for other people. The two other small models did the same (+1.8, +2.5). All inside the noise.
- On my data, the score went up while the map got worse. My map score climbs steadily as the transcript gets louder, and the guardrail breaks at the very first step. That's the whole reason guardrails exist. The score rewarded what the guardrail could see was a failure. (Part of that climb is real: even with siblings excluded, the transcript adds about 8 points for me, probably because I say things like "that can go in my Forked folder" out loud and never put it in the title.)
- A guardrail that passes for one group can fail for another. Everyone else stays under 80 at every blend. I break at 0.9. My transcripts are shorter and I have 15 projects, so I'm not a typical user, and my data is the one that showed the problem.
5. Drawing the map
A model gives me 218 points in 384 dimensions. A screen has two. Getting from one to the other took four steps, and each one has its own way of lying to you.
- 1 · UMAP: squash the dimensions
- Dimensionality reduction keeps as much of the structure as it can while throwing dimensions away. UMAP is good at keeping local neighbourhoods: things that were close stay close. It isn't good at global distance, so two clusters on opposite sides of the screen aren't necessarily more different than two that touch. Read a UMAP picture like a subway map, not a street map.
- 2 · HDBSCAN: find the topics
- A density-based clusterer: topics are crowded regions, separated by empty space. Unlike k-means, you don't tell it how many topics to find, and it's allowed to say "this one doesn't belong anywhere". For a pile of to-dos that honesty matters. Some thoughts really are one-offs.
- 3 · c-TF-IDF: name the topics
- Treat each cluster as one long document (all its titles joined), then rank words by how common they are here compared with every other cluster. That finds the words that make a cluster distinctive, not just frequent. I had to add my own stop words, because "add", "task" and "follow" turn up everywhere in voice notes and say nothing.
- 4 · nearest thoughts: back to full resolution
- Click a dot and the map lists its closest thoughts, measured with cosine in the full 384 dimensions, not distance on screen. Squashing to 2D bends distances, so anything that answers a question uses the real vectors. The picture is only for looking.
One small trick makes the "with context" toggle readable. UMAP starts from a random arrangement, so two separate runs scatter the same thoughts in completely different places. The context layout starts from the title layout and a fixed random seed, so when you flip the toggle, points move rather than reshuffle, and you can watch which ones slide together. With context: 18 topics instead of 21, and 28 unsorted instead of 36. Fewer loose ends looks tidier, and it's exactly the dumps collapsing from step 4.
It's still a lovely thing to look at. There's a topic for digging out the yard, one for building the shed, one for pilates, one for a photo shoot, one for the app's beta, and a little island about the reptile terrarium. A few months of rambling, sorted into a picture without my filing any of it. (One surprise: c-TF-IDF loves people's names, because nothing is more distinctive to one topic than the person it's about. Fine on my own laptop. Not fine on anything anyone else will ever see.)
What's next
- Fine-tune: train gte-small two ways and compare. Triplet loss pulls the closest same-project thread nearer and pushes the closest other-project thread away. In-batch negatives, the industry default, assumes everything else in a batch is "different", even when it isn't.
- Put the fine-tuned model on the map next to the stock one, so you can see what training changed (if anything).
- Wait for new thoughts. Everything created after 12 October is a sealed test set: when there's enough of it, every choice in this post gets graded on it once, honestly.
Honest caveat: 218 of my threads and about 600 shared ones make this a learning project, not a benchmark. Ask me again in a few thousand thoughts.