Ashish Vaswani's Transformer paper landed while I was sleeping in the lab

The whiteboard behind our machines said Do Not Shutdown.

That was the most important sentence in the room. We shared a handful of GPUs
between everyone in the lab, and a shutdown meant somebody’s training run died
and somebody’s week went with it.

This was the MindGarage, the deep learning lab at TU Kaiserslautern, and I was
a master’s student there with a thesis to finish.

What I was actually trying to do

My thesis question was simple to say and hard to answer. Could a neural network
show me its reasoning, not just its answer?

In 2016 and 2017 that was not the fashionable question. Neural network research
was on a steep rise, everyone wanted bigger models and better scores, and I was
sitting in a corner asking whether the model could explain itself. I still think
it was the right question. I just did not know how long the field would take to
agree.

The experiments needed GPU time. Lots of it, and the queue never got shorter.

So I started sleeping in the lab.

Not as a heroic thing. As a practical one. If I was there when a run finished,
I could start the next one immediately instead of losing the machine to whoever
woke up first. I got a few hours on a chair, the runs got a few more hours on
the GPUs, and my thesis moved. Winter helped in its own way, the machines kept
the room warm, which I have
written about before.

Then the paper

In June 2017, Ashish Vaswani
and seven co-authors at Google published
Attention Is All You Need.

I was near the end. The last set of results was almost in, the writing was
mostly done, and I had a plan for that week: finish the runs, go back to my
room, sleep in an actual bed.

My bed was calling. I did not go.

I read the paper in the lab and could not put it down. The idea was not a small
improvement on what we were doing. It removed the part everyone had accepted as
necessary. No recurrence. No reading a sentence one word at a time. Attention
alone, which meant the whole sequence could be processed at once, which meant
training could finally use the hardware properly.

Sitting in a room where the entire bottleneck of my life was GPU hours, that
last part was not an abstract idea. It was the difference between a week and a
night.

I stayed a few more days.

Nine years later I used the same architecture for something much smaller. My mother sends me a good-morning image on WhatsApp every day, so I built 5 AI agents that answer her the same way, every morning at six. Check it out ↗

What it cost me

I stopped, and I started running experiments again.
That decision cost me months
I had not budgeted, and my supervisor had to hear a new plan for a thesis that
was supposed to be finished.

It was completely worth it, and I would do it again.

Not because I predicted what was coming. Nobody in that lab said the words
“this will lead to ChatGPT”. We said something much smaller and more honest:
this is better, and I want to know why.

Where it led

Three years later my team built
some of the first transformer language models for Indian languages.
Five years later I was selling speech and document models built on that
architecture to banks and broadcasters. Today my team uses it for legal research
agents where a wrong answer is somebody’s case.

And the thing I was actually asking about in 2017, whether a model can show its
reasoning, is now the centre of the field. It took the rest of the world a while
to get to my thesis question, which is the closest thing to a happy ending a
graduate student gets.

The whiteboard was right, by the way. Do not shutdown.