---
title: "Ashish Vaswani's Transformer paper landed while I was sleeping in the lab"
description: "In 2017 I was sleeping in the MindGarage lab to get more GPU hours for my master's thesis on making neural networks show their reasoning. Then Attention Is All You Need appeared, and my bed had to wait a few more days."
author: Ayushman Dash
date: 2026-09-07
canonical: https://aiwithayushman.com/blog/ashish-vaswani-attention-paper-landed-while-i-slept-in-the-lab
---
# Ashish Vaswani's Transformer paper landed while I was sleeping in the lab
The whiteboard behind our machines said **Do Not Shutdown**.
That was the most important sentence in the room. We shared a handful of GPUs
between everyone in the lab, and a shutdown meant somebody's training run died
and somebody's week went with it.
This was the MindGarage, the deep learning lab at TU Kaiserslautern, and I was
a master's student there with a thesis to finish.
## What I was actually trying to do
My thesis question was simple to say and hard to answer. Could a neural network
show me its reasoning, not just its answer?
In 2016 and 2017 that was not the fashionable question. Neural network research
was on a steep rise, everyone wanted bigger models and better scores, and I was
sitting in a corner asking whether the model could explain itself. I still think
it was the right question. I just did not know how long the field would take to
agree.
The experiments needed GPU time. Lots of it, and the queue never got shorter.
So I started sleeping in the lab.
Not as a heroic thing. As a practical one. If I was there when a run finished,
I could start the next one immediately instead of losing the machine to whoever
woke up first. I got a few hours on a chair, the runs got a few more hours on
the GPUs, and my thesis moved. Winter helped in its own way, the machines kept
the room warm, which I have
[written about before](/blog/how-i-used-nvidia-gpus-to-keep-myself-warm-during-peak-german-winters).
## Then the paper
In June 2017, [Ashish Vaswani](/blog/ashish-vaswani-the-lesser-known-titan-who-built-the-future-of-ai)
and seven co-authors at Google published
[Attention Is All You Need](https://arxiv.org/abs/1706.03762).
I was near the end. The last set of results was almost in, the writing was
mostly done, and I had a plan for that week: finish the runs, go back to my
room, sleep in an actual bed.
My bed was calling. I did not go.
I read the paper in the lab and could not put it down. The idea was not a small
improvement on what we were doing. It removed the part everyone had accepted as
necessary. No recurrence. No reading a sentence one word at a time. Attention
alone, which meant the whole sequence could be processed at once, which meant
training could finally use the hardware properly.
Sitting in a room where the entire bottleneck of my life was GPU hours, that
last part was not an abstract idea. It was the difference between a week and a
night.
I stayed a few more days.
Nine years later I used the same architecture for something much smaller. My mother sends me a good-morning image on WhatsApp every day, so I built 5 AI agents that answer her the same way, every morning at six.Check it out ↗
## What it cost me
I stopped, and I started running experiments again.
[That decision cost me months](/blog/i-was-almost-done-with-my-thesis-when-the-transformer-paper-landed)
I had not budgeted, and my supervisor had to hear a new plan for a thesis that
was supposed to be finished.
It was completely worth it, and I would do it again.
Not because I predicted what was coming. Nobody in that lab said the words
"this will lead to ChatGPT". We said something much smaller and more honest:
this is better, and I want to know why.
## Where it led
Three years later my team built
[some of the first transformer language models for Indian languages](/research).
Five years later I was selling speech and document models built on that
architecture to banks and broadcasters. Today my team uses it for legal research
agents where a wrong answer is somebody's case.
And the thing I was actually asking about in 2017, whether a model can show its
reasoning, is now the centre of the field. It took the rest of the world a while
to get to my thesis question, which is the closest thing to a happy ending a
graduate student gets.
The whiteboard was right, by the way. Do not shutdown.
## What I was actually trying to do
My thesis question was simple to say and hard to answer. Could a neural network
show me its reasoning, not just its answer?
In 2016 and 2017 that was not the fashionable question. Neural network research
was on a steep rise, everyone wanted bigger models and better scores, and I was
sitting in a corner asking whether the model could explain itself. I still think
it was the right question. I just did not know how long the field would take to
agree.
The experiments needed GPU time. Lots of it, and the queue never got shorter.
So I started sleeping in the lab.
Not as a heroic thing. As a practical one. If I was there when a run finished,
I could start the next one immediately instead of losing the machine to whoever
woke up first. I got a few hours on a chair, the runs got a few more hours on
the GPUs, and my thesis moved. Winter helped in its own way, the machines kept
the room warm, which I have
[written about before](/blog/how-i-used-nvidia-gpus-to-keep-myself-warm-during-peak-german-winters).
## Then the paper
In June 2017, [Ashish Vaswani](/blog/ashish-vaswani-the-lesser-known-titan-who-built-the-future-of-ai)
and seven co-authors at Google published
[Attention Is All You Need](https://arxiv.org/abs/1706.03762).
I was near the end. The last set of results was almost in, the writing was
mostly done, and I had a plan for that week: finish the runs, go back to my
room, sleep in an actual bed.
My bed was calling. I did not go.
I read the paper in the lab and could not put it down. The idea was not a small
improvement on what we were doing. It removed the part everyone had accepted as
necessary. No recurrence. No reading a sentence one word at a time. Attention
alone, which meant the whole sequence could be processed at once, which meant
training could finally use the hardware properly.
Sitting in a room where the entire bottleneck of my life was GPU hours, that
last part was not an abstract idea. It was the difference between a week and a
night.
I stayed a few more days.