Transformers Beyond Text: Is Attention Only an NLP Trick?

If you have ever used ChatGPT, you have used a Transformer. It is the architecture behind almost every modern AI system that feels smart. But there is a misconception I run into constantly: that Transformers are a natural-language tool, something built for text and text only.

They are not. The Transformer is one of the most general ideas in modern machine learning, and the engine inside it, attention, shows up in places that have nothing to do with language: images, audio, proteins, and yes, graphs. Let me walk through what a Transformer actually is, then answer the two questions I get asked most.

So what is a Transformer, really?

Introduced in 2017 in a paper with the now-famous title Attention Is All You Need, the Transformer threw out the recurrence and convolutions that older models leaned on and kept a single trick: self-attention.

The idea is simpler than it sounds. Take a sequence of tokens (for text, think words). For each token the model asks: which other tokens should I pay attention to right now? It answers with three learned vectors per token, a Query, a Key, and a Value:

• The Query is what this token is looking for.

• The Key is what each token offers.

• Matching queries against keys gives attention weights, how much every token matters to every other token.

• Each token's new representation is then a weighted blend of everyone's Values.

Because every token can look at every other token directly, the model captures long-range relationships in a single step, and it does so in parallel, which is exactly why Transformers train so well on GPUs. Here is what those attention weights look like in practice, the classic example where the word "it" learns to point back at "animal":

That is essentially the whole magic. Stack this operation a few dozen times, add some normalization and feed-forward layers, and you have the backbone of GPT, BERT, and the rest.

But isn't this just for language?

This is the misconception. Nothing about attention assumes words. A "token" is just a vector, and it can represent anything you can slice your data into.

The cleanest example is vision. The Vision Transformer (ViT) takes an image, cuts it into a grid of small patches, flattens each patch into a vector, and feeds that sequence into the exact same Transformer. No convolutions required, and at large scale it matches or beats traditional CNNs.

And it does not stop there:

• Audio: speech models like Whisper treat sound as a sequence of spectrogram frames.

• Biology: AlphaFold and protein language models like ESM run attention over amino-acid sequences to predict 3D structure.

• Time series: forecasting on sensor, demand, and financial data increasingly uses attention over time steps.

• Reinforcement learning: the Decision Transformer reframes control as sequence modeling.

• Multimodal: models like CLIP place text and images in the same attention-driven space.

Same architecture, different tokens. That generality is the real reason Transformers took over.

Does attention show up in Graph Neural Networks?

Yes, and this is where it gets beautiful.

Graph Neural Networks (GNNs) work on data that is naturally a graph: social networks, molecules, road maps, recommendation systems. A node updates itself by aggregating messages from its neighbors, and the obvious question is how much weight to give each neighbor. Graph Attention Networks (GAT) answer it with attention: each node scores its neighbors and blends their features accordingly. Sound familiar?

Here is the punchline that reframed how I think about all of this: a Transformer is a Graph Neural Network running on a fully-connected graph. In a Transformer every token attends to every other token, so the underlying "graph" is complete, and attention is just a learned, data-dependent set of edge weights. Positional encodings are how we hand the model some structure on top of that complete graph.

So attention was not borrowed from NLP and bolted onto graphs. Attention was graph-native all along. NLP just happened to be where it got famous.

The one idea to remember

Strip away the jargon and a Transformer does one thing: it lets every piece of your data decide, on the fly, which other pieces it should listen to. Words, image patches, audio frames, graph nodes, if you can turn it into tokens and define what "related" means, attention applies.

That is why "is this only for NLP?" is the wrong question. The better one is: what is the right way to tokenize my problem?

Why I care

At BrightLearn this is very concrete. When we model how a student moves through a lesson, the signal is rarely the last thing they did, it is the relationship between answers, hesitations, and concepts spread across an entire session. Attention is a natural fit for that kind of "what relates to what" problem, whether the tokens are words in an answer or steps in a learning path.

If you have been treating Transformers as a text tool, it is worth a second look. The architecture is far more general than the hype suggests, and that generality is the whole point.