Resources

The Transformer Cookbook

The chefs of FLaNN are pleased to present The Transformer Cookbook! Inside you will find detailed recipes for cooking up a variety of algorithms using the weights of a transformer. Hungry for an induction head? Craving a Dyck language recognizer? The Transformer Cookbook has you covered.

The Survey

The survey paper, What Formal Languages Can Transformers Express?, maps out the landscape of formal language expressvity results for transformers. The paper aims to clarify and unify the intricate and seemingly contradictory results in this area.

RASP

RASP (Restricted Access Sequence Processing) is a programming language designed to express transformer computations in a human-readable format, first introduced in the paper Thinking Like Transformers by Gail Weiss et al. Several tools have been developed for working with RASP and subsequent variants:

For RASP, there is the repl and a Python version by Sasha Rush.
For B-RASP, there is a web simulator by Andy Yang.
For C-RASP, there is an interpreter by Alexander Koller.