Introduction to Regular Expressions

2d:null 2e:["$","title",null,{"children":"ATTACK | Israel Gonzalez-Brooks | Observable"}] 33:T9e2,## GETTING STARTED ![c0045c9d-c6ca-449a-bb5d-849f22383452.jpeg](${await FileAttachment("c0045c9d-c6ca-449a-bb5d-849f22383452.jpeg").url()}) **Introduction ** The following sections give you a quick overview of transformer architectures - What are transformers? - Why use transformers? - Are there drawbacks of transformers? - What are the most important transformer architectures? **Preparation** Preparation steps include installing and setting up the necessary libraries, as well as getting familiar with some basic concepts in machine learning (ML) and natural language processing (NLP). - What do I need to know beforehand? - Additional Resources - Summary Transformers are a family of neural network architectures, originally introduced in the paper [Attention is All You Need](https://arxiv.org/pdf/1706.03762.pdf). Transformers are the first neural network architecture to make use of the attention mechanism extensively throughout the network. The attention mechanism was first proposed in the paper [A Neural Attention Model for Sentence Summarization](https://aclweb.org/anthology/D/D15/D15-1166.pdf) and later refined in the paper [Effective Approaches to Attention-based Neural Machine Translation](https://arxiv.org/pdf/1508.04025.pdf) The attention mechanism was originally used to allow a neural network model to learn to pay attention to different parts of the input sequence based on their importance. In the transformers architecture, the attention mechanism is used not only to decide which parts of the input sequence to pay attention to, but also which parts of the 'output sequence' to pay attention to. In order to allow the model to pay attention to the output sequence, the output sequence is fed back into the network as another input. This is known as 'self-attention'. In addition, the output from the network at each point in the output sequence is fed back into the input at the same point in the output sequence. This is known as 'encoder-decoder attention'. The combination of self-attention and encoder-decoder attention allows the model to make use of the entire output sequence as well as the entire input sequence when generating each point in the output sequence. This is useful when the output sequence is related to the input sequence, such as in machine translation. The transformer architecture is most commonly used in the field of **Natural Language Processing**, but has also been applied to **Computer Vision**, where it has produced state-of-the-art results. ---34:T922,## APPLIED TRANSFORMER ARCHITECTURES ![926a2f08-37a9-4d4f-8999-b68cb6409717.jpeg](${await FileAttachment("926a2f08-37a9-4d4f-8999-b68cb6409717.jpeg").url()}) ### TRANSFORMER BUILDING BLOCKS **Self-Attention** There are two main types of attention: self-attention, and attention. Self-attention refers to the type of attention that is used by the Transformer model. The Transformer model is not a traditional neural network; instead, it is a sequence-to-sequence model that uses a combination of recurrent neural networks and self-attention to process sequential data. The Transformer model was designed to be a sequence-to-sequence model that could process long sequences of data without the need for a convolutional neural network. In this book, we will focus on the Transformer model, as well as other models that use self-attention. ### TRANSFORMER MODEL COMPONENTS **Encoder-Decoder** The encoder is a neural network that is used for encoding the input sequence. The encoder transforms the input sequence into an representation that other parts of the architecture can use. This internal representation is then used by the decoder to generate the output sequence. The encoder and decoder are usually implemented as separate neural networks. **Sparse Attention** The sparse attention mechanism was first introduced in the paper "Attention Is All You Need" by Vaswani et al. It reduces the number of parameters in the model by using a sparse matrix to represent the attention map. This has the effect of reducing the number of trainable parameters and speeds up the training process. **Reordered Attention** The reordered attention mechanism was first introduced in the paper "Reformer: The Efficient Transformer" by Kitaev et al. This mechanism reorders the given input, with attention applied to the reordered version instead. ### APPLIED TRANSFORMER MODELS **The Transformer The Universal Transformer The Sparse Transformer The Reformer The Longformer The Switch Transformer ** ### APPLIED COMBINATIONS OF TRANSFORMER MODELS **Transformer-XL XLNet RoBERTa ** ### APPLIED TRANSFORMER ARCHITECTURES FOR VISION **Vision Transformers (*ViT*) Distillation-Enhanced Vision Transformers (*DeiT*) T2T-ViT** ### APPLIED TRANSFORMER ARCHITECTURES FOR GENERATION **OpenAI GPT GPT-2 GPT-3 ** ---35:T4238, Introduction to Regular Expressions

36:[{"id":"05bdafb71126fbfe6bf033c794a64505290dcda971f6ca92336cfbcfbdabb3b8efbacf21c25290eec3180c843911a8e947955fa92af592b02c449dba11013a27","href":"https://static.observableusercontent.com/files/05bdafb71126fbfe6bf033c794a64505290dcda971f6ca92336cfbcfbdabb3b8efbacf21c25290eec3180c843911a8e947955fa92af592b02c449dba11013a27","name":"03561269-9b80-4a61-8e8e-1b81f95faabc.jpeg","type":"image/jpeg","lastModified":1669086738538.81,"size":41262},{"id":"a8481a46b75f573aa2abe8b8f0ca70f1f42a4dea9ebc30ae2967eb7b3f59072363e877cd3b3f080e1abc9cd56eeb7cb680a137cd919f8aaa3a45406448b4ee4c","href":"https://static.observableusercontent.com/files/a8481a46b75f573aa2abe8b8f0ca70f1f42a4dea9ebc30ae2967eb7b3f59072363e877cd3b3f080e1abc9cd56eeb7cb680a137cd919f8aaa3a45406448b4ee4c","name":"13f7015c-201d-451f-bc9d-38fbf751f4b7.jpeg","type":"image/jpeg","lastModified":1669090078025.405,"size":43891},{"id":"556c5cae874c43cfe4c5c2ed1e2001df3270649b559fde73799db6fd89f3e9b377b23f64050bc46cc33d31c8e34cff0bae044faec35dd63af156768d77fbe2a8","href":"https://static.observableusercontent.com/files/556c5cae874c43cfe4c5c2ed1e2001df3270649b559fde73799db6fd89f3e9b377b23f64050bc46cc33d31c8e34cff0bae044faec35dd63af156768d77fbe2a8","name":"69314682-d1ae-424c-9319-8990c1a6a5d3.jpeg","type":"image/jpeg","lastModified":1669090325237.736,"size":47734},{"id":"f4ba55ad43e1a8de23df6d9ec0215fda0e47988884f1c9d0daebfac20276cf1cd2e153ebf865a111283f2f4e763b4a57e8e98792591b9cc428180bcff1c566f1","href":"https://static.observableusercontent.com/files/f4ba55ad43e1a8de23df6d9ec0215fda0e47988884f1c9d0daebfac20276cf1cd2e153ebf865a111283f2f4e763b4a57e8e98792591b9cc428180bcff1c566f1","name":"6e50d626-70dc-433e-a8af-1dc6c1692587.jpeg","type":"image/jpeg","lastModified":1669090251857.347,"size":36518},{"id":"aebe54adad6438505a30a64e8e5a9985eecb2c5e30a6f5227bc6de070d923d53f553c5489ced25d146a2a421fc811872c249116d5fe22970bfa38dec709c462b","href":"https://static.observableusercontent.com/files/aebe54adad6438505a30a64e8e5a9985eecb2c5e30a6f5227bc6de070d923d53f553c5489ced25d146a2a421fc811872c249116d5fe22970bfa38dec709c462b","name":"926a2f08-37a9-4d4f-8999-b68cb6409717.jpeg","type":"image/jpeg","lastModified":1669090221510.08,"size":45520},{"id":"5fcf3f15beb70f44b170b124faa27e3edc45a9665b49f6e70a06f162de0ce6ba933f6abe6d91a00dfbd59a34f291904e94578a1054e1638e8d7d5f1251f87839","href":"https://static.observableusercontent.com/files/5fcf3f15beb70f44b170b124faa27e3edc45a9665b49f6e70a06f162de0ce6ba933f6abe6d91a00dfbd59a34f291904e94578a1054e1638e8d7d5f1251f87839","name":"c0045c9d-c6ca-449a-bb5d-849f22383452.jpeg","type":"image/jpeg","lastModified":1669090182872.169,"size":44556}] 31:{"id":"efbe5cf1b5f53f67","slug":"attack","title":"ATTACK","thumbnail":"a4c13375d18580a91e5364c1d950b92e223d7306f43b72d941ccf57319f7fccd","access_level":"public","create_time":"$D2022-11-22T00:03:50.410Z","update_time":"$D2023-01-17T01:13:28.527Z","view_time":null,"first_public_time":"$D2022-11-22T04:12:59.558Z","has_unpublished_edits":null,"fork":{"id":"662080314eb2cb2c","slug":"blog-post","title":"Blog Post","owner":{"id":"f35c755083683fe5","login":"observablehq","name":"Observable"}},"stars":1,"starred":false,"owner":{"type":"team","id":"b6cc08c484e885c2","login":"iz","name":"Israel Gonzalez-Brooks","avatar_url":"https://avatars-next.observableusercontent.com/b18cd83a73bc56f94cb9d184956c2a4fb3e685f9b74d25627832cf49d44a16c9","tier":"starter_2024"},"version":146,"publish_version":null,"role":null,"authored":false,"authors":[{"type":"individual","id":"3edb53fdfab76653","login":"iz","name":"Israel Gonzalez-Brooks","avatar_url":"https://avatars-next.observableusercontent.com/b18cd83a73bc56f94cb9d184956c2a4fb3e685f9b74d25627832cf49d44a16c9","tier":null}],"members":[{"type":"individual","id":"3edb53fdfab76653","login":"iz","name":"Israel Gonzalez-Brooks","avatar_url":"https://avatars-next.observableusercontent.com/b18cd83a73bc56f94cb9d184956c2a4fb3e685f9b74d25627832cf49d44a16c9","tier":null,"role":"editor"}],"comments":[],"custom_thumbnail":false,"trashed":false,"lapsed":null,"subscribed":null,"forks":1,"collection":null,"type":"notebook","body":{"title":"ATTACK","cells":[{"id":0,"value":"# ***ATTACK***","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":78,"value":"b804a618e8e1b81f95faabc = FileAttachment(\"03561269-9b80-4a61-8e8e-1b81f95faabc.jpeg\").image()","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":41,"value":"---\n### Applied Techniques in Transformer Architectures for Compression of Knowledge","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":129,"value":"## PREFACE \n\n![13f7015c-201d-451f-bc9d-38fbf751f4b7.jpeg](${await FileAttachment(\"13f7015c-201d-451f-bc9d-38fbf751f4b7.jpeg\").url()}) \nIn the past few years, transformers architectures have been the focus of a lot of research, especially in the field of **natural language processing**.\nA lot of people are talking about neural networks, but few people truly understand what these architectures are and how they work.\nThe more opaque these technologies get, the more difficult they will be to democratize and spread.\nThis book is a small attempt to help fill that gap.\n\n---\n","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":35,"value":"$33","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":100,"value":"$34","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":101,"value":"## DEVELOPING TRANSFORMER-BASED MODELS \n![6e50d626-70dc-433e-a8af-1dc6c1692587.jpeg](${await FileAttachment(\"6e50d626-70dc-433e-a8af-1dc6c1692587.jpeg\").url()}) \n\n### TRAINING\n**Training** The training phase of a transformer-based model is the process of optimizing the parameters that are used in the model.\nThis involves creating a model from scratch, or loading a model and then training it.\n\n**Pre-Training** \n**Fine-Tuning** \n\n### APPLICATION\n**Inference** The inference (basically the user-facing \"app\") phase of a transformer-based model is the process of running the model on data that is not part of the training dataset.\nThis is done using a model that has been previously trained and is loaded.\nThis is where the model is used to make predictions, generate text, and perform other downstream tasks.\n\n### RESULTS\nEvaluation \n\n---","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":103,"value":"## LOOKING FORWARD\n![69314682-d1ae-424c-9319-8990c1a6a5d3.jpeg](${await FileAttachment(\"69314682-d1ae-424c-9319-8990c1a6a5d3.jpeg\").url()}) \n\n### TRENDS IN DEVELOPMENT\nStudying the transformer architectures listed in this book makes it clear that there are noticeable trends in this domain.\nWe'll try to look at these trends and figure out in what ways they are impacting the future of artificial intelligence applications.\n\n### FUTURE DIRECTIONS\n...\n\n---","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":105,"value":"## CONCLUSIONS\n\n---\n","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":116,"value":"## APPENDIX A: TRANSFORMER ARCHITECTURE SETTINGS\nReference collection of the most important settings used in the various transformer architectures.\nThis includes the use of multi-head attention, the use of relative position embedding, and of sparse attention.\n\n---","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":118,"value":"## APPENDIX B: TRANSFORMER ARCHITECTURE REFERENCE\nPapers, tutorials, and other related works for deeper dives into transformer architectures.\n\n---\n","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":120,"value":"## APPENDIX C: TRANSLATION OF PAPERS\nNon-English resources and publications related to transformers.\nEach resource is accompanied by a translation of the title and a short summary.\n\n---\n\n","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":124,"value":"## REFERNCES\n\n---","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":122,"value":"## INDEX","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":73,"value":"$35","pinned":false,"hidden":false,"mode":"html","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"}],"files":"$W36","stdlib":"1","titling":"auto"}} 32:"$31:body"