dair-ai /ML-Papers-Explained
Explanation to key concepts in ML
★ 8,931⑂ 738👁 167⑃ 25● 8▤ 0
More from this author
+0
Daily
8,931
stars
738
forks
167
watchers
25
prs
8
issues
0
releases
README
ML Papers Explained Explanations to key concepts in ML Language Models | Paper | Date | Description | |---|---|---| | Transformer | June 2017 | An Encoder Decoder model, that introduced multihead attention mechanism for language translation task. | | Elmo | February 2018 | Deep contextualized word representations that captures both intricate aspects of word usage and contextual variations across language contexts. | | Marian MT | April 2018 | A Neural Machine Translation framework written entirely in C++ with minimal dependencies, designed for high training and translation speed. | | GPT | June 2018 | A Decoder only transformer which is autoregressively pretrained and then finetuned for specific downstream tasks using task-aware input transformations. | | BERT | October 2018 | Introduced pre-training for Encoder Transformers. Uses unified architecture across different tasks. | | Transformer XL | January 2019 | Extends the original Transformer model to handle longer sequences of text by introducing recurrence into the self-attention mechanism. | | XLM | January 2019 | Proposes two methods to learn cross-lingual language models (XLMs): one unsupervised that only relies on monolingual data, and one supervised that leverages parallel data with a new cross-lingual language model objective. | | GPT 2 | February 2019 | Demonstrates that language models begin to learn various language processing tasks without any explicit supervision. | | Sparse Transformer | April 2019 | Introduced sparse factorizations of the attention matrix to reduce the time and memory consumption to O(n√ n) in terms of sequence lengths. | | UniLM | May 2019 | Utilizes a shared Transformer network and specific self-attention masks to excel in both language understanding and generation tasks. | | XLNet | June 2019 | Extension of the Transformer-XL, pre-trained using a new method that combines ideas from AR and AE objectives. | | RoBERTa | July 2019 | Built upon BERT, by carefully optimizing hyperparameters and training data size to improve performance on various language tasks . | | Sentence BERT | August 2019 | A modification of BERT that uses siamese and triplet network structures to derive sentence embeddings that can be compared using cosine-similarity. | | CTRL | September 2019 | A 1.63B language model that can generate text conditioned on control codes that govern style, content, and task-specific behavior, allowing for more explicit control over text generation. | | Tiny BERT | September 2019 | Uses attention transfer, and task specific distillation for distilling BERT. | | ALBERT | September 2019 | Presents certain parameter reduction techniques to lower memory consumption and increase the training speed of BERT. | | Distil BERT | October 2019 | Distills BERT on very large batches leveraging gradient accumulation, using dynamic masking and without the next sentence prediction objective. | | Distil RoBERTa | October 2019 | Distillation of RoBERTa, using the same techniqu