"For those who could be interested, here is a short article on how I used the results of this model to train a LSTM model for signature detection in emails:\n",
"For those who could be interested, here is a short article on how I used the results of this model to train a LSTM model for signature detection in emails:\n",
"> The model introduction and model weights originate from https://huggingface.co/Jean-Baptiste/roberta-large-ner-english and were converted to PaddlePaddle format for ease of use in PaddleNLP."
"Continue trained mengzi-bert-base with 20G financial news and research reports. Masked language modeling(MLM), part-of-speech(POS) tagging and sentence order prediction(SOP) are used as training task.\n"
]
},
{
"cell_type": "markdown",
"id": "9aa78f76",
"metadata": {},
"source": [
"[Mengzi: Towards Lightweight yet Ingenious Pre-trained Models for Chinese](https://arxiv.org/abs/2110.06696)\n"
"Continue trained mengzi-bert-base with 20G financial news and research reports. Masked language modeling(MLM), part-of-speech(POS) tagging and sentence order prediction(SOP) are used as training task.\n"
]
},
{
"cell_type": "markdown",
"id": "26c65092",
"metadata": {},
"source": [
"[Mengzi: Towards Lightweight yet Ingenious Pre-trained Models for Chinese](https://arxiv.org/abs/2110.06696)\n"
"title={Mengzi: Towards Lightweight yet Ingenious Pre-trained Models for Chinese},\n",
"author={Zhuosheng Zhang and Hanqing Zhang and Keming Chen and Yuhang Guo and Jingyun Hua and Yulong Wang and Ming Zhou},\n",
"year={2021},\n",
"eprint={2110.06696},\n",
"archivePrefix={arXiv},\n",
"primaryClass={cs.CL}\n",
"}\n",
"```"
]
},
{
"cell_type": "markdown",
"id": "f25bda96",
"metadata": {},
"source": [
"> The model introduction and model weights originate from https://huggingface.co/Langboat/mengzi-bert-base-fin and were converted to PaddlePaddle format for ease of use in PaddleNLP."
"This model is a fine-tuned version of the [spanish BERT model](https://huggingface.co/dccuchile/bert-base-spanish-wwm-cased) with the Spanish portion of the XNLI dataset. \n"
"> The model introduction and model weights originate from https://huggingface.co/Recognai/bert-base-spanish-wwm-cased-xnli and were converted to PaddlePaddle format for ease of use in PaddleNLP."
"> The model introduction and model weights originate from [https://huggingface.co/allenai/macaw-large](https://huggingface.co/allenai/macaw-large) and were converted to PaddlePaddle format for ease of use in PaddleNLP."
"SPECTER is a pre-trained language model to generate document-level embedding of documents. It is pre-trained on a a powerful signal of document-level relatedness: the citation graph. Unlike existing pretrained language models, SPECTER can be easily applied to downstream applications without task-specific fine-tuning.\n",
"\n",
"Paper: [SPECTER: Document-level Representation Learning using Citation-informed Transformers](https://arxiv.org/pdf/2004.07180.pdf)\n",
"SPECTER is a pre-trained language model to generate document-level embedding of documents. It is pre-trained on a a powerful signal of document-level relatedness: the citation graph. Unlike existing pretrained language models, SPECTER can be easily applied to downstream applications without task-specific fine-tuning.\n",
"\n",
"Paper: [SPECTER: Document-level Representation Learning using Citation-informed Transformers](https://arxiv.org/pdf/2004.07180.pdf)\n",
"> The model introduction and model weights originate from [https://huggingface.co/allenai/specter](https://huggingface.co/allenai/specter) and were converted to PaddlePaddle format for ease of use in PaddleNLP."
"> The model introduction and model weights originate from [https://huggingface.co/alvaroalon2/biobert_chemical_ner](https://huggingface.co/alvaroalon2/biobert_chemical_ner) and were converted to PaddlePaddle format for ease of use in PaddleNLP."
"> The model introduction and model weights originate from [https://huggingface.co/alvaroalon2/biobert_diseases_ner](https://huggingface.co/alvaroalon2/biobert_diseases_ner) and were converted to PaddlePaddle format for ease of use in PaddleNLP."
"> The model introduction and model weights originate from [https://huggingface.co/alvaroalon2/biobert_genetic_ner](https://huggingface.co/alvaroalon2/biobert_genetic_ner) and were converted to PaddlePaddle format for ease of use in PaddleNLP."
"**Input:** Supports over 100 Languages. See [List of supported languages](https://github.com/google-research/bert/blob/master/multilingual.md#list-of-languages) for all available.\n"
]
},
{
"cell_type": "markdown",
"id": "8b40d5de",
"metadata": {},
"source": [
"**Purpose:** This module takes a search query [1] and a passage [2] and calculates if the passage matches the query.\n",
"It can be used as an improvement for Elasticsearch Results and boosts the relevancy by up to 100%.\n"
]
},
{
"cell_type": "markdown",
"id": "c9d89366",
"metadata": {},
"source": [
"**Architecture:** On top of BERT there is a Densly Connected NN which takes the 768 Dimensional [CLS] Token as input and provides the output ([Arxiv](https://arxiv.org/abs/1901.04085)).\n"
]
},
{
"cell_type": "markdown",
"id": "29745195",
"metadata": {},
"source": [
"**Output:** Just a single value between between -10 and 10. Better matching query,passage pairs tend to have a higher a score.\n"
]
},
{
"cell_type": "markdown",
"id": "010a4d92",
"metadata": {},
"source": [
"## Intended uses & limitations\n",
"Both query[1] and passage[2] have to fit in 512 Tokens.\n",
"As you normally want to rerank the first dozens of search results keep in mind the inference time of approximately 300 ms/query.\n"
"This model is trained using the [**Microsoft MS Marco Dataset**](https://microsoft.github.io/msmarco/ \"Microsoft MS Marco\"). This training dataset contains approximately 400M tuples of a query, relevant and non-relevant passages. All datasets used for training and evaluating are listed in this [table](https://github.com/microsoft/MSMARCO-Passage-Ranking#data-information-and-formating). The used dataset for training is called *Train Triples Large*, while the evaluation was made on *Top 1000 Dev*. There are 6,900 queries in total in the development dataset, where each query is mapped to top 1,000 passage retrieved using BM25 from MS MARCO corpus.\n"
"**Input:** Supports over 100 Languages. See [List of supported languages](https://github.com/google-research/bert/blob/master/multilingual.md#list-of-languages) for all available.\n"
]
},
{
"cell_type": "markdown",
"id": "fedf5cb8",
"metadata": {},
"source": [
"**Purpose:** This module takes a search query [1] and a passage [2] and calculates if the passage matches the query.\n",
"It can be used as an improvement for Elasticsearch Results and boosts the relevancy by up to 100%.\n"
]
},
{
"cell_type": "markdown",
"id": "146e3be4",
"metadata": {},
"source": [
"**Architecture:** On top of BERT there is a Densly Connected NN which takes the 768 Dimensional [CLS] Token as input and provides the output ([Arxiv](https://arxiv.org/abs/1901.04085)).\n"
]
},
{
"cell_type": "markdown",
"id": "772c5c82",
"metadata": {},
"source": [
"**Output:** Just a single value between between -10 and 10. Better matching query,passage pairs tend to have a higher a score.\n"
]
},
{
"cell_type": "markdown",
"id": "e5974e46",
"metadata": {},
"source": [
"## Intended uses & limitations\n",
"Both query[1] and passage[2] have to fit in 512 Tokens.\n",
"As you normally want to rerank the first dozens of search results keep in mind the inference time of approximately 300 ms/query.\n"
"This model is trained using the [**Microsoft MS Marco Dataset**](https://microsoft.github.io/msmarco/ \"Microsoft MS Marco\"). This training dataset contains approximately 400M tuples of a query, relevant and non-relevant passages. All datasets used for training and evaluating are listed in this [table](https://github.com/microsoft/MSMARCO-Passage-Ranking#data-information-and-formating). The used dataset for training is called *Train Triples Large*, while the evaluation was made on *Top 1000 Dev*. There are 6,900 queries in total in the development dataset, where each query is mapped to top 1,000 passage retrieved using BM25 from MS MARCO corpus.\n"
]
},
{
"cell_type": "markdown",
"id": "79af5e42",
"metadata": {},
"source": [
"> The model introduction and model weights originate from [https://huggingface.co/amberoad/bert-multilingual-passage-reranking-msmarco](https://huggingface.co/amberoad/bert-multilingual-passage-reranking-msmarco) and were converted to PaddlePaddle format for ease of use in PaddleNLP.\n"
"See the GPT2 model card for considerations on limitations and bias. See the GPT2 documentation for details on GPT2.\n"
]
},
{
"cell_type": "markdown",
"id": "8295e28d",
"metadata": {},
"source": [
"## Comparison to dbmdz/german-gpt2\n"
]
},
{
"cell_type": "markdown",
"id": "c0f50f67",
"metadata": {},
"source": [
"I evaluated both GerPT2-large and the other German GPT2, dbmdz/german-gpt2 on the [CC-100](http://data.statmt.org/cc-100/) dataset and on the German Wikipedia:\n"
"Thanks to [Hugging Face](https://huggingface.co) for awesome tools and infrastructure.\n",
"Huge thanks to [Artus Krohn-Grimberghe](https://twitter.com/artuskg) at [LYTiQ](https://www.lytiq.de/) for making this possible by sponsoring the resources used for training.\n",
"See the GPT2 model card for considerations on limitations and bias. See the GPT2 documentation for details on GPT2.\n"
]
},
{
"cell_type": "markdown",
"id": "5b4f950b",
"metadata": {},
"source": [
"## Comparison to dbmdz/german-gpt2\n"
]
},
{
"cell_type": "markdown",
"id": "95be6eb8",
"metadata": {},
"source": [
"I evaluated both GerPT2-large and the other German GPT2, dbmdz/german-gpt2 on the [CC-100](http://data.statmt.org/cc-100/) dataset and on the German Wikipedia:\n"
"Thanks to [Hugging Face](https://huggingface.co) for awesome tools and infrastructure.\n",
"Huge thanks to [Artus Krohn-Grimberghe](https://twitter.com/artuskg) at [LYTiQ](https://www.lytiq.de/) for making this possible by sponsoring the resources used for training.\n",
"\n",
"> The model introduction and model weights originate from [https://huggingface.co/benjamin/gerpt2-large](https://huggingface.co/benjamin/gerpt2-large) and were converted to PaddlePaddle format for ease of use in PaddleNLP.\n"
"Kaggle에 학습을 위해 정제한(아래 `clean`처리를 거친) Dataset을 공개하였습니다!\n"
]
},
{
"cell_type": "markdown",
"id": "95a25c77",
"metadata": {},
"source": [
"직접 다운받으셔서 다양한 Task에 학습을 진행해보세요 :)\n"
]
},
{
"cell_type": "markdown",
"id": "edd96db1",
"metadata": {},
"source": [
"공개된 한국어 BERT는 대부분 한국어 위키, 뉴스 기사, 책 등 잘 정제된 데이터를 기반으로 학습한 모델입니다. 한편, 실제로 NSMC와 같은 댓글형 데이터셋은 정제되지 않았고 구어체 특징에 신조어가 많으며, 오탈자 등 공식적인 글쓰기에서 나타나지 않는 표현들이 빈번하게 등장합니다.\n"
]
},
{
"cell_type": "markdown",
"id": "a2df738b",
"metadata": {},
"source": [
"KcBERT는 위와 같은 특성의 데이터셋에 적용하기 위해, 네이버 뉴스에서 댓글과 대댓글을 수집해, 토크나이저와 BERT모델을 처음부터 학습한 Pretrained BERT 모델입니다.\n"
]
},
{
"cell_type": "markdown",
"id": "a0eb4ad8",
"metadata": {},
"source": [
"KcBERT는 Huggingface의 Transformers 라이브러리를 통해 간편히 불러와 사용할 수 있습니다. (별도의 파일 다운로드가 필요하지 않습니다.)\n"
]
},
{
"cell_type": "markdown",
"id": "d1c07267",
"metadata": {},
"source": [
"## KcBERT Performance\n"
]
},
{
"cell_type": "markdown",
"id": "52872aa3",
"metadata": {},
"source": [
"- Finetune 코드는 https://github.com/Beomi/KcBERT-finetune 에서 찾아보실 수 있습니다.\n"
"Kaggle에 학습을 위해 정제한(아래 `clean`처리를 거친) Dataset을 공개하였습니다!\n"
]
},
{
"cell_type": "markdown",
"id": "691c1f27",
"metadata": {},
"source": [
"직접 다운받으셔서 다양한 Task에 학습을 진행해보세요 :)\n"
]
},
{
"cell_type": "markdown",
"id": "36ec915c",
"metadata": {},
"source": [
"공개된 한국어 BERT는 대부분 한국어 위키, 뉴스 기사, 책 등 잘 정제된 데이터를 기반으로 학습한 모델입니다. 한편, 실제로 NSMC와 같은 댓글형 데이터셋은 정제되지 않았고 구어체 특징에 신조어가 많으며, 오탈자 등 공식적인 글쓰기에서 나타나지 않는 표현들이 빈번하게 등장합니다.\n"
]
},
{
"cell_type": "markdown",
"id": "b5b8d7d7",
"metadata": {},
"source": [
"KcBERT는 위와 같은 특성의 데이터셋에 적용하기 위해, 네이버 뉴스에서 댓글과 대댓글을 수집해, 토크나이저와 BERT모델을 처음부터 학습한 Pretrained BERT 모델입니다.\n"
]
},
{
"cell_type": "markdown",
"id": "b0095da8",
"metadata": {},
"source": [
"KcBERT는 Huggingface의 Transformers 라이브러리를 통해 간편히 불러와 사용할 수 있습니다. (별도의 파일 다운로드가 필요하지 않습니다.)\n"
]
},
{
"cell_type": "markdown",
"id": "4bf51d97",
"metadata": {},
"source": [
"## KcBERT Performance\n"
]
},
{
"cell_type": "markdown",
"id": "9679c8b9",
"metadata": {},
"source": [
"- Finetune 코드는 https://github.com/Beomi/KcBERT-finetune 에서 찾아보실 수 있습니다.\n"
"> The model introduction and model weights originate from [https://huggingface.co/beomi/kcbert-base](https://huggingface.co/beomi/kcbert-base) and were converted to PaddlePaddle format for ease of use in PaddleNLP.\n"
"This is a very small distilled version of the bert-base-multilingual-cased model for Russian and English (45 MB, 12M parameters). There is also an **updated version of this model**, rubert-tiny2, with a larger vocabulary and better quality on practically all Russian NLU tasks.\n"
]
},
{
"cell_type": "markdown",
"id": "59944441",
"metadata": {},
"source": [
"This model is useful if you want to fine-tune it for a relatively simple Russian task (e.g. NER or sentiment classification), and you care more about speed and size than about accuracy. It is approximately x10 smaller and faster than a base-sized BERT. Its `[CLS]` embeddings can be used as a sentence representation aligned between Russian and English.\n"
]
},
{
"cell_type": "markdown",
"id": "c0e2918f",
"metadata": {},
"source": [
"It was trained on the [Yandex Translate corpus](https://translate.yandex.ru/corpus), [OPUS-100](https://huggingface.co/datasets/opus100) and Tatoeba, using MLM loss distilled from bert-base-multilingual-cased, translation ranking loss, and `[CLS]` embeddings distilled from LaBSE, rubert-base-cased-sentence, Laser and USE.\n"
]
},
{
"cell_type": "markdown",
"id": "b0c0158e",
"metadata": {},
"source": [
"There is a more detailed [description in Russian](https://habr.com/ru/post/562064/).\n"
]
},
{
"cell_type": "markdown",
"id": "28ce4026",
"metadata": {},
"source": [
"Sentence embeddings can be produced as follows:\n"
"This is a very small distilled version of the bert-base-multilingual-cased model for Russian and English (45 MB, 12M parameters). There is also an **updated version of this model**, rubert-tiny2, with a larger vocabulary and better quality on practically all Russian NLU tasks.\n"
]
},
{
"cell_type": "markdown",
"id": "5e7c8c35",
"metadata": {},
"source": [
"This model is useful if you want to fine-tune it for a relatively simple Russian task (e.g. NER or sentiment classification), and you care more about speed and size than about accuracy. It is approximately x10 smaller and faster than a base-sized BERT. Its `[CLS]` embeddings can be used as a sentence representation aligned between Russian and English.\n"
]
},
{
"cell_type": "markdown",
"id": "bc3c5717",
"metadata": {},
"source": [
"It was trained on the [Yandex Translate corpus](https://translate.yandex.ru/corpus), [OPUS-100](https://huggingface.co/datasets/opus100) and Tatoeba, using MLM loss (distilled from bert-base-multilingual-cased\n",
"), translation ranking loss, and `[CLS]` embeddings distilled from LaBSE, rubert-base-cased-sentence, Laser and USE.\n"
]
},
{
"cell_type": "markdown",
"id": "2db0a3ee",
"metadata": {},
"source": [
"There is a more detailed [description in Russian](https://habr.com/ru/post/562064/).\n"
]
},
{
"cell_type": "markdown",
"id": "c3a52477",
"metadata": {},
"source": [
"Sentence embeddings can be produced as follows:\n"
"> The model introduction and model weights originate from [https://huggingface.co/cointegrated/rubert-tiny](https://huggingface.co/cointegrated/rubert-tiny) and were converted to PaddlePaddle format for ease of use in PaddleNLP.\n"
"This is an updated version of cointegrated/rubert-tiny: a small Russian BERT-based encoder with high-quality sentence embeddings. This [post in Russian](https://habr.com/ru/post/669674/) gives more details.\n"
]
},
{
"cell_type": "markdown",
"id": "08d9a049",
"metadata": {},
"source": [
"The differences from the previous version include:\n",
"- a larger vocabulary: 83828 tokens instead of 29564;\n",
"- larger supported sequences: 2048 instead of 512;\n",
"- sentence embeddings approximate LaBSE closer than before;\n",
"- meaningful segment embeddings (tuned on the NLI task)\n",
"- the model is focused only on Russian.\n"
]
},
{
"cell_type": "markdown",
"id": "8a7ba50b",
"metadata": {},
"source": [
"The model should be used as is to produce sentence embeddings (e.g. for KNN classification of short texts) or fine-tuned for a downstream task.\n"
]
},
{
"cell_type": "markdown",
"id": "184e1cc6",
"metadata": {},
"source": [
"Sentence embeddings can be produced as follows:\n"
"This is an updated version of cointegrated/rubert-tiny: a small Russian BERT-based encoder with high-quality sentence embeddings. This [post in Russian](https://habr.com/ru/post/669674/) gives more details.\n"
]
},
{
"cell_type": "markdown",
"id": "801acf5c",
"metadata": {},
"source": [
"The differences from the previous version include:\n",
"- a larger vocabulary: 83828 tokens instead of 29564;\n",
"- larger supported sequences: 2048 instead of 512;\n",
"- sentence embeddings approximate LaBSE closer than before;\n",
"- meaningful segment embeddings (tuned on the NLI task)\n",
"- the model is focused only on Russian.\n"
]
},
{
"cell_type": "markdown",
"id": "f2c7dbc1",
"metadata": {},
"source": [
"The model should be used as is to produce sentence embeddings (e.g. for KNN classification of short texts) or fine-tuned for a downstream task.\n"
]
},
{
"cell_type": "markdown",
"id": "9ff63df2",
"metadata": {},
"source": [
"Sentence embeddings can be produced as follows:\n"
"> The model introduction and model weights originate from [https://huggingface.co/cointegrated/rubert-tiny2](https://huggingface.co/cointegrated/rubert-tiny2) and were converted to PaddlePaddle format for ease of use in PaddleNLP.\n"
"This model was trained on the [MS Marco Passage Ranking](https://github.com/microsoft/MSMARCO-Passage-Ranking) task.\n"
]
},
{
"cell_type": "markdown",
"id": "0e8686b5",
"metadata": {},
"source": [
"The model can be used for Information Retrieval: Given a query, encode the query will all possible passages (e.g. retrieved with ElasticSearch). Then sort the passages in a decreasing order. See [SBERT.net Retrieve & Re-rank](https://www.sbert.net/examples/applications/retrieve_rerank/README.html) for more details. The training code is available here: [SBERT.net Training MS Marco](https://github.com/UKPLab/sentence-transformers/tree/master/examples/training/ms_marco)\n"
"In the following table, we provide various pre-trained Cross-Encoders together with their performance on the [TREC Deep Learning 2019](https://microsoft.github.io/TREC-2019-Deep-Learning/) and the [MS Marco Passage Reranking](https://github.com/microsoft/MSMARCO-Passage-Ranking/) dataset.\n"
"This model was trained on the [MS Marco Passage Ranking](https://github.com/microsoft/MSMARCO-Passage-Ranking) task.\n"
]
},
{
"cell_type": "markdown",
"id": "1535e90f",
"metadata": {},
"source": [
"The model can be used for Information Retrieval: Given a query, encode the query will all possible passages (e.g. retrieved with ElasticSearch). Then sort the passages in a decreasing order. See [SBERT.net Retrieve & Re-rank](https://www.sbert.net/examples/applications/retrieve_rerank/README.html) for more details. The training code is available here: [SBERT.net Training MS Marco](https://github.com/UKPLab/sentence-transformers/tree/master/examples/training/ms_marco)\n"
"In the following table, we provide various pre-trained Cross-Encoders together with their performance on the [TREC Deep Learning 2019](https://microsoft.github.io/TREC-2019-Deep-Learning/) and the [MS Marco Passage Reranking](https://github.com/microsoft/MSMARCO-Passage-Ranking/) dataset.\n"
"> The model introduction and model weights originate from [https://huggingface.co/cross-encoder/ms-marco-MiniLM-L-12-v2](https://huggingface.co/cross-encoder/ms-marco-MiniLM-L-12-v2) and were converted to PaddlePaddle format for ease of use in PaddleNLP.\n"
"This model was trained on the [MS Marco Passage Ranking](https://github.com/microsoft/MSMARCO-Passage-Ranking) task.\n"
]
},
{
"cell_type": "markdown",
"id": "dcf2e434",
"metadata": {},
"source": [
"The model can be used for Information Retrieval: Given a query, encode the query will all possible passages (e.g. retrieved with ElasticSearch). Then sort the passages in a decreasing order. See [SBERT.net Retrieve & Re-rank](https://www.sbert.net/examples/applications/retrieve_rerank/README.html) for more details. The training code is available here: [SBERT.net Training MS Marco](https://github.com/UKPLab/sentence-transformers/tree/master/examples/training/ms_marco)\n"
"In the following table, we provide various pre-trained Cross-Encoders together with their performance on the [TREC Deep Learning 2019](https://microsoft.github.io/TREC-2019-Deep-Learning/) and the [MS Marco Passage Reranking](https://github.com/microsoft/MSMARCO-Passage-Ranking/) dataset.\n"
"This model was trained on the [MS Marco Passage Ranking](https://github.com/microsoft/MSMARCO-Passage-Ranking) task.\n"
]
},
{
"cell_type": "markdown",
"id": "185acb77",
"metadata": {},
"source": [
"The model can be used for Information Retrieval: Given a query, encode the query will all possible passages (e.g. retrieved with ElasticSearch). Then sort the passages in a decreasing order. See [SBERT.net Retrieve & Re-rank](https://www.sbert.net/examples/applications/retrieve_rerank/README.html) for more details. The training code is available here: [SBERT.net Training MS Marco](https://github.com/UKPLab/sentence-transformers/tree/master/examples/training/ms_marco)\n"
"In the following table, we provide various pre-trained Cross-Encoders together with their performance on the [TREC Deep Learning 2019](https://microsoft.github.io/TREC-2019-Deep-Learning/) and the [MS Marco Passage Reranking](https://github.com/microsoft/MSMARCO-Passage-Ranking/) dataset.\n"
"> The model introduction and model weights originate from [https://huggingface.co/cross-encoder/ms-marco-TinyBERT-L-2](https://huggingface.co/cross-encoder/ms-marco-TinyBERT-L-2) and were converted to PaddlePaddle format for ease of use in PaddleNLP.\n"
"# Cross-Encoder for Natural Language Inference\n",
"This model was trained using [SentenceTransformers](https://sbert.net) [Cross-Encoder](https://www.sbert.net/examples/applications/cross-encoder/README.html) class.\n"
]
},
{
"cell_type": "markdown",
"id": "e01fe90a",
"metadata": {},
"source": [
"## Training Data\n",
"The model was trained on the [SNLI](https://nlp.stanford.edu/projects/snli/) and [MultiNLI](https://cims.nyu.edu/~sbowman/multinli/) datasets. For a given sentence pair, it will output three scores corresponding to the labels: contradiction, entailment, neutral.\n"
]
},
{
"cell_type": "markdown",
"id": "ff850419",
"metadata": {},
"source": [
"## Performance\n",
"For evaluation results, see [SBERT.net - Pretrained Cross-Encoder](https://www.sbert.net/docs/pretrained_cross-encoders.html#nli).\n"
"# Cross-Encoder for Natural Language Inference\n",
"This model was trained using [SentenceTransformers](https://sbert.net) [Cross-Encoder](https://www.sbert.net/examples/applications/cross-encoder/README.html) class.\n"
]
},
{
"cell_type": "markdown",
"id": "daf01f92",
"metadata": {},
"source": [
"## Training Data\n",
"The model was trained on the [SNLI](https://nlp.stanford.edu/projects/snli/) and [MultiNLI](https://cims.nyu.edu/~sbowman/multinli/) datasets. For a given sentence pair, it will output three scores corresponding to the labels: contradiction, entailment, neutral.\n"
]
},
{
"cell_type": "markdown",
"id": "805a7294",
"metadata": {},
"source": [
"## Performance\n",
"For evaluation results, see [SBERT.net - Pretrained Cross-Encoder](https://www.sbert.net/docs/pretrained_cross-encoders.html#nli).\n"
"> The model introduction and model weights originate from [https://huggingface.co/cross-encoder/nli-MiniLM2-L6-H768](https://huggingface.co/cross-encoder/nli-MiniLM2-L6-H768) and were converted to PaddlePaddle format for ease of use in PaddleNLP.\n"
"# Cross-Encoder for Natural Language Inference\n",
"This model was trained using [SentenceTransformers](https://sbert.net) [Cross-Encoder](https://www.sbert.net/examples/applications/cross-encoder/README.html) class.\n"
]
},
{
"cell_type": "markdown",
"id": "ec682169",
"metadata": {},
"source": [
"## Training Data\n",
"The model was trained on the [SNLI](https://nlp.stanford.edu/projects/snli/) and [MultiNLI](https://cims.nyu.edu/~sbowman/multinli/) datasets. For a given sentence pair, it will output three scores corresponding to the labels: contradiction, entailment, neutral.\n"
]
},
{
"cell_type": "markdown",
"id": "ba993930",
"metadata": {},
"source": [
"## Performance\n",
"For evaluation results, see [SBERT.net - Pretrained Cross-Encoder](https://www.sbert.net/docs/pretrained_cross-encoders.html#nli).\n"
"# Cross-Encoder for Natural Language Inference\n",
"This model was trained using [SentenceTransformers](https://sbert.net) [Cross-Encoder](https://www.sbert.net/examples/applications/cross-encoder/README.html) class.\n"
]
},
{
"cell_type": "markdown",
"id": "f2d88a35",
"metadata": {},
"source": [
"## Training Data\n",
"The model was trained on the [SNLI](https://nlp.stanford.edu/projects/snli/) and [MultiNLI](https://cims.nyu.edu/~sbowman/multinli/) datasets. For a given sentence pair, it will output three scores corresponding to the labels: contradiction, entailment, neutral.\n"
]
},
{
"cell_type": "markdown",
"id": "d982bc91",
"metadata": {},
"source": [
"## Performance\n",
"For evaluation results, see [SBERT.net - Pretrained Cross-Encoder](https://www.sbert.net/docs/pretrained_cross-encoders.html#nli).\n"
"> The model introduction and model weights originate from [https://huggingface.co/cross-encoder/nli-distilroberta-base](https://huggingface.co/cross-encoder/nli-distilroberta-base) and were converted to PaddlePaddle format for ease of use in PaddleNLP.\n"
"# Cross-Encoder for Natural Language Inference\n",
"This model was trained using [SentenceTransformers](https://sbert.net) [Cross-Encoder](https://www.sbert.net/examples/applications/cross-encoder/README.html) class.\n"
]
},
{
"cell_type": "markdown",
"id": "26cf9863",
"metadata": {},
"source": [
"## Training Data\n",
"The model was trained on the [SNLI](https://nlp.stanford.edu/projects/snli/) and [MultiNLI](https://cims.nyu.edu/~sbowman/multinli/) datasets. For a given sentence pair, it will output three scores corresponding to the labels: contradiction, entailment, neutral.\n"
]
},
{
"cell_type": "markdown",
"id": "913c77b3",
"metadata": {},
"source": [
"## Performance\n",
"For evaluation results, see [SBERT.net - Pretrained Cross-Encoder](https://www.sbert.net/docs/pretrained_cross-encoders.html#nli).\n"
"# Cross-Encoder for Natural Language Inference\n",
"This model was trained using [SentenceTransformers](https://sbert.net) [Cross-Encoder](https://www.sbert.net/examples/applications/cross-encoder/README.html) class.\n"
]
},
{
"cell_type": "markdown",
"id": "6b47f4c6",
"metadata": {},
"source": [
"## Training Data\n",
"The model was trained on the [SNLI](https://nlp.stanford.edu/projects/snli/) and [MultiNLI](https://cims.nyu.edu/~sbowman/multinli/) datasets. For a given sentence pair, it will output three scores corresponding to the labels: contradiction, entailment, neutral.\n"
]
},
{
"cell_type": "markdown",
"id": "39bc9190",
"metadata": {},
"source": [
"## Performance\n",
"For evaluation results, see [SBERT.net - Pretrained Cross-Encoder](https://www.sbert.net/docs/pretrained_cross-encoders.html#nli).\n"
"The model introduction and model weights originate from [https://huggingface.co/cross-encoder/nli-roberta-base](https://huggingface.co/cross-encoder/nli-roberta-base) and were converted to PaddlePaddle format for ease of use in PaddleNLP.\n"
"# Cross-Encoder for Quora Duplicate Questions Detection\n",
"This model was trained using [SentenceTransformers](https://sbert.net) [Cross-Encoder](https://www.sbert.net/examples/applications/cross-encoder/README.html) class.\n"
]
},
{
"cell_type": "markdown",
"id": "dcc58a5d",
"metadata": {},
"source": [
"## Training Data\n",
"This model was trained on the [Quora Duplicate Questions](https://www.quora.com/q/quoradata/First-Quora-Dataset-Release-Question-Pairs) dataset. The model will predict a score between 0 and 1 how likely the two given questions are duplicates.\n"
]
},
{
"cell_type": "markdown",
"id": "4f967914",
"metadata": {},
"source": [
"Note: The model is not suitable to estimate the similarity of questions, e.g. the two questions \"How to learn Java\" and \"How to learn Python\" will result in a rahter low score, as these are not duplicates.\n"
"# Cross-Encoder for Quora Duplicate Questions Detection\n",
"This model was trained using [SentenceTransformers](https://sbert.net) [Cross-Encoder](https://www.sbert.net/examples/applications/cross-encoder/README.html) class.\n"
]
},
{
"cell_type": "markdown",
"id": "71def254",
"metadata": {},
"source": [
"## Training Data\n",
"This model was trained on the [Quora Duplicate Questions](https://www.quora.com/q/quoradata/First-Quora-Dataset-Release-Question-Pairs) dataset. The model will predict a score between 0 and 1 how likely the two given questions are duplicates.\n"
]
},
{
"cell_type": "markdown",
"id": "10f2b17c",
"metadata": {},
"source": [
"Note: The model is not suitable to estimate the similarity of questions, e.g. the two questions \"How to learn Java\" and \"How to learn Python\" will result in a rahter low score, as these are not duplicates.\n"
"> The model introduction and model weights originate from [https://huggingface.co/cross-encoder/quora-distilroberta-base](https://huggingface.co/cross-encoder/quora-distilroberta-base) and were converted to PaddlePaddle format for ease of use in PaddleNLP.\n"
"# Cross-Encoder for Quora Duplicate Questions Detection\n",
"This model was trained using [SentenceTransformers](https://sbert.net) [Cross-Encoder](https://www.sbert.net/examples/applications/cross-encoder/README.html) class.\n"
]
},
{
"cell_type": "markdown",
"id": "e743234a",
"metadata": {},
"source": [
"## Training Data\n",
"This model was trained on the [Quora Duplicate Questions](https://www.quora.com/q/quoradata/First-Quora-Dataset-Release-Question-Pairs) dataset. The model will predict a score between 0 and 1 how likely the two given questions are duplicates.\n"
]
},
{
"cell_type": "markdown",
"id": "f755b608",
"metadata": {},
"source": [
"Note: The model is not suitable to estimate the similarity of questions, e.g. the two questions \"How to learn Java\" and \"How to learn Python\" will result in a rahter low score, as these are not duplicates.\n"
"# Cross-Encoder for Quora Duplicate Questions Detection\n",
"This model was trained using [SentenceTransformers](https://sbert.net) [Cross-Encoder](https://www.sbert.net/examples/applications/cross-encoder/README.html) class.\n"
]
},
{
"cell_type": "markdown",
"id": "36bf7390",
"metadata": {},
"source": [
"## Training Data\n",
"This model was trained on the [Quora Duplicate Questions](https://www.quora.com/q/quoradata/First-Quora-Dataset-Release-Question-Pairs) dataset. The model will predict a score between 0 and 1 how likely the two given questions are duplicates.\n"
]
},
{
"cell_type": "markdown",
"id": "5aa29571",
"metadata": {},
"source": [
"Note: The model is not suitable to estimate the similarity of questions, e.g. the two questions \"How to learn Java\" and \"How to learn Python\" will result in a rahter low score, as these are not duplicates.\n"
"> The model introduction and model weights originate from [https://huggingface.co/cross-encoder/quora-roberta-base](https://huggingface.co/cross-encoder/quora-roberta-base) and were converted to PaddlePaddle format for ease of use in PaddleNLP.\n"
"# Cross-Encoder for Quora Duplicate Questions Detection\n",
"This model was trained using [SentenceTransformers](https://sbert.net) [Cross-Encoder](https://www.sbert.net/examples/applications/cross-encoder/README.html) class.\n"
]
},
{
"cell_type": "markdown",
"id": "4fc17643",
"metadata": {},
"source": [
"## Training Data\n",
"This model was trained on the [STS benchmark dataset](http://ixa2.si.ehu.eus/stswiki/index.php/STSbenchmark). The model will predict a score between 0 and 1 how for the semantic similarity of two sentences.\n"
]
},
{
"cell_type": "markdown",
"id": "f66fb11e",
"metadata": {},
"source": [
"## Usage and Performance\n"
]
},
{
"cell_type": "markdown",
"id": "fd12128b",
"metadata": {},
"source": [
"Pre-trained models can be used like this:\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "d0d04e39",
"metadata": {},
"outputs": [],
"source": [
"!pip install --upgrade paddlenlp"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "d07e31aa",
"metadata": {
"collapsed": true
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/root/miniconda3/envs/paddle/lib/python3.7/site-packages/tqdm/auto.py:22: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
" from .autonotebook import tqdm as notebook_tqdm\n",
"\u001b[32m[2022-11-21 02:38:07,127] [ INFO]\u001b[0m - Downloading model_config.json from https://bj.bcebos.com/paddlenlp/models/community/cross-encoder/stsb-TinyBERT-L-4/model_config.json\u001b[0m\n",
"\u001b[32m[2022-11-21 02:38:07,197] [ INFO]\u001b[0m - We are using <class 'paddlenlp.transformers.bert.modeling.BertModel'> to load 'cross-encoder/stsb-TinyBERT-L-4'.\u001b[0m\n",
"\u001b[32m[2022-11-21 02:38:07,198] [ INFO]\u001b[0m - Downloading https://bj.bcebos.com/paddlenlp/models/community/cross-encoder/stsb-TinyBERT-L-4/model_state.pdparams and saved to /root/.paddlenlp/models/cross-encoder/stsb-TinyBERT-L-4\u001b[0m\n",
"\u001b[32m[2022-11-21 02:38:07,198] [ INFO]\u001b[0m - Downloading model_state.pdparams from https://bj.bcebos.com/paddlenlp/models/community/cross-encoder/stsb-TinyBERT-L-4/model_state.pdparams\u001b[0m\n",
"\u001b[32m[2022-11-21 02:38:09,661] [ INFO]\u001b[0m - Weights from pretrained model not used in BertModel: ['classifier.weight', 'classifier.bias']\u001b[0m\n"
"# Cross-Encoder for Quora Duplicate Questions Detection\n",
"This model was trained using [SentenceTransformers](https://sbert.net) [Cross-Encoder](https://www.sbert.net/examples/applications/cross-encoder/README.html) class.\n"
]
},
{
"cell_type": "markdown",
"id": "c3be9ab9",
"metadata": {},
"source": [
"## Training Data\n",
"This model was trained on the [STS benchmark dataset](http://ixa2.si.ehu.eus/stswiki/index.php/STSbenchmark). The model will predict a score between 0 and 1 how for the semantic similarity of two sentences.\n"
"You can use this model also without sentence_transformers and by just using ``AutoModel`` class\n",
"> The model introduction and model weights originate from [https://huggingface.co/cross-encoder/stsb-TinyBERT-L-4](https://huggingface.co/cross-encoder/stsb-TinyBERT-L-4) and were converted to PaddlePaddle format for ease of use in PaddleNLP.\n"
"# Cross-Encoder for Quora Duplicate Questions Detection\n",
"This model was trained using [SentenceTransformers](https://sbert.net) [Cross-Encoder](https://www.sbert.net/examples/applications/cross-encoder/README.html) class.\n"
]
},
{
"cell_type": "markdown",
"id": "c62db00c",
"metadata": {},
"source": [
"## Training Data\n",
"This model was trained on the [STS benchmark dataset](http://ixa2.si.ehu.eus/stswiki/index.php/STSbenchmark). The model will predict a score between 0 and 1 how for the semantic similarity of two sentences.\n"
"# Cross-Encoder for Quora Duplicate Questions Detection\n",
"This model was trained using [SentenceTransformers](https://sbert.net) [Cross-Encoder](https://www.sbert.net/examples/applications/cross-encoder/README.html) class.\n"
]
},
{
"cell_type": "markdown",
"id": "6e6d61e4",
"metadata": {},
"source": [
"## Training Data\n",
"This model was trained on the [STS benchmark dataset](http://ixa2.si.ehu.eus/stswiki/index.php/STSbenchmark). The model will predict a score between 0 and 1 how for the semantic similarity of two sentences.\n"
"> The model introduction and model weights originate from [https://huggingface.co/cross-encoder/stsb-distilroberta-base](https://huggingface.co/cross-encoder/stsb-distilroberta-base) and were converted to PaddlePaddle format for ease of use in PaddleNLP.\n"
"# Cross-Encoder for Quora Duplicate Questions Detection\n",
"This model was trained using [SentenceTransformers](https://sbert.net) [Cross-Encoder](https://www.sbert.net/examples/applications/cross-encoder/README.html) class.\n"
]
},
{
"cell_type": "markdown",
"id": "6e5557d3",
"metadata": {},
"source": [
"## Training Data\n",
"This model was trained on the [STS benchmark dataset](http://ixa2.si.ehu.eus/stswiki/index.php/STSbenchmark). The model will predict a score between 0 and 1 how for the semantic similarity of two sentences.\n"
"# Cross-Encoder for Quora Duplicate Questions Detection\n",
"This model was trained using [SentenceTransformers](https://sbert.net) [Cross-Encoder](https://www.sbert.net/examples/applications/cross-encoder/README.html) class.\n"
]
},
{
"cell_type": "markdown",
"id": "5406455e",
"metadata": {},
"source": [
"## Training Data\n",
"This model was trained on the [STS benchmark dataset](http://ixa2.si.ehu.eus/stswiki/index.php/STSbenchmark). The model will predict a score between 0 and 1 how for the semantic similarity of two sentences.\n"
"> The model introduction and model weights originate from [https://huggingface.co/cross-encoder/stsb-roberta-base](https://huggingface.co/cross-encoder/stsb-roberta-base) and were converted to PaddlePaddle format for ease of use in PaddleNLP.\n"
"# Cross-Encoder for Quora Duplicate Questions Detection\n",
"This model was trained using [SentenceTransformers](https://sbert.net) [Cross-Encoder](https://www.sbert.net/examples/applications/cross-encoder/README.html) class.\n"
]
},
{
"cell_type": "markdown",
"id": "e4d8f5f6",
"metadata": {},
"source": [
"## Training Data\n",
"This model was trained on the [STS benchmark dataset](http://ixa2.si.ehu.eus/stswiki/index.php/STSbenchmark). The model will predict a score between 0 and 1 how for the semantic similarity of two sentences.\n"
"# Cross-Encoder for Quora Duplicate Questions Detection\n",
"This model was trained using [SentenceTransformers](https://sbert.net) [Cross-Encoder](https://www.sbert.net/examples/applications/cross-encoder/README.html) class.\n"
]
},
{
"cell_type": "markdown",
"id": "92f483ed",
"metadata": {},
"source": [
"## Training Data\n",
"This model was trained on the [STS benchmark dataset](http://ixa2.si.ehu.eus/stswiki/index.php/STSbenchmark). The model will predict a score between 0 and 1 how for the semantic similarity of two sentences.\n"
"> The model introduction and model weights originate from [https://huggingface.co/cross-encoder/stsb-roberta-large](https://huggingface.co/cross-encoder/stsb-roberta-large) and were converted to PaddlePaddle format for ease of use in PaddleNLP.\n"
"> The model introduction and model weights originate from [https://huggingface.co/dbmdz/bert-base-german-cased](https://huggingface.co/dbmdz/bert-base-german-cased) and were converted to PaddlePaddle format for ease of use in PaddleNLP.\n"
"> The model introduction and model weights originate from [https://huggingface.co/dbmdz/bert-base-german-uncased](https://huggingface.co/dbmdz/bert-base-german-uncased) and were converted to PaddlePaddle format for ease of use in PaddleNLP.\n"
"> The model introduction and model weights originate from [https://huggingface.co/dbmdz/bert-base-italian-uncased](https://huggingface.co/dbmdz/bert-base-italian-uncased) and were converted to PaddlePaddle format for ease of use in PaddleNLP.\n"
"> The model introduction and model weights originate from [https://huggingface.co/dbmdz/bert-base-italian-xxl-cased](https://huggingface.co/dbmdz/bert-base-italian-xxl-cased) and were converted to PaddlePaddle format for ease of use in PaddleNLP.\n"
"> The model introduction and model weights originate from [https://huggingface.co/dbmdz/bert-base-turkish-128k-cased](https://huggingface.co/dbmdz/bert-base-turkish-128k-cased) and were converted to PaddlePaddle format for ease of use in PaddleNLP.\n"
"> The model introduction and model weights originate from [https://huggingface.co/dbmdz/bert-base-turkish-cased](https://huggingface.co/dbmdz/bert-base-turkish-cased) and were converted to PaddlePaddle format for ease of use in PaddleNLP.\n"
"> The model introduction and model weights originate from [https://huggingface.co/dbmdz/bert-base-turkish-uncased](https://huggingface.co/dbmdz/bert-base-turkish-uncased) and were converted to PaddlePaddle format for ease of use in PaddleNLP.\n"
"An generative AI made using [microsoft/DialoGPT-small](https://huggingface.co/microsoft/DialoGPT-small).\n"
]
},
{
"cell_type": "markdown",
"id": "008bcb8d",
"metadata": {},
"source": [
"Recommended to use along with an [AIML Chatbot](https://github.com/deepsarda/Aeona-Aiml) to reduce load, get better replies, add name and personality to your bot.\n",
"Using an AIML Chatbot will allow you to hardcode some replies also.\n"
]
},
{
"cell_type": "markdown",
"id": "b4bfb9cd",
"metadata": {},
"source": [
"## Evaluation\n",
"Below is a comparison of Aeona vs. other baselines on the mixed dataset given above using automatic evaluation metrics.\n"
"An generative AI made using [microsoft/DialoGPT-small](https://huggingface.co/microsoft/DialoGPT-small).\n"
]
},
{
"cell_type": "markdown",
"id": "a65479b8",
"metadata": {},
"source": [
"Recommended to use along with an [AIML Chatbot](https://github.com/deepsarda/Aeona-Aiml) to reduce load, get better replies, add name and personality to your bot.\n",
"Using an AIML Chatbot will allow you to hardcode some replies also.\n"
]
},
{
"cell_type": "markdown",
"id": "ea390590",
"metadata": {},
"source": [
"## Evaluation\n",
"Below is a comparison of Aeona vs. other baselines on the mixed dataset given above using automatic evaluation metrics.\n"
"> The model introduction and model weights originate from [https://huggingface.co/deepparag/Aeona](https://huggingface.co/deepparag/Aeona) and were converted to PaddlePaddle format for ease of use in PaddleNLP.\n"
"We bring NLP to the industry via open source!\n",
"Our focus: Industry specific language models & large scale QA systems.\n"
]
},
{
"cell_type": "markdown",
"id": "d65be46f",
"metadata": {},
"source": [
"> The model introduction and model weights originate from [https://huggingface.co/deepset/roberta-base-squad2-distilled](https://huggingface.co/deepset/roberta-base-squad2-distilled) and were converted to PaddlePaddle format for ease of use in PaddleNLP.\n"
"**bert-base-NER** is a fine-tuned BERT model that is ready to use for **Named Entity Recognition** and achieves **state-of-the-art performance** for the NER task. It has been trained to recognize four types of entities: location (LOC), organizations (ORG), person (PER) and Miscellaneous (MISC).\n"
]
},
{
"cell_type": "markdown",
"id": "58641459",
"metadata": {},
"source": [
"Specifically, this model is a *bert-base-cased* model that was fine-tuned on the English version of the standard [CoNLL-2003 Named Entity Recognition](https://www.aclweb.org/anthology/W03-0419.pdf) dataset.\n"
]
},
{
"cell_type": "markdown",
"id": "9da0ddda",
"metadata": {},
"source": [
"If you'd like to use a larger BERT-large model fine-tuned on the same dataset, a **bert-large-NER** version is also available.\n"
"**bert-base-NER** is a fine-tuned BERT model that is ready to use for **Named Entity Recognition** and achieves **state-of-the-art performance** for the NER task. It has been trained to recognize four types of entities: location (LOC), organizations (ORG), person (PER) and Miscellaneous (MISC).\n"
]
},
{
"cell_type": "markdown",
"id": "738f98db",
"metadata": {},
"source": [
"Specifically, this model is a *bert-base-cased* model that was fine-tuned on the English version of the standard [CoNLL-2003 Named Entity Recognition](https://www.aclweb.org/anthology/W03-0419.pdf) dataset.\n"
]
},
{
"cell_type": "markdown",
"id": "03c5db03",
"metadata": {},
"source": [
"If you'd like to use a larger BERT-large model fine-tuned on the same dataset, a **bert-large-NER** version is also available.\n"
"> The model introduction and model weights originate from [https://huggingface.co/dslim/bert-base-NER](https://huggingface.co/dslim/bert-base-NER) and were converted to PaddlePaddle format for ease of use in PaddleNLP.\n"
"**bert-large-NER** is a fine-tuned BERT model that is ready to use for **Named Entity Recognition** and achieves **state-of-the-art performance** for the NER task. It has been trained to recognize four types of entities: location (LOC), organizations (ORG), person (PER) and Miscellaneous (MISC).\n"
]
},
{
"cell_type": "markdown",
"id": "2d53a70b",
"metadata": {},
"source": [
"Specifically, this model is a *bert-large-cased* model that was fine-tuned on the English version of the standard [CoNLL-2003 Named Entity Recognition](https://www.aclweb.org/anthology/W03-0419.pdf) dataset.\n"
]
},
{
"cell_type": "markdown",
"id": "60c2ceb7",
"metadata": {},
"source": [
"If you'd like to use a smaller BERT model fine-tuned on the same dataset, a **bert-base-NER** version is also available.\n"
"**bert-large-NER** is a fine-tuned BERT model that is ready to use for **Named Entity Recognition** and achieves **state-of-the-art performance** for the NER task. It has been trained to recognize four types of entities: location (LOC), organizations (ORG), person (PER) and Miscellaneous (MISC).\n"
]
},
{
"cell_type": "markdown",
"id": "84ed4f8f",
"metadata": {},
"source": [
"Specifically, this model is a *bert-large-cased* model that was fine-tuned on the English version of the standard [CoNLL-2003 Named Entity Recognition](https://www.aclweb.org/anthology/W03-0419.pdf) dataset.\n"
]
},
{
"cell_type": "markdown",
"id": "b4acb8a4",
"metadata": {},
"source": [
"If you'd like to use a smaller BERT model fine-tuned on the same dataset, a **bert-base-NER** version is also available.\n"
"> The model introduction and model weights originate from [https://huggingface.co/dslim/bert-large-NER](https://huggingface.co/dslim/bert-large-NER) and were converted to PaddlePaddle format for ease of use in PaddleNLP.\n"
"- We'd like to thank [Sampo Pyysalo](https://github.com/spyysalo) from TurkuNLP for helping us out with the compute needed to pretrain the v1.0 BERT models. He's awesome!\n",
"- We'd like to thank [Sampo Pyysalo](https://github.com/spyysalo) from TurkuNLP for helping us out with the compute needed to pretrain the v1.0 BERT models. He's awesome!\n",
"> The model introduction and model weights originate from [https://huggingface.co/dumitrescustefan/bert-base-romanian-cased-v1](https://huggingface.co/dumitrescustefan/bert-base-romanian-cased-v1) and were converted to PaddlePaddle format for ease of use in PaddleNLP.\n"
"The BERT **base**, **uncased** model for Romanian, trained on a 15GB corpus, version \n"
"- We'd like to thank [Sampo Pyysalo](https://github.com/spyysalo) from TurkuNLP for helping us out with the compute needed to pretrain the v1.0 BERT models. He's awesome!\n",
"The BERT **base**, **uncased** model for Romanian, trained on a 15GB corpus, version \n"
"- We'd like to thank [Sampo Pyysalo](https://github.com/spyysalo) from TurkuNLP for helping us out with the compute needed to pretrain the v1.0 BERT models. He's awesome!\n",
"> The model introduction and model weights originate from [https://huggingface.co/dumitrescustefan/bert-base-romanian-uncased-v1](https://huggingface.co/dumitrescustefan/bert-base-romanian-uncased-v1) and were converted to PaddlePaddle format for ease of use in PaddleNLP.\n"
"The [Publicly Available Clinical BERT Embeddings](https://arxiv.org/abs/1904.03323) paper contains four unique clinicalBERT models: initialized with BERT-Base (`cased_L-12_H-768_A-12`) or BioBERT (`BioBERT-Base v1.0 + PubMed 200K + PMC 270K`) & trained on either all MIMIC notes or only discharge summaries.\n"
]
},
{
"cell_type": "markdown",
"id": "24aaa0b1",
"metadata": {},
"source": [
"This model card describes the Bio+Clinical BERT model, which was initialized from [BioBERT](https://arxiv.org/abs/1901.08746) & trained on all MIMIC notes.\n"
"The [Publicly Available Clinical BERT Embeddings](https://arxiv.org/abs/1904.03323) paper contains four unique clinicalBERT models: initialized with BERT-Base (`cased_L-12_H-768_A-12`) or BioBERT (`BioBERT-Base v1.0 + PubMed 200K + PMC 270K`) & trained on either all MIMIC notes or only discharge summaries.\n"
]
},
{
"cell_type": "markdown",
"id": "9ab166b8",
"metadata": {},
"source": [
"This model card describes the Bio+Clinical BERT model, which was initialized from [BioBERT](https://arxiv.org/abs/1901.08746) & trained on all MIMIC notes.\n"
"> The model introduction and model weights originate from [https://huggingface.co/emilyalsentzer/Bio_ClinicalBERT](https://huggingface.co/emilyalsentzer/Bio_ClinicalBERT) and were converted to PaddlePaddle format for ease of use in PaddleNLP.\n"
"# ClinicalBERT - Bio + Discharge Summary BERT Model\n"
]
},
{
"cell_type": "markdown",
"id": "e2d38260",
"metadata": {},
"source": [
"The [Publicly Available Clinical BERT Embeddings](https://arxiv.org/abs/1904.03323) paper contains four unique clinicalBERT models: initialized with BERT-Base (`cased_L-12_H-768_A-12`) or BioBERT (`BioBERT-Base v1.0 + PubMed 200K + PMC 270K`) & trained on either all MIMIC notes or only discharge summaries.\n"
]
},
{
"cell_type": "markdown",
"id": "1c92755e",
"metadata": {},
"source": [
"This model card describes the Bio+Discharge Summary BERT model, which was initialized from [BioBERT](https://arxiv.org/abs/1901.08746) & trained on only discharge summaries from MIMIC.\n"
"# ClinicalBERT - Bio + Discharge Summary BERT Model\n"
]
},
{
"cell_type": "markdown",
"id": "4d8e4f1f",
"metadata": {},
"source": [
"The [Publicly Available Clinical BERT Embeddings](https://arxiv.org/abs/1904.03323) paper contains four unique clinicalBERT models: initialized with BERT-Base (`cased_L-12_H-768_A-12`) or BioBERT (`BioBERT-Base v1.0 + PubMed 200K + PMC 270K`) & trained on either all MIMIC notes or only discharge summaries.\n"
]
},
{
"cell_type": "markdown",
"id": "83bf8287",
"metadata": {},
"source": [
"This model card describes the Bio+Discharge Summary BERT model, which was initialized from [BioBERT](https://arxiv.org/abs/1901.08746) & trained on only discharge summaries from MIMIC.\n"
"> The model introduction and model weights originate from [https://huggingface.co/emilyalsentzer/Bio_Discharge_Summary_BERT](https://huggingface.co/emilyalsentzer/Bio_Discharge_Summary_BERT) and were converted to PaddlePaddle format for ease of use in PaddleNLP.\n"
"[Google's T5](https://ai.googleblog.com/2020/02/exploring-transfer-learning-with-t5.html) Version 1.1 - LM-Adapted\n",
"\n",
"\n",
"## Version 1.1 - LM-Adapted\n",
"\n",
"[T5 Version 1.1 - LM Adapted](https://github.com/google-research/text-to-text-transfer-transformer/blob/main/released_checkpoints.md#lm-adapted-t511lm100k) includes the following improvements compared to the original T5 model:\n",
"\n",
"- GEGLU activation in feed-forward hidden layer, rather than ReLU - see [here](https://arxiv.org/abs/2002.05202).\n",
"\n",
"- Dropout was turned off in pre-training (quality win). Dropout should be re-enabled during fine-tuning.\n",
"\n",
"- Pre-trained on C4 only without mixing in the downstream tasks.\n",
"\n",
"- no parameter sharing between embedding and classifier layer\n",
"\n",
"- \"xl\" and \"xxl\" replace \"3B\" and \"11B\". The model shapes are a bit different - larger `d_model` and smaller `num_heads` and `d_ff`.\n",
"\n",
"and is pretrained on both the denoising and language modeling objective.\n",
"\n",
"More specifically, this checkpoint is initialized from T5 Version 1.1 - Base\n",
"and then trained for an additional 100K steps on the LM objective discussed in the [T5 paper](https://arxiv.org/pdf/1910.10683.pdf).\n",
"This adaptation improves the ability of the model to be used for prompt tuning.\n",
"\n",
"**Note**: A popular fine-tuned version of the *T5 Version 1.1 - LM Adapted* model is BigScience's T0pp.\n",
"\n",
"Pretraining Dataset: C4\n",
"\n",
"Paper: [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://arxiv.org/pdf/1910.10683.pdf)\n",
"\n",
"Authors: *Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J. Liu*\n",
"\n",
"\n",
"## Abstract\n",
"\n",
"Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts every language problem into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled datasets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new “Colossal Clean Crawled Corpus”, we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our dataset, pre-trained models, and code.\n",
"[Google's T5](https://ai.googleblog.com/2020/02/exploring-transfer-learning-with-t5.html) Version 1.1 - LM-Adapted\n",
"\n",
"\n",
"## Version 1.1 - LM-Adapted\n",
"\n",
"[T5 Version 1.1 - LM Adapted](https://github.com/google-research/text-to-text-transfer-transformer/blob/main/released_checkpoints.md#lm-adapted-t511lm100k) includes the following improvements compared to the original T5 model:\n",
"\n",
"- GEGLU activation in feed-forward hidden layer, rather than ReLU - see [here](https://arxiv.org/abs/2002.05202).\n",
"\n",
"- Dropout was turned off in pre-training (quality win). Dropout should be re-enabled during fine-tuning.\n",
"\n",
"- Pre-trained on C4 only without mixing in the downstream tasks.\n",
"\n",
"- no parameter sharing between embedding and classifier layer\n",
"\n",
"- \"xl\" and \"xxl\" replace \"3B\" and \"11B\". The model shapes are a bit different - larger `d_model` and smaller `num_heads` and `d_ff`.\n",
"\n",
"and is pretrained on both the denoising and language modeling objective.\n",
"\n",
"More specifically, this checkpoint is initialized from T5 Version 1.1 - Base\n",
"and then trained for an additional 100K steps on the LM objective discussed in the [T5 paper](https://arxiv.org/pdf/1910.10683.pdf).\n",
"This adaptation improves the ability of the model to be used for prompt tuning.\n",
"\n",
"**Note**: A popular fine-tuned version of the *T5 Version 1.1 - LM Adapted* model is BigScience's T0pp.\n",
"\n",
"Pretraining Dataset: C4\n",
"\n",
"Paper: [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://arxiv.org/pdf/1910.10683.pdf)\n",
"\n",
"Authors: *Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J. Liu*\n",
"\n",
"\n",
"## Abstract\n",
"\n",
"Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts every language problem into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled datasets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new “Colossal Clean Crawled Corpus”, we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our dataset, pre-trained models, and code.\n",
"> The model introduction and model weights originate from [https://huggingface.co/google/t5-base-lm-adapt](https://huggingface.co/google/t5-base-lm-adapt) and were converted to PaddlePaddle format for ease of use in PaddleNLP."
"[Google's T5](https://ai.googleblog.com/2020/02/exploring-transfer-learning-with-t5.html) Version 1.1 - LM-Adapted\n",
"\n",
"\n",
"## Version 1.1 - LM-Adapted\n",
"\n",
"[T5 Version 1.1 - LM Adapted](https://github.com/google-research/text-to-text-transfer-transformer/blob/main/released_checkpoints.md#lm-adapted-t511lm100k) includes the following improvements compared to the original [T5 model](https://huggingface.co/t5-large):\n",
"\n",
"- GEGLU activation in feed-forward hidden layer, rather than ReLU - see [here](https://arxiv.org/abs/2002.05202).\n",
"\n",
"- Dropout was turned off in pre-training (quality win). Dropout should be re-enabled during fine-tuning.\n",
"\n",
"- Pre-trained on C4 only without mixing in the downstream tasks.\n",
"\n",
"- no parameter sharing between embedding and classifier layer\n",
"\n",
"- \"xl\" and \"xxl\" replace \"3B\" and \"11B\". The model shapes are a bit different - larger `d_model` and smaller `num_heads` and `d_ff`.\n",
"\n",
"and is pretrained on both the denoising and language modeling objective.\n",
"\n",
"More specifically, this checkpoint is initialized from T5 Version 1.1 - Large\n",
"and then trained for an additional 100K steps on the LM objective discussed in the [T5 paper](https://arxiv.org/pdf/1910.10683.pdf).\n",
"This adaptation improves the ability of the model to be used for prompt tuning.\n",
"\n",
"**Note**: A popular fine-tuned version of the *T5 Version 1.1 - LM Adapted* model is BigScience's T0pp.\n",
"\n",
"Pretraining Dataset: C4\n",
"\n",
"Paper: [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://arxiv.org/pdf/1910.10683.pdf)\n",
"\n",
"Authors: *Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J. Liu*\n",
"\n",
"\n",
"## Abstract\n",
"\n",
"Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts every language problem into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled datasets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new “Colossal Clean Crawled Corpus”, we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our dataset, pre-trained models, and code.\n",
"[Google's T5](https://ai.googleblog.com/2020/02/exploring-transfer-learning-with-t5.html) Version 1.1 - LM-Adapted\n",
"\n",
"\n",
"## Version 1.1 - LM-Adapted\n",
"\n",
"[T5 Version 1.1 - LM Adapted](https://github.com/google-research/text-to-text-transfer-transformer/blob/main/released_checkpoints.md#lm-adapted-t511lm100k) includes the following improvements compared to the original [T5 model](https://huggingface.co/t5-large):\n",
"\n",
"- GEGLU activation in feed-forward hidden layer, rather than ReLU - see [here](https://arxiv.org/abs/2002.05202).\n",
"\n",
"- Dropout was turned off in pre-training (quality win). Dropout should be re-enabled during fine-tuning.\n",
"\n",
"- Pre-trained on C4 only without mixing in the downstream tasks.\n",
"\n",
"- no parameter sharing between embedding and classifier layer\n",
"\n",
"- \"xl\" and \"xxl\" replace \"3B\" and \"11B\". The model shapes are a bit different - larger `d_model` and smaller `num_heads` and `d_ff`.\n",
"\n",
"and is pretrained on both the denoising and language modeling objective.\n",
"\n",
"More specifically, this checkpoint is initialized from T5 Version 1.1 - Large\n",
"and then trained for an additional 100K steps on the LM objective discussed in the [T5 paper](https://arxiv.org/pdf/1910.10683.pdf).\n",
"This adaptation improves the ability of the model to be used for prompt tuning.\n",
"\n",
"**Note**: A popular fine-tuned version of the *T5 Version 1.1 - LM Adapted* model is BigScience's T0pp.\n",
"\n",
"Pretraining Dataset: C4\n",
"\n",
"Paper: [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://arxiv.org/pdf/1910.10683.pdf)\n",
"\n",
"Authors: *Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J. Liu*\n",
"\n",
"\n",
"## Abstract\n",
"\n",
"Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts every language problem into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled datasets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new “Colossal Clean Crawled Corpus”, we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our dataset, pre-trained models, and code.\n",
"> The model introduction and model weights originate from [https://huggingface.co/google/t5-large-lm-adapt](https://huggingface.co/google/t5-large-lm-adapt) and were converted to PaddlePaddle format for ease of use in PaddleNLP."
"[Google's T5](https://ai.googleblog.com/2020/02/exploring-transfer-learning-with-t5.html) Version 1.1 - LM-Adapted\n",
"\n",
"\n",
"## Version 1.1 - LM-Adapted\n",
"\n",
"[T5 Version 1.1 - LM Adapted](https://github.com/google-research/text-to-text-transfer-transformer/blob/main/released_checkpoints.md#lm-adapted-t511lm100k) includes the following improvements compared to the original [T5 model](https://huggingface.co/t5-small):\n",
"\n",
"- GEGLU activation in feed-forward hidden layer, rather than ReLU - see [here](https://arxiv.org/abs/2002.05202).\n",
"\n",
"- Dropout was turned off in pre-training (quality win). Dropout should be re-enabled during fine-tuning.\n",
"\n",
"- Pre-trained on C4 only without mixing in the downstream tasks.\n",
"\n",
"- no parameter sharing between embedding and classifier layer\n",
"\n",
"- \"xl\" and \"xxl\" replace \"3B\" and \"11B\". The model shapes are a bit different - larger `d_model` and smaller `num_heads` and `d_ff`.\n",
"\n",
"and is pretrained on both the denoising and language modeling objective.\n",
"\n",
"More specifically, this checkpoint is initialized from T5 Version 1.1 - Small\n",
"and then trained for an additional 100K steps on the LM objective discussed in the [T5 paper](https://arxiv.org/pdf/1910.10683.pdf).\n",
"This adaptation improves the ability of the model to be used for prompt tuning.\n",
"\n",
"**Note**: A popular fine-tuned version of the *T5 Version 1.1 - LM Adapted* model is BigScience's T0pp.\n",
"\n",
"Pretraining Dataset: C4\n",
"\n",
"Paper: [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://arxiv.org/pdf/1910.10683.pdf)\n",
"\n",
"Authors: *Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J. Liu*\n",
"\n",
"\n",
"## Abstract\n",
"\n",
"Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts every language problem into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled datasets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new “Colossal Clean Crawled Corpus”, we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our dataset, pre-trained models, and code.\n",
"[Google's T5](https://ai.googleblog.com/2020/02/exploring-transfer-learning-with-t5.html) Version 1.1 - LM-Adapted\n",
"\n",
"\n",
"## Version 1.1 - LM-Adapted\n",
"\n",
"[T5 Version 1.1 - LM Adapted](https://github.com/google-research/text-to-text-transfer-transformer/blob/main/released_checkpoints.md#lm-adapted-t511lm100k) includes the following improvements compared to the original [T5 model](https://huggingface.co/t5-small):\n",
"\n",
"- GEGLU activation in feed-forward hidden layer, rather than ReLU - see [here](https://arxiv.org/abs/2002.05202).\n",
"\n",
"- Dropout was turned off in pre-training (quality win). Dropout should be re-enabled during fine-tuning.\n",
"\n",
"- Pre-trained on C4 only without mixing in the downstream tasks.\n",
"\n",
"- no parameter sharing between embedding and classifier layer\n",
"\n",
"- \"xl\" and \"xxl\" replace \"3B\" and \"11B\". The model shapes are a bit different - larger `d_model` and smaller `num_heads` and `d_ff`.\n",
"\n",
"and is pretrained on both the denoising and language modeling objective.\n",
"\n",
"More specifically, this checkpoint is initialized from [T5 Version 1.1 - Small](https://huggingface.co/google/https://huggingface.co/google/t5-v1_1-small)\n",
"and then trained for an additional 100K steps on the LM objective discussed in the [T5 paper](https://arxiv.org/pdf/1910.10683.pdf).\n",
"This adaptation improves the ability of the model to be used for prompt tuning.\n",
"\n",
"**Note**: A popular fine-tuned version of the *T5 Version 1.1 - LM Adapted* model is BigScience's T0pp.\n",
"\n",
"Pretraining Dataset: C4\n",
"\n",
"Paper: [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://arxiv.org/pdf/1910.10683.pdf)\n",
"\n",
"Authors: *Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J. Liu*\n",
"\n",
"\n",
"## Abstract\n",
"\n",
"Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts every language problem into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled datasets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new “Colossal Clean Crawled Corpus”, we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our dataset, pre-trained models, and code.\n",
"> The model introduction and model weights originate from [https://huggingface.co/google/t5-small-lm-adapt](https://huggingface.co/google/t5-small-lm-adapt) and were converted to PaddlePaddle format for ease of use in PaddleNLP."
"[Google's T5](https://ai.googleblog.com/2020/02/exploring-transfer-learning-with-t5.html) Version 1.1\n",
"\n",
"\n",
"## Version 1.1\n",
"\n",
"[T5 Version 1.1](https://github.com/google-research/text-to-text-transfer-transformer/blob/master/released_checkpoints.md#t511) includes the following improvements compared to the original T5 model- GEGLU activation in feed-forward hidden layer, rather than ReLU - see [here](https://arxiv.org/abs/2002.05202).\n",
"\n",
"- Dropout was turned off in pre-training (quality win). Dropout should be re-enabled during fine-tuning.\n",
"\n",
"- Pre-trained on C4 only without mixing in the downstream tasks.\n",
"\n",
"- no parameter sharing between embedding and classifier layer\n",
"\n",
"- \"xl\" and \"xxl\" replace \"3B\" and \"11B\". The model shapes are a bit different - larger `d_model` and smaller `num_heads` and `d_ff`.\n",
"\n",
"**Note**: T5 Version 1.1 was only pre-trained on C4 excluding any supervised training. Therefore, this model has to be fine-tuned before it is useable on a downstream task.\n",
"Pretraining Dataset: C4\n",
"\n",
"Paper: [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://arxiv.org/pdf/1910.10683.pdf)\n",
"\n",
"Authors: *Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J. Liu*\n",
"\n",
"\n",
"## Abstract\n",
"\n",
"Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts every language problem into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled datasets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new “Colossal Clean Crawled Corpus”, we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our dataset, pre-trained models, and code.\n",
"[Google's T5](https://ai.googleblog.com/2020/02/exploring-transfer-learning-with-t5.html) Version 1.1\n",
"\n",
"\n",
"## Version 1.1\n",
"\n",
"[T5 Version 1.1](https://github.com/google-research/text-to-text-transfer-transformer/blob/master/released_checkpoints.md#t511) includes the following improvements compared to the original T5 model- GEGLU activation in feed-forward hidden layer, rather than ReLU - see [here](https://arxiv.org/abs/2002.05202).\n",
"\n",
"- Dropout was turned off in pre-training (quality win). Dropout should be re-enabled during fine-tuning.\n",
"\n",
"- Pre-trained on C4 only without mixing in the downstream tasks.\n",
"\n",
"- no parameter sharing between embedding and classifier layer\n",
"\n",
"- \"xl\" and \"xxl\" replace \"3B\" and \"11B\". The model shapes are a bit different - larger `d_model` and smaller `num_heads` and `d_ff`.\n",
"\n",
"**Note**: T5 Version 1.1 was only pre-trained on C4 excluding any supervised training. Therefore, this model has to be fine-tuned before it is useable on a downstream task.\n",
"Pretraining Dataset: C4\n",
"\n",
"Paper: [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://arxiv.org/pdf/1910.10683.pdf)\n",
"\n",
"Authors: *Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J. Liu*\n",
"\n",
"\n",
"## Abstract\n",
"\n",
"Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts every language problem into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled datasets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new “Colossal Clean Crawled Corpus”, we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our dataset, pre-trained models, and code.\n",
"> The model introduction and model weights originate from [https://huggingface.co/google/t5-v1_1-base](https://huggingface.co/google/t5-v1_1-base) and were converted to PaddlePaddle format for ease of use in PaddleNLP."
"[Google's T5](https://ai.googleblog.com/2020/02/exploring-transfer-learning-with-t5.html) Version 1.1\n",
"\n",
"\n",
"## Version 1.1\n",
"\n",
"[T5 Version 1.1](https://github.com/google-research/text-to-text-transfer-transformer/blob/master/released_checkpoints.md#t511) includes the following improvements compared to the original T5 model- GEGLU activation in feed-forward hidden layer, rather than ReLU - see [here](https://arxiv.org/abs/2002.05202).\n",
"\n",
"- Dropout was turned off in pre-training (quality win). Dropout should be re-enabled during fine-tuning.\n",
"\n",
"- Pre-trained on C4 only without mixing in the downstream tasks.\n",
"\n",
"- no parameter sharing between embedding and classifier layer\n",
"\n",
"- \"xl\" and \"xxl\" replace \"3B\" and \"11B\". The model shapes are a bit different - larger `d_model` and smaller `num_heads` and `d_ff`.\n",
"\n",
"**Note**: T5 Version 1.1 was only pre-trained on C4 excluding any supervised training. Therefore, this model has to be fine-tuned before it is useable on a downstream task.\n",
"Pretraining Dataset: C4\n",
"\n",
"Paper: [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://arxiv.org/pdf/1910.10683.pdf)\n",
"\n",
"Authors: *Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J. Liu*\n",
"\n",
"\n",
"## Abstract\n",
"\n",
"Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts every language problem into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled datasets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new “Colossal Clean Crawled Corpus”, we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our dataset, pre-trained models, and code.\n",
"[Google's T5](https://ai.googleblog.com/2020/02/exploring-transfer-learning-with-t5.html) Version 1.1\n",
"\n",
"\n",
"## Version 1.1\n",
"\n",
"[T5 Version 1.1](https://github.com/google-research/text-to-text-transfer-transformer/blob/master/released_checkpoints.md#t511) includes the following improvements compared to the original T5 model- GEGLU activation in feed-forward hidden layer, rather than ReLU - see [here](https://arxiv.org/abs/2002.05202).\n",
"\n",
"- Dropout was turned off in pre-training (quality win). Dropout should be re-enabled during fine-tuning.\n",
"\n",
"- Pre-trained on C4 only without mixing in the downstream tasks.\n",
"\n",
"- no parameter sharing between embedding and classifier layer\n",
"\n",
"- \"xl\" and \"xxl\" replace \"3B\" and \"11B\". The model shapes are a bit different - larger `d_model` and smaller `num_heads` and `d_ff`.\n",
"\n",
"**Note**: T5 Version 1.1 was only pre-trained on C4 excluding any supervised training. Therefore, this model has to be fine-tuned before it is useable on a downstream task.\n",
"Pretraining Dataset: C4\n",
"\n",
"Paper: [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://arxiv.org/pdf/1910.10683.pdf)\n",
"\n",
"Authors: *Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J. Liu*\n",
"\n",
"\n",
"## Abstract\n",
"\n",
"Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts every language problem into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled datasets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new “Colossal Clean Crawled Corpus”, we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our dataset, pre-trained models, and code.\n",
"> The model introduction and model weights originate from [https://huggingface.co/google/t5-v1_1-large](https://huggingface.co/google/t5-v1_1-large) and were converted to PaddlePaddle format for ease of use in PaddleNLP."
"[Google's T5](https://ai.googleblog.com/2020/02/exploring-transfer-learning-with-t5.html) Version 1.1\n",
"\n",
"\n",
"## Version 1.1\n",
"\n",
"[T5 Version 1.1](https://github.com/google-research/text-to-text-transfer-transformer/blob/master/released_checkpoints.md#t511) includes the following improvements compared to the original T5 model- GEGLU activation in feed-forward hidden layer, rather than ReLU - see [here](https://arxiv.org/abs/2002.05202).\n",
"\n",
"- Dropout was turned off in pre-training (quality win). Dropout should be re-enabled during fine-tuning.\n",
"\n",
"- Pre-trained on C4 only without mixing in the downstream tasks.\n",
"\n",
"- no parameter sharing between embedding and classifier layer\n",
"\n",
"- \"xl\" and \"xxl\" replace \"3B\" and \"11B\". The model shapes are a bit different - larger `d_model` and smaller `num_heads` and `d_ff`.\n",
"\n",
"**Note**: T5 Version 1.1 was only pre-trained on C4 excluding any supervised training. Therefore, this model has to be fine-tuned before it is useable on a downstream task.\n",
"Other Community Checkpoints: [here](https://huggingface.co/models?search=t5-v1_1)\n",
"\n",
"Paper: [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://arxiv.org/pdf/1910.10683.pdf)\n",
"\n",
"Authors: *Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J. Liu*\n",
"\n",
"\n",
"## Abstract\n",
"\n",
"Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts every language problem into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled datasets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new “Colossal Clean Crawled Corpus”, we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our dataset, pre-trained models, and code.\n",
"[Google's T5](https://ai.googleblog.com/2020/02/exploring-transfer-learning-with-t5.html) Version 1.1\n",
"\n",
"\n",
"## Version 1.1\n",
"\n",
"[T5 Version 1.1](https://github.com/google-research/text-to-text-transfer-transformer/blob/master/released_checkpoints.md#t511) includes the following improvements compared to the original T5 model- GEGLU activation in feed-forward hidden layer, rather than ReLU - see [here](https://arxiv.org/abs/2002.05202).\n",
"\n",
"- Dropout was turned off in pre-training (quality win). Dropout should be re-enabled during fine-tuning.\n",
"\n",
"- Pre-trained on C4 only without mixing in the downstream tasks.\n",
"\n",
"- no parameter sharing between embedding and classifier layer\n",
"\n",
"- \"xl\" and \"xxl\" replace \"3B\" and \"11B\". The model shapes are a bit different - larger `d_model` and smaller `num_heads` and `d_ff`.\n",
"\n",
"**Note**: T5 Version 1.1 was only pre-trained on C4 excluding any supervised training. Therefore, this model has to be fine-tuned before it is useable on a downstream task.\n",
"Other Community Checkpoints: [here](https://huggingface.co/models?search=t5-v1_1)\n",
"\n",
"Paper: [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://arxiv.org/pdf/1910.10683.pdf)\n",
"\n",
"Authors: *Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J. Liu*\n",
"\n",
"\n",
"## Abstract\n",
"\n",
"Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts every language problem into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled datasets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new “Colossal Clean Crawled Corpus”, we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our dataset, pre-trained models, and code.\n",
"> The model introduction and model weights originate from [https://huggingface.co/google/t5-v1_1-small](https://huggingface.co/google/t5-v1_1-small) and were converted to PaddlePaddle format for ease of use in PaddleNLP."
"title={Pre-Training with Whole Word Masking for Chinese BERT},\n",
"author={Cui, Yiming and Che, Wanxiang and Liu, Ting and Qin, Bing and Yang, Ziqing and Wang, Shijin and Hu, Guoping},\n",
"journal={arXiv preprint arXiv:1906.08101},\n",
"year={2019}\n",
"}\n",
"```"
]
},
{
"cell_type": "markdown",
"id": "86de1995",
"metadata": {},
"source": [
"> The model introduction and model weights originate from [https://huggingface.co/hfl/chinese-bert-wwm-ext](https://huggingface.co/hfl/chinese-bert-wwm-ext) and were converted to PaddlePaddle format for ease of use in PaddleNLP."
"title={Pre-Training with Whole Word Masking for Chinese BERT},\n",
"author={Cui, Yiming and Che, Wanxiang and Liu, Ting and Qin, Bing and Yang, Ziqing and Wang, Shijin and Hu, Guoping},\n",
"journal={arXiv preprint arXiv:1906.08101},\n",
"year={2019}\n",
"}\n",
"```"
]
},
{
"cell_type": "markdown",
"id": "86de1995",
"metadata": {},
"source": [
"> The model introduction and model weights originate from [https://huggingface.co/hfl/chinese-bert-wwm-ext](https://huggingface.co/hfl/chinese-bert-wwm-ext) and were converted to PaddlePaddle format for ease of use in PaddleNLP."
"title={Pre-Training with Whole Word Masking for Chinese BERT},\n",
"author={Cui, Yiming and Che, Wanxiang and Liu, Ting and Qin, Bing and Yang, Ziqing and Wang, Shijin and Hu, Guoping},\n",
"journal={arXiv preprint arXiv:1906.08101},\n",
"year={2019}\n",
"}\n",
"```"
]
},
{
"cell_type": "markdown",
"id": "86de1995",
"metadata": {},
"source": [
"> The model introduction and model weights originate from [https://huggingface.co/hfl/chinese-roberta-wwm-ext-large](https://huggingface.co/hfl/chinese-roberta-wwm-ext-large) and were converted to PaddlePaddle format for ease of use in PaddleNLP."
"title={Pre-Training with Whole Word Masking for Chinese BERT},\n",
"author={Cui, Yiming and Che, Wanxiang and Liu, Ting and Qin, Bing and Yang, Ziqing and Wang, Shijin and Hu, Guoping},\n",
"journal={arXiv preprint arXiv:1906.08101},\n",
"year={2019}\n",
"}\n",
"```"
]
},
{
"cell_type": "markdown",
"id": "86de1995",
"metadata": {},
"source": [
"> The model introduction and model weights originate from [https://huggingface.co/hfl/chinese-roberta-wwm-ext](https://huggingface.co/hfl/chinese-roberta-wwm-ext) and were converted to PaddlePaddle format for ease of use in PaddleNLP."
"title={Pre-Training with Whole Word Masking for Chinese BERT},\n",
"author={Cui, Yiming and Che, Wanxiang and Liu, Ting and Qin, Bing and Yang, Ziqing and Wang, Shijin and Hu, Guoping},\n",
"journal={arXiv preprint arXiv:1906.08101},\n",
"year={2019}\n",
"}\n",
"```"
]
},
{
"cell_type": "markdown",
"id": "86de1995",
"metadata": {},
"source": [
"> The model introduction and model weights originate from [https://huggingface.co/hfl/rbt3](https://huggingface.co/hfl/rbt3) and were converted to PaddlePaddle format for ease of use in PaddleNLP."
"> The model introduction and model weights originate from https://huggingface.co/bert-base-cased and were converted to PaddlePaddle format for ease of use in PaddleNLP."
"You can get more details from [Bert in PaddleNLP](https://github.com/PaddlePaddle/PaddleNLP/blob/develop/model_zoo/bert/README.md)。"
]
},
{
"cell_type": "markdown",
"id": "f407e80e",
"metadata": {},
"source": [
"**Update April 3rd, 2020**: we updated the vocabulary file on deepset's s3 to conform with the default tokenization of punctuation tokens.\n",
"For details see the related [FARM issue](https://github.com/deepset-ai/FARM/issues/60). If you want to use the old vocab we have also uploaded a deepset/bert-base-german-cased-oldvocab model.\n"
"> The model introduction and model weights originate from https://huggingface.co/bert-base-german-cased and were converted to PaddlePaddle format for ease of use in PaddleNLP."
"> The model introduction and model weights originate from https://huggingface.co/bert-base-multilingual-cased and were converted to PaddlePaddle format for ease of use in PaddleNLP.\n"
"> The model introduction and model weights originate from https://huggingface.co/bert-base-multilingual-uncased and were converted to PaddlePaddle format for ease of use in PaddleNLP."
"详细内容请看[Bert in PaddleNLP](https://github.com/PaddlePaddle/PaddleNLP/blob/develop/model_zoo/bert/README.md)。"
]
},
{
"cell_type": "markdown",
"id": "d348c680",
"metadata": {},
"source": [
"Pretrained model on English language using a masked language modeling (MLM) objective. It was introduced in\n",
"[this paper](https://arxiv.org/abs/1810.04805) and first released in\n",
"[this repository](https://github.com/google-research/bert). This model is uncased: it does not make a difference\n",
"between english and English.\n"
]
},
{
"cell_type": "markdown",
"id": "9a790b40",
"metadata": {},
"source": [
"Disclaimer: The team releasing BERT did not write a model card for this model so this model card has been written by\n",
"the Hugging Face team.\n"
]
},
{
"cell_type": "markdown",
"id": "985d2894",
"metadata": {},
"source": [
"## Model description\n"
]
},
{
"cell_type": "markdown",
"id": "985bd7ee",
"metadata": {},
"source": [
"BERT is a transformers model pretrained on a large corpus of English data in a self-supervised fashion. This means it\n",
"was pretrained on the raw texts only, with no humans labeling them in any way (which is why it can use lots of\n",
"publicly available data) with an automatic process to generate inputs and labels from those texts. More precisely, it\n",
"was pretrained with two objectives:\n"
]
},
{
"cell_type": "markdown",
"id": "2e1ee5f4",
"metadata": {},
"source": [
"- Masked language modeling (MLM): taking a sentence, the model randomly masks 15% of the words in the input then run\n",
"the entire masked sentence through the model and has to predict the masked words. This is different from traditional\n",
"recurrent neural networks (RNNs) that usually see the words one after the other, or from autoregressive models like\n",
"GPT which internally masks the future tokens. It allows the model to learn a bidirectional representation of the\n",
"sentence.\n",
"- Next sentence prediction (NSP): the models concatenates two masked sentences as inputs during pretraining. Sometimes\n",
"they correspond to sentences that were next to each other in the original text, sometimes not. The model then has to\n",
"predict if the two sentences were following each other or not.\n"
]
},
{
"cell_type": "markdown",
"id": "ae584a51",
"metadata": {},
"source": [
"This way, the model learns an inner representation of the English language that can then be used to extract features\n",
"useful for downstream tasks: if you have a dataset of labeled sentences, for instance, you can train a standard\n",
"classifier using the features produced by the BERT model as inputs.\n"
]
},
{
"cell_type": "markdown",
"id": "a4d48848",
"metadata": {},
"source": [
"## Model variations\n"
]
},
{
"cell_type": "markdown",
"id": "dcb46068",
"metadata": {},
"source": [
"BERT has originally been released in base and large variations, for cased and uncased input text. The uncased models also strips out an accent markers.\n",
"Chinese and multilingual uncased and cased versions followed shortly after.\n",
"Modified preprocessing with whole word masking has replaced subpiece masking in a following work, with the release of two models.\n",
"Other 24 smaller models are released afterward.\n"
]
},
{
"cell_type": "markdown",
"id": "bdf3ec7e",
"metadata": {},
"source": [
"The detailed release history can be found on the [google-research/bert readme](https://github.com/google-research/bert/blob/master/README.md) on github.\n"
"You can get more details from [Bert in PaddleNLP](https://github.com/PaddlePaddle/PaddleNLP/blob/develop/model_zoo/bert/README.md)。"
]
},
{
"cell_type": "markdown",
"id": "e25590e2",
"metadata": {},
"source": [
"Pretrained model on English language using a masked language modeling (MLM) objective. It was introduced in\n",
"[this paper](https://arxiv.org/abs/1810.04805) and first released in\n",
"[this repository](https://github.com/google-research/bert). This model is uncased: it does not make a difference\n",
"between english and English.\n"
]
},
{
"cell_type": "markdown",
"id": "632646c9",
"metadata": {},
"source": [
"Disclaimer: The team releasing BERT did not write a model card for this model so this model card has been written by\n",
"the Hugging Face team.\n"
]
},
{
"cell_type": "markdown",
"id": "6d37733d",
"metadata": {},
"source": [
"## Model description\n"
]
},
{
"cell_type": "markdown",
"id": "20eb0099",
"metadata": {},
"source": [
"BERT is a transformers model pretrained on a large corpus of English data in a self-supervised fashion. This means it\n",
"was pretrained on the raw texts only, with no humans labeling them in any way (which is why it can use lots of\n",
"publicly available data) with an automatic process to generate inputs and labels from those texts. More precisely, it\n",
"was pretrained with two objectives:\n"
]
},
{
"cell_type": "markdown",
"id": "a43bc44c",
"metadata": {},
"source": [
"- Masked language modeling (MLM): taking a sentence, the model randomly masks 15% of the words in the input then run\n",
"the entire masked sentence through the model and has to predict the masked words. This is different from traditional\n",
"recurrent neural networks (RNNs) that usually see the words one after the other, or from autoregressive models like\n",
"GPT which internally masks the future tokens. It allows the model to learn a bidirectional representation of the\n",
"sentence.\n",
"- Next sentence prediction (NSP): the models concatenates two masked sentences as inputs during pretraining. Sometimes\n",
"they correspond to sentences that were next to each other in the original text, sometimes not. The model then has to\n",
"predict if the two sentences were following each other or not.\n"
]
},
{
"cell_type": "markdown",
"id": "3ea31760",
"metadata": {},
"source": [
"This way, the model learns an inner representation of the English language that can then be used to extract features\n",
"useful for downstream tasks: if you have a dataset of labeled sentences, for instance, you can train a standard\n",
"classifier using the features produced by the BERT model as inputs.\n"
]
},
{
"cell_type": "markdown",
"id": "c44e01b0",
"metadata": {},
"source": [
"## Model variations\n"
]
},
{
"cell_type": "markdown",
"id": "6cb3e530",
"metadata": {},
"source": [
"BERT has originally been released in base and large variations, for cased and uncased input text. The uncased models also strips out an accent markers.\n",
"Chinese and multilingual uncased and cased versions followed shortly after.\n",
"Modified preprocessing with whole word masking has replaced subpiece masking in a following work, with the release of two models.\n",
"Other 24 smaller models are released afterward.\n"
]
},
{
"cell_type": "markdown",
"id": "557a417a",
"metadata": {},
"source": [
"The detailed release history can be found on the [google-research/bert readme](https://github.com/google-research/bert/blob/master/README.md) on github.\n"
"> The model introduction and model weights originate from https://huggingface.co/bert-base-uncased and were converted to PaddlePaddle format for ease of use in PaddleNLP."
"# BERT large model (cased) whole word masking finetuned on SQuAD\n",
"\n",
"详细内容请看[Bert in PaddleNLP](https://github.com/PaddlePaddle/PaddleNLP/blob/develop/model_zoo/bert/README.md)。"
]
},
{
"cell_type": "markdown",
"id": "7804aeec",
"metadata": {},
"source": [
"Pretrained model on English language using a masked language modeling (MLM) objective. It was introduced in\n",
"[this paper](https://arxiv.org/abs/1810.04805) and first released in\n",
"[this repository](https://github.com/google-research/bert). This model is cased: it makes a difference between english and English.\n"
]
},
{
"cell_type": "markdown",
"id": "9ee7c4ee",
"metadata": {},
"source": [
"Differently to other BERT models, this model was trained with a new technique: Whole Word Masking. In this case, all of the tokens corresponding to a word are masked at once. The overall masking rate remains the same.\n"
]
},
{
"cell_type": "markdown",
"id": "2198ff25",
"metadata": {},
"source": [
"The training is identical -- each masked WordPiece token is predicted independently.\n"
]
},
{
"cell_type": "markdown",
"id": "159c04c3",
"metadata": {},
"source": [
"After pre-training, this model was fine-tuned on the SQuAD dataset with one of our fine-tuning scripts. See below for more information regarding this fine-tuning.\n"
]
},
{
"cell_type": "markdown",
"id": "cec53443",
"metadata": {},
"source": [
"Disclaimer: The team releasing BERT did not write a model card for this model so this model card has been written by\n",
"the Hugging Face team.\n"
]
},
{
"cell_type": "markdown",
"id": "0a6d113e",
"metadata": {},
"source": [
"## Model description\n"
]
},
{
"cell_type": "markdown",
"id": "3776a729",
"metadata": {},
"source": [
"BERT is a transformers model pretrained on a large corpus of English data in a self-supervised fashion. This means it\n",
"was pretrained on the raw texts only, with no humans labelling them in any way (which is why it can use lots of\n",
"publicly available data) with an automatic process to generate inputs and labels from those texts. More precisely, it\n",
"was pretrained with two objectives:\n"
]
},
{
"cell_type": "markdown",
"id": "8ef5e147",
"metadata": {},
"source": [
"- Masked language modeling (MLM): taking a sentence, the model randomly masks 15% of the words in the input then run\n",
"the entire masked sentence through the model and has to predict the masked words. This is different from traditional\n",
"recurrent neural networks (RNNs) that usually see the words one after the other, or from autoregressive models like\n",
"GPT which internally mask the future tokens. It allows the model to learn a bidirectional representation of the\n",
"sentence.\n",
"- Next sentence prediction (NSP): the models concatenates two masked sentences as inputs during pretraining. Sometimes\n",
"they correspond to sentences that were next to each other in the original text, sometimes not. The model then has to\n",
"predict if the two sentences were following each other or not.\n"
]
},
{
"cell_type": "markdown",
"id": "f494c97f",
"metadata": {},
"source": [
"This way, the model learns an inner representation of the English language that can then be used to extract features\n",
"useful for downstream tasks: if you have a dataset of labeled sentences for instance, you can train a standard\n",
"classifier using the features produced by the BERT model as inputs.\n"
"# BERT large model (cased) whole word masking finetuned on SQuAD\n",
"\n",
"You can get more details from [Bert in PaddleNLP](https://github.com/PaddlePaddle/PaddleNLP/blob/develop/model_zoo/bert/README.md)。"
]
},
{
"cell_type": "markdown",
"id": "7105fb8c",
"metadata": {},
"source": [
"Pretrained model on English language using a masked language modeling (MLM) objective. It was introduced in\n",
"[this paper](https://arxiv.org/abs/1810.04805) and first released in\n",
"[this repository](https://github.com/google-research/bert). This model is cased: it makes a difference between english and English.\n"
]
},
{
"cell_type": "markdown",
"id": "e3d8b394",
"metadata": {},
"source": [
"Differently to other BERT models, this model was trained with a new technique: Whole Word Masking. In this case, all of the tokens corresponding to a word are masked at once. The overall masking rate remains the same.\n"
]
},
{
"cell_type": "markdown",
"id": "be078628",
"metadata": {},
"source": [
"The training is identical -- each masked WordPiece token is predicted independently.\n"
]
},
{
"cell_type": "markdown",
"id": "278aee7f",
"metadata": {},
"source": [
"After pre-training, this model was fine-tuned on the SQuAD dataset with one of our fine-tuning scripts. See below for more information regarding this fine-tuning.\n"
]
},
{
"cell_type": "markdown",
"id": "ce69aca2",
"metadata": {},
"source": [
"Disclaimer: The team releasing BERT did not write a model card for this model so this model card has been written by\n",
"the Hugging Face team.\n"
]
},
{
"cell_type": "markdown",
"id": "89b52c17",
"metadata": {},
"source": [
"## Model description\n"
]
},
{
"cell_type": "markdown",
"id": "2ebe9e94",
"metadata": {},
"source": [
"BERT is a transformers model pretrained on a large corpus of English data in a self-supervised fashion. This means it\n",
"was pretrained on the raw texts only, with no humans labelling them in any way (which is why it can use lots of\n",
"publicly available data) with an automatic process to generate inputs and labels from those texts. More precisely, it\n",
"was pretrained with two objectives:\n"
]
},
{
"cell_type": "markdown",
"id": "7131c024",
"metadata": {},
"source": [
"- Masked language modeling (MLM): taking a sentence, the model randomly masks 15% of the words in the input then run\n",
"the entire masked sentence through the model and has to predict the masked words. This is different from traditional\n",
"recurrent neural networks (RNNs) that usually see the words one after the other, or from autoregressive models like\n",
"GPT which internally mask the future tokens. It allows the model to learn a bidirectional representation of the\n",
"sentence.\n",
"- Next sentence prediction (NSP): the models concatenates two masked sentences as inputs during pretraining. Sometimes\n",
"they correspond to sentences that were next to each other in the original text, sometimes not. The model then has to\n",
"predict if the two sentences were following each other or not.\n"
]
},
{
"cell_type": "markdown",
"id": "4a8e4aea",
"metadata": {},
"source": [
"This way, the model learns an inner representation of the English language that can then be used to extract features\n",
"useful for downstream tasks: if you have a dataset of labeled sentences for instance, you can train a standard\n",
"classifier using the features produced by the BERT model as inputs.\n"
"> The model introduction and model weights originate from https://huggingface.co/bert-large-cased-whole-word-masking-finetuned-squad and were converted to PaddlePaddle format for ease of use in PaddleNLP.\n"
"# BERT large model (cased) whole word masking\n",
"\n",
"详细内容请看[Bert in PaddleNLP](https://github.com/PaddlePaddle/PaddleNLP/blob/develop/model_zoo/bert/README.md)。"
]
},
{
"cell_type": "markdown",
"id": "9e7590bd",
"metadata": {},
"source": [
"Pretrained model on English language using a masked language modeling (MLM) objective. It was introduced in\n",
"[this paper](https://arxiv.org/abs/1810.04805) and first released in\n",
"[this repository](https://github.com/google-research/bert). This model is cased: it makes a difference between english and English.\n"
]
},
{
"cell_type": "markdown",
"id": "2456751c",
"metadata": {},
"source": [
"Differently to other BERT models, this model was trained with a new technique: Whole Word Masking. In this case, all of the tokens corresponding to a word are masked at once. The overall masking rate remains the same.\n"
]
},
{
"cell_type": "markdown",
"id": "204d6ee6",
"metadata": {},
"source": [
"The training is identical -- each masked WordPiece token is predicted independently.\n"
]
},
{
"cell_type": "markdown",
"id": "743ff269",
"metadata": {},
"source": [
"Disclaimer: The team releasing BERT did not write a model card for this model so this model card has been written by\n",
"the Hugging Face team.\n"
]
},
{
"cell_type": "markdown",
"id": "bce1ffcc",
"metadata": {},
"source": [
"## Model description\n"
]
},
{
"cell_type": "markdown",
"id": "d5d83b7c",
"metadata": {},
"source": [
"BERT is a transformers model pretrained on a large corpus of English data in a self-supervised fashion. This means it\n",
"was pretrained on the raw texts only, with no humans labelling them in any way (which is why it can use lots of\n",
"publicly available data) with an automatic process to generate inputs and labels from those texts. More precisely, it\n",
"was pretrained with two objectives:\n"
]
},
{
"cell_type": "markdown",
"id": "38a98598",
"metadata": {},
"source": [
"- Masked language modeling (MLM): taking a sentence, the model randomly masks 15% of the words in the input then run\n",
"the entire masked sentence through the model and has to predict the masked words. This is different from traditional\n",
"recurrent neural networks (RNNs) that usually see the words one after the other, or from autoregressive models like\n",
"GPT which internally mask the future tokens. It allows the model to learn a bidirectional representation of the\n",
"sentence.\n",
"- Next sentence prediction (NSP): the models concatenates two masked sentences as inputs during pretraining. Sometimes\n",
"they correspond to sentences that were next to each other in the original text, sometimes not. The model then has to\n",
"predict if the two sentences were following each other or not.\n"
]
},
{
"cell_type": "markdown",
"id": "89b5e554",
"metadata": {},
"source": [
"This way, the model learns an inner representation of the English language that can then be used to extract features\n",
"useful for downstream tasks: if you have a dataset of labeled sentences for instance, you can train a standard\n",
"classifier using the features produced by the BERT model as inputs.\n"
"# BERT large model (cased) whole word masking\n",
"\n",
"You can get more details from [Bert in PaddleNLP](https://github.com/PaddlePaddle/PaddleNLP/blob/develop/model_zoo/bert/README.md)。"
]
},
{
"cell_type": "markdown",
"id": "6814fe73",
"metadata": {},
"source": [
"Pretrained model on English language using a masked language modeling (MLM) objective. It was introduced in\n",
"[this paper](https://arxiv.org/abs/1810.04805) and first released in\n",
"[this repository](https://github.com/google-research/bert). This model is cased: it makes a difference between english and English.\n"
]
},
{
"cell_type": "markdown",
"id": "7a6b1b28",
"metadata": {},
"source": [
"Differently to other BERT models, this model was trained with a new technique: Whole Word Masking. In this case, all of the tokens corresponding to a word are masked at once. The overall masking rate remains the same.\n"
]
},
{
"cell_type": "markdown",
"id": "e6c8ddc5",
"metadata": {},
"source": [
"The training is identical -- each masked WordPiece token is predicted independently.\n"
]
},
{
"cell_type": "markdown",
"id": "dfcd9c6b",
"metadata": {},
"source": [
"Disclaimer: The team releasing BERT did not write a model card for this model so this model card has been written by\n",
"the Hugging Face team.\n"
]
},
{
"cell_type": "markdown",
"id": "d758dbd9",
"metadata": {},
"source": [
"## Model description\n"
]
},
{
"cell_type": "markdown",
"id": "c4e44287",
"metadata": {},
"source": [
"BERT is a transformers model pretrained on a large corpus of English data in a self-supervised fashion. This means it\n",
"was pretrained on the raw texts only, with no humans labelling them in any way (which is why it can use lots of\n",
"publicly available data) with an automatic process to generate inputs and labels from those texts. More precisely, it\n",
"was pretrained with two objectives:\n"
]
},
{
"cell_type": "markdown",
"id": "d07abc2a",
"metadata": {},
"source": [
"- Masked language modeling (MLM): taking a sentence, the model randomly masks 15% of the words in the input then run\n",
"the entire masked sentence through the model and has to predict the masked words. This is different from traditional\n",
"recurrent neural networks (RNNs) that usually see the words one after the other, or from autoregressive models like\n",
"GPT which internally mask the future tokens. It allows the model to learn a bidirectional representation of the\n",
"sentence.\n",
"- Next sentence prediction (NSP): the models concatenates two masked sentences as inputs during pretraining. Sometimes\n",
"they correspond to sentences that were next to each other in the original text, sometimes not. The model then has to\n",
"predict if the two sentences were following each other or not.\n"
]
},
{
"cell_type": "markdown",
"id": "5fcb83d6",
"metadata": {},
"source": [
"This way, the model learns an inner representation of the English language that can then be used to extract features\n",
"useful for downstream tasks: if you have a dataset of labeled sentences for instance, you can train a standard\n",
"classifier using the features produced by the BERT model as inputs.\n"
]
},
{
"cell_type": "markdown",
"id": "1be2f6a5",
"metadata": {},
"source": [
"This model has the following configuration:\n"
]
},
{
"cell_type": "markdown",
"id": "cd047a65",
"metadata": {},
"source": [
"- 24-layer\n",
"- 1024 hidden dimension\n",
"- 16 attention heads\n",
"- 336M parameters.\n"
]
},
{
"cell_type": "markdown",
"id": "93925c79",
"metadata": {},
"source": [
"## Intended uses & limitations\n"
]
},
{
"cell_type": "markdown",
"id": "f6c1f9b9",
"metadata": {},
"source": [
"You can use the raw model for either masked language modeling or next sentence prediction, but it's mostly intended to\n",
"be fine-tuned on a downstream task. See the [model hub](https://huggingface.co/models?filter=bert) to look for\n",
"fine-tuned versions on a task that interests you.\n"
]
},
{
"cell_type": "markdown",
"id": "a682ee5c",
"metadata": {},
"source": [
"Note that this model is primarily aimed at being fine-tuned on tasks that use the whole sentence (potentially masked)\n",
"to make decisions, such as sequence classification, token classification or question answering. For tasks such as text\n",
"generation you should look at model like GPT2.\n"
]
},
{
"cell_type": "markdown",
"id": "394e6456",
"metadata": {},
"source": [
"### How to use\n"
]
},
{
"cell_type": "markdown",
"id": "9e5fdb9a",
"metadata": {},
"source": [
"You can use this model directly with a pipeline for masked language modeling:\n"
"> The model introduction and model weights originate from https://huggingface.co/bert-large-cased-whole-word-masking and were converted to PaddlePaddle format for ease of use in PaddleNLP."
"> The model introduction and model weights originate from https://huggingface.co/bert-large-cased and were converted to PaddlePaddle format for ease of use in PaddleNLP."
"# BERT large model (uncased) whole word masking finetuned on SQuAD\n"
]
},
{
"cell_type": "markdown",
"id": "724df271",
"metadata": {},
"source": [
"Pretrained model on English language using a masked language modeling (MLM) objective. It was introduced in\n",
"[this paper](https://arxiv.org/abs/1810.04805) and first released in\n",
"[this repository](https://github.com/google-research/bert). This model is uncased: it does not make a difference\n",
"between english and English.\n"
]
},
{
"cell_type": "markdown",
"id": "f2b9e3bf",
"metadata": {},
"source": [
"Differently to other BERT models, this model was trained with a new technique: Whole Word Masking. In this case, all of the tokens corresponding to a word are masked at once. The overall masking rate remains the same.\n"
]
},
{
"cell_type": "markdown",
"id": "6566eb12",
"metadata": {},
"source": [
"The training is identical -- each masked WordPiece token is predicted independently.\n"
]
},
{
"cell_type": "markdown",
"id": "7b45422b",
"metadata": {},
"source": [
"After pre-training, this model was fine-tuned on the SQuAD dataset with one of our fine-tuning scripts. See below for more information regarding this fine-tuning.\n"
]
},
{
"cell_type": "markdown",
"id": "c9957f91",
"metadata": {},
"source": [
"Disclaimer: The team releasing BERT did not write a model card for this model so this model card has been written by\n",
"the Hugging Face team.\n"
]
},
{
"cell_type": "markdown",
"id": "43cba468",
"metadata": {},
"source": [
"## Model description\n"
]
},
{
"cell_type": "markdown",
"id": "457bfeee",
"metadata": {},
"source": [
"BERT is a transformers model pretrained on a large corpus of English data in a self-supervised fashion. This means it\n",
"was pretrained on the raw texts only, with no humans labelling them in any way (which is why it can use lots of\n",
"publicly available data) with an automatic process to generate inputs and labels from those texts. More precisely, it\n",
"was pretrained with two objectives:\n"
]
},
{
"cell_type": "markdown",
"id": "77c83270",
"metadata": {},
"source": [
"- Masked language modeling (MLM): taking a sentence, the model randomly masks 15% of the words in the input then run\n",
"the entire masked sentence through the model and has to predict the masked words. This is different from traditional\n",
"recurrent neural networks (RNNs) that usually see the words one after the other, or from autoregressive models like\n",
"GPT which internally mask the future tokens. It allows the model to learn a bidirectional representation of the\n",
"sentence.\n",
"- Next sentence prediction (NSP): the models concatenates two masked sentences as inputs during pretraining. Sometimes\n",
"they correspond to sentences that were next to each other in the original text, sometimes not. The model then has to\n",
"predict if the two sentences were following each other or not.\n"
]
},
{
"cell_type": "markdown",
"id": "0ba87de6",
"metadata": {},
"source": [
"This way, the model learns an inner representation of the English language that can then be used to extract features\n",
"useful for downstream tasks: if you have a dataset of labeled sentences for instance, you can train a standard\n",
"classifier using the features produced by the BERT model as inputs.\n"
"# BERT large model (uncased) whole word masking finetuned on SQuAD\n",
"\n",
"You can get more details from [Bert in PaddleNLP](https://github.com/PaddlePaddle/PaddleNLP/blob/develop/model_zoo/bert/README.md)。"
]
},
{
"cell_type": "markdown",
"id": "afef45e0",
"metadata": {},
"source": [
"Pretrained model on English language using a masked language modeling (MLM) objective. It was introduced in\n",
"[this paper](https://arxiv.org/abs/1810.04805) and first released in\n",
"[this repository](https://github.com/google-research/bert). This model is uncased: it does not make a difference\n",
"between english and English.\n"
]
},
{
"cell_type": "markdown",
"id": "c94536b9",
"metadata": {},
"source": [
"Differently to other BERT models, this model was trained with a new technique: Whole Word Masking. In this case, all of the tokens corresponding to a word are masked at once. The overall masking rate remains the same.\n"
]
},
{
"cell_type": "markdown",
"id": "50254dea",
"metadata": {},
"source": [
"The training is identical -- each masked WordPiece token is predicted independently.\n"
]
},
{
"cell_type": "markdown",
"id": "4b482be9",
"metadata": {},
"source": [
"After pre-training, this model was fine-tuned on the SQuAD dataset with one of our fine-tuning scripts. See below for more information regarding this fine-tuning.\n"
]
},
{
"cell_type": "markdown",
"id": "adfc36af",
"metadata": {},
"source": [
"Disclaimer: The team releasing BERT did not write a model card for this model so this model card has been written by\n",
"the Hugging Face team.\n"
]
},
{
"cell_type": "markdown",
"id": "22f554a7",
"metadata": {},
"source": [
"## Model description\n"
]
},
{
"cell_type": "markdown",
"id": "eccd3048",
"metadata": {},
"source": [
"BERT is a transformers model pretrained on a large corpus of English data in a self-supervised fashion. This means it\n",
"was pretrained on the raw texts only, with no humans labelling them in any way (which is why it can use lots of\n",
"publicly available data) with an automatic process to generate inputs and labels from those texts. More precisely, it\n",
"was pretrained with two objectives:\n"
]
},
{
"cell_type": "markdown",
"id": "3d4098e8",
"metadata": {},
"source": [
"- Masked language modeling (MLM): taking a sentence, the model randomly masks 15% of the words in the input then run\n",
"the entire masked sentence through the model and has to predict the masked words. This is different from traditional\n",
"recurrent neural networks (RNNs) that usually see the words one after the other, or from autoregressive models like\n",
"GPT which internally mask the future tokens. It allows the model to learn a bidirectional representation of the\n",
"sentence.\n",
"- Next sentence prediction (NSP): the models concatenates two masked sentences as inputs during pretraining. Sometimes\n",
"they correspond to sentences that were next to each other in the original text, sometimes not. The model then has to\n",
"predict if the two sentences were following each other or not.\n"
]
},
{
"cell_type": "markdown",
"id": "1047d1ad",
"metadata": {},
"source": [
"This way, the model learns an inner representation of the English language that can then be used to extract features\n",
"useful for downstream tasks: if you have a dataset of labeled sentences for instance, you can train a standard\n",
"classifier using the features produced by the BERT model as inputs.\n"
"> The model introduction and model weights originate from https://huggingface.co/bert-large-uncased-whole-word-masking-finetuned-squad and were converted to PaddlePaddle format for ease of use in PaddleNLP.\n"
"# BERT large model (uncased) whole word masking\n",
"\n",
"详细内容请看[Bert in PaddleNLP](https://github.com/PaddlePaddle/PaddleNLP/blob/develop/model_zoo/bert/README.md)。"
]
},
{
"cell_type": "markdown",
"id": "af8c3816",
"metadata": {},
"source": [
"Pretrained model on English language using a masked language modeling (MLM) objective. It was introduced in\n",
"[this paper](https://arxiv.org/abs/1810.04805) and first released in\n",
"[this repository](https://github.com/google-research/bert). This model is uncased: it does not make a difference\n",
"between english and English.\n"
]
},
{
"cell_type": "markdown",
"id": "c103e84b",
"metadata": {},
"source": [
"Differently to other BERT models, this model was trained with a new technique: Whole Word Masking. In this case, all of the tokens corresponding to a word are masked at once. The overall masking rate remains the same.\n"
]
},
{
"cell_type": "markdown",
"id": "19a76368",
"metadata": {},
"source": [
"The training is identical -- each masked WordPiece token is predicted independently.\n"
]
},
{
"cell_type": "markdown",
"id": "67f11a2c",
"metadata": {},
"source": [
"Disclaimer: The team releasing BERT did not write a model card for this model so this model card has been written by\n",
"the Hugging Face team.\n"
]
},
{
"cell_type": "markdown",
"id": "778cf97d",
"metadata": {},
"source": [
"## Model description\n"
]
},
{
"cell_type": "markdown",
"id": "dddbb307",
"metadata": {},
"source": [
"BERT is a transformers model pretrained on a large corpus of English data in a self-supervised fashion. This means it\n",
"was pretrained on the raw texts only, with no humans labelling them in any way (which is why it can use lots of\n",
"publicly available data) with an automatic process to generate inputs and labels from those texts. More precisely, it\n",
"was pretrained with two objectives:\n"
]
},
{
"cell_type": "markdown",
"id": "40becad1",
"metadata": {},
"source": [
"- Masked language modeling (MLM): taking a sentence, the model randomly masks 15% of the words in the input then run\n",
"the entire masked sentence through the model and has to predict the masked words. This is different from traditional\n",
"recurrent neural networks (RNNs) that usually see the words one after the other, or from autoregressive models like\n",
"GPT which internally mask the future tokens. It allows the model to learn a bidirectional representation of the\n",
"sentence.\n",
"- Next sentence prediction (NSP): the models concatenates two masked sentences as inputs during pretraining. Sometimes\n",
"they correspond to sentences that were next to each other in the original text, sometimes not. The model then has to\n",
"predict if the two sentences were following each other or not.\n"
]
},
{
"cell_type": "markdown",
"id": "3fc265b6",
"metadata": {},
"source": [
"This way, the model learns an inner representation of the English language that can then be used to extract features\n",
"useful for downstream tasks: if you have a dataset of labeled sentences for instance, you can train a standard\n",
"classifier using the features produced by the BERT model as inputs.\n"
"# BERT large model (uncased) whole word masking\n",
"\n",
"You can get more details from [Bert in PaddleNLP](https://github.com/PaddlePaddle/PaddleNLP/blob/develop/model_zoo/bert/README.md)。"
]
},
{
"cell_type": "markdown",
"id": "14552c09",
"metadata": {},
"source": [
"Pretrained model on English language using a masked language modeling (MLM) objective. It was introduced in\n",
"[this paper](https://arxiv.org/abs/1810.04805) and first released in\n",
"[this repository](https://github.com/google-research/bert). This model is uncased: it does not make a difference\n",
"between english and English.\n"
]
},
{
"cell_type": "markdown",
"id": "78d1e4a0",
"metadata": {},
"source": [
"Differently to other BERT models, this model was trained with a new technique: Whole Word Masking. In this case, all of the tokens corresponding to a word are masked at once. The overall masking rate remains the same.\n"
]
},
{
"cell_type": "markdown",
"id": "cdbe484a",
"metadata": {},
"source": [
"The training is identical -- each masked WordPiece token is predicted independently.\n"
]
},
{
"cell_type": "markdown",
"id": "fdbba80d",
"metadata": {},
"source": [
"Disclaimer: The team releasing BERT did not write a model card for this model so this model card has been written by\n",
"the Hugging Face team.\n"
]
},
{
"cell_type": "markdown",
"id": "aba33624",
"metadata": {},
"source": [
"## Model description\n"
]
},
{
"cell_type": "markdown",
"id": "459ca6e6",
"metadata": {},
"source": [
"BERT is a transformers model pretrained on a large corpus of English data in a self-supervised fashion. This means it\n",
"was pretrained on the raw texts only, with no humans labelling them in any way (which is why it can use lots of\n",
"publicly available data) with an automatic process to generate inputs and labels from those texts. More precisely, it\n",
"was pretrained with two objectives:\n"
]
},
{
"cell_type": "markdown",
"id": "65f2ae1a",
"metadata": {},
"source": [
"- Masked language modeling (MLM): taking a sentence, the model randomly masks 15% of the words in the input then run\n",
"the entire masked sentence through the model and has to predict the masked words. This is different from traditional\n",
"recurrent neural networks (RNNs) that usually see the words one after the other, or from autoregressive models like\n",
"GPT which internally mask the future tokens. It allows the model to learn a bidirectional representation of the\n",
"sentence.\n",
"- Next sentence prediction (NSP): the models concatenates two masked sentences as inputs during pretraining. Sometimes\n",
"they correspond to sentences that were next to each other in the original text, sometimes not. The model then has to\n",
"predict if the two sentences were following each other or not.\n"
]
},
{
"cell_type": "markdown",
"id": "86e8d7eb",
"metadata": {},
"source": [
"This way, the model learns an inner representation of the English language that can then be used to extract features\n",
"useful for downstream tasks: if you have a dataset of labeled sentences for instance, you can train a standard\n",
"classifier using the features produced by the BERT model as inputs.\n"
]
},
{
"cell_type": "markdown",
"id": "b81821d8",
"metadata": {},
"source": [
"This model has the following configuration:\n"
]
},
{
"cell_type": "markdown",
"id": "3a576172",
"metadata": {},
"source": [
"- 24-layer\n",
"- 1024 hidden dimension\n",
"- 16 attention heads\n",
"- 336M parameters.\n"
]
},
{
"cell_type": "markdown",
"id": "0038f06c",
"metadata": {},
"source": [
"## Intended uses & limitations\n"
]
},
{
"cell_type": "markdown",
"id": "ba8c18de",
"metadata": {},
"source": [
"You can use the raw model for either masked language modeling or next sentence prediction, but it's mostly intended to\n",
"be fine-tuned on a downstream task. See the [model hub](https://huggingface.co/models?filter=bert) to look for\n",
"fine-tuned versions on a task that interests you.\n"
]
},
{
"cell_type": "markdown",
"id": "bb72ad39",
"metadata": {},
"source": [
"Note that this model is primarily aimed at being fine-tuned on tasks that use the whole sentence (potentially masked)\n",
"to make decisions, such as sequence classification, token classification or question answering. For tasks such as text\n",
"generation you should look at model like GPT2.\n"
]
},
{
"cell_type": "markdown",
"id": "54b59ca8",
"metadata": {},
"source": [
"### How to use\n"
]
},
{
"cell_type": "markdown",
"id": "a0ff2a80",
"metadata": {},
"source": [
"You can use this model directly with a pipeline for masked language modeling:\n"
"> The model introduction and model weights originate from https://huggingface.co/bert-large-uncased-whole-word-masking and were converted to PaddlePaddle format for ease of use in PaddleNLP.\n"
"> The model introduction and model weights originate from https://huggingface.co/bert-large-uncased and were converted to PaddlePaddle format for ease of use in PaddleNLP."
"# Model Card for DistilBERT base multilingual (cased)\n",
"\n",
"详细内容请看[Bert in PaddleNLP](https://github.com/PaddlePaddle/PaddleNLP/blob/develop/model_zoo/bert/README.md)。\n"
]
},
{
"cell_type": "markdown",
"id": "a1024bec",
"metadata": {},
"source": [
"## Model Description\n"
]
},
{
"cell_type": "markdown",
"id": "bcdfe024",
"metadata": {},
"source": [
"This model is a distilled version of the [BERT base multilingual model](https://huggingface.co/bert-base-multilingual-cased/). The code for the distillation process can be found [here](https://github.com/huggingface/transformers/tree/main/examples/research_projects/distillation). This model is cased: it does make a difference between english and English.\n"
]
},
{
"cell_type": "markdown",
"id": "5051aaa6",
"metadata": {},
"source": [
"The model is trained on the concatenation of Wikipedia in 104 different languages listed [here](https://github.com/google-research/bert/blob/master/multilingual.md#list-of-languages).\n",
"The model has 6 layers, 768 dimension and 12 heads, totalizing 134M parameters (compared to 177M parameters for mBERT-base).\n",
"On average, this model, referred to as DistilmBERT, is twice as fast as mBERT-base.\n"
]
},
{
"cell_type": "markdown",
"id": "cdddc273",
"metadata": {},
"source": [
"We encourage potential users of this model to check out the [BERT base multilingual model card](https://huggingface.co/bert-base-multilingual-cased) to learn more about usage, limitations and potential biases.\n"
]
},
{
"cell_type": "markdown",
"id": "8eebedbf",
"metadata": {},
"source": [
"- **Developed by:** Victor Sanh, Lysandre Debut, Julien Chaumond, Thomas Wolf (Hugging Face)\n",
"- **Model type:** Transformer-based language model\n",
"- **Language(s) (NLP):** 104 languages; see full list [here](https://github.com/google-research/bert/blob/master/multilingual.md#list-of-languages)\n",
"- **License:** Apache 2.0\n",
"- **Related Models:** [BERT base multilingual model](https://huggingface.co/bert-base-multilingual-cased)\n",
"# Model Card for DistilBERT base multilingual (cased)\n",
"\n",
"You can get more details from [Bert in PaddleNLP](https://github.com/PaddlePaddle/PaddleNLP/blob/develop/model_zoo/bert/README.md)。"
]
},
{
"cell_type": "markdown",
"id": "53f1b1c2",
"metadata": {},
"source": [
"This model is a distilled version of the [BERT base multilingual model](https://huggingface.co/bert-base-multilingual-cased/). The code for the distillation process can be found [here](https://github.com/huggingface/transformers/tree/main/examples/research_projects/distillation). This model is cased: it does make a difference between english and English.\n"
]
},
{
"cell_type": "markdown",
"id": "f417583b",
"metadata": {},
"source": [
"The model is trained on the concatenation of Wikipedia in 104 different languages listed [here](https://github.com/google-research/bert/blob/master/multilingual.md#list-of-languages).\n",
"The model has 6 layers, 768 dimension and 12 heads, totalizing 134M parameters (compared to 177M parameters for mBERT-base).\n",
"On average, this model, referred to as DistilmBERT, is twice as fast as mBERT-base.\n",
"\n",
"- **Developed by:** Victor Sanh, Lysandre Debut, Julien Chaumond, Thomas Wolf (Hugging Face)\n",
"- **Model type:** Transformer-based language model\n",
"- **Language(s) (NLP):** 104 languages; see full list [here](https://github.com/google-research/bert/blob/master/multilingual.md#list-of-languages)\n",
"- **License:** Apache 2.0\n",
"- **Related Models:** [BERT base multilingual model](https://huggingface.co/bert-base-multilingual-cased)\n",
" title={DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter},\n",
" author={Victor Sanh and Lysandre Debut and Julien Chaumond and Thomas Wolf},\n",
" journal={ArXiv},\n",
" year={2019},\n",
" volume={abs/1910.01108}\n",
"}\n",
"```\n"
]
},
{
"cell_type": "markdown",
"id": "0ee03d6a",
"metadata": {},
"source": [
"> The model introduction and model weights originate from https://huggingface.co/distilbert-base-multilingual-cased and were converted to PaddlePaddle format for ease of use in PaddleNLP.\n"
"详细内容请看[GPT2 in PaddleNLP](https://github.com/PaddlePaddle/PaddleNLP/blob/develop/model_zoo/gpt/README.md)。"
]
},
{
"cell_type": "markdown",
"id": "20c299c9",
"metadata": {},
"source": [
"DistilGPT2 (short for Distilled-GPT2) is an English-language model pre-trained with the supervision of the smallest version of Generative Pre-trained Transformer 2 (GPT-2). Like GPT-2, DistilGPT2 can be used to generate text. Users of this model card should also consider information about the design, training, and limitations of GPT-2.\n"
]
},
{
"cell_type": "markdown",
"id": "c624b3d1",
"metadata": {},
"source": [
"## Model Details\n"
]
},
{
"cell_type": "markdown",
"id": "92002396",
"metadata": {},
"source": [
"- **Developed by:** Hugging Face\n",
"- **Model type:** Transformer-based Language Model\n",
"- **Language:** English\n",
"- **License:** Apache 2.0\n",
"- **Model Description:** DistilGPT2 is an English-language model pre-trained with the supervision of the 124 million parameter version of GPT-2. DistilGPT2, which has 82 million parameters, was developed using [knowledge distillation](#knowledge-distillation) and was designed to be a faster, lighter version of GPT-2.\n",
"- **Resources for more information:** See this repository for more about Distil\\* (a class of compressed models including Distilled-GPT2), [Sanh et al. (2019)](https://arxiv.org/abs/1910.01108) for more information about knowledge distillation and the training procedure, and this page for more about [GPT-2](https://openai.com/blog/better-language-models/).\n"
"title={DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter},\n",
"author={Sanh, Victor and Debut, Lysandre and Chaumond, Julien and Wolf, Thomas},\n",
"booktitle={NeurIPS EMC^2 Workshop},\n",
"year={2019}\n",
"}\n",
"```"
]
},
{
"cell_type": "markdown",
"id": "7966636a",
"metadata": {},
"source": [
"## Glossary\n"
]
},
{
"cell_type": "markdown",
"id": "533038ef",
"metadata": {},
"source": [
"-\t<a name=\"knowledge-distillation\">**Knowledge Distillation**</a>: As described in [Sanh et al. (2019)](https://arxiv.org/pdf/1910.01108.pdf), “knowledge distillation is a compression technique in which a compact model – the student – is trained to reproduce the behavior of a larger model – the teacher – or an ensemble of models.” Also see [Bucila et al. (2006)](https://www.cs.cornell.edu/~caruana/compression.kdd06.pdf) and [Hinton et al. (2015)](https://arxiv.org/abs/1503.02531).\n"
"You can get more details from [GPT2 in PaddleNLP](https://github.com/PaddlePaddle/PaddleNLP/blob/develop/model_zoo/gpt/README.md)."
]
},
{
"cell_type": "markdown",
"id": "f3ab8949",
"metadata": {},
"source": [
"DistilGPT2 (short for Distilled-GPT2) is an English-language model pre-trained with the supervision of the smallest version of Generative Pre-trained Transformer 2 (GPT-2). Like GPT-2, DistilGPT2 can be used to generate text. Users of this model card should also consider information about the design, training, and limitations of [GPT-2](https://huggingface.co/gpt2).\n"
]
},
{
"cell_type": "markdown",
"id": "c6fbc1da",
"metadata": {},
"source": [
"## Model Details\n"
]
},
{
"cell_type": "markdown",
"id": "e2929e2f",
"metadata": {},
"source": [
"- **Developed by:** Hugging Face\n",
"- **Model type:** Transformer-based Language Model\n",
"- **Language:** English\n",
"- **License:** Apache 2.0\n",
"- **Model Description:** DistilGPT2 is an English-language model pre-trained with the supervision of the 124 million parameter version of GPT-2. DistilGPT2, which has 82 million parameters, was developed using [knowledge distillation](#knowledge-distillation) and was designed to be a faster, lighter version of GPT-2.\n",
"- **Resources for more information:** See [this repository](https://github.com/huggingface/transformers/tree/main/examples/research_projects/distillation) for more about Distil\\* (a class of compressed models including Distilled-GPT2), [Sanh et al. (2019)](https://arxiv.org/abs/1910.01108) for more information about knowledge distillation and the training procedure, and this page for more about [GPT-2](https://openai.com/blog/better-language-models/).\n"
"title={DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter},\n",
"author={Sanh, Victor and Debut, Lysandre and Chaumond, Julien and Wolf, Thomas},\n",
"booktitle={NeurIPS EMC^2 Workshop},\n",
"year={2019}\n",
"}\n",
"```"
]
},
{
"cell_type": "markdown",
"id": "7d2aaec2",
"metadata": {},
"source": [
"## Glossary\n"
]
},
{
"cell_type": "markdown",
"id": "004026dd",
"metadata": {},
"source": [
"-\t<a name=\"knowledge-distillation\">**Knowledge Distillation**</a>: As described in [Sanh et al. (2019)](https://arxiv.org/pdf/1910.01108.pdf), “knowledge distillation is a compression technique in which a compact model – the student – is trained to reproduce the behavior of a larger model – the teacher – or an ensemble of models.” Also see [Bucila et al. (2006)](https://www.cs.cornell.edu/~caruana/compression.kdd06.pdf) and [Hinton et al. (2015)](https://arxiv.org/abs/1503.02531).\n"
"> The model introduction and model weights originate from [https://huggingface.co/distilgpt2](https://huggingface.co/distilgpt2) and were converted to PaddlePaddle format for ease of use in PaddleNLP.\n"
"This model is a distilled version of the [RoBERTa-base model](https://huggingface.co/roberta-base). It follows the same training procedure as [DistilBERT](https://huggingface.co/distilbert-base-uncased).\n",
"The code for the distillation process can be found [here](https://github.com/huggingface/transformers/tree/master/examples/distillation).\n",
"This model is case-sensitive: it makes a difference between english and English.\n"
]
},
{
"cell_type": "markdown",
"id": "10aefe84",
"metadata": {},
"source": [
"The model has 6 layers, 768 dimension and 12 heads, totalizing 82M parameters (compared to 125M parameters for RoBERTa-base).\n",
"On average DistilRoBERTa is twice as fast as Roberta-base.\n"
]
},
{
"cell_type": "markdown",
"id": "d7ebd775",
"metadata": {},
"source": [
"We encourage users of this model card to check out the [RoBERTa-base model card](https://huggingface.co/roberta-base) to learn more about usage, limitations and potential biases.\n"
]
},
{
"cell_type": "markdown",
"id": "423d28b1",
"metadata": {},
"source": [
"- **Developed by:** Victor Sanh, Lysandre Debut, Julien Chaumond, Thomas Wolf (Hugging Face)\n",
"- **Model type:** Transformer-based language model\n",
"- **Language(s) (NLP):** English\n",
"- **License:** Apache 2.0\n",
"- **Related Models:** [RoBERTa-base model card](https://huggingface.co/roberta-base)\n",
"> The model introduction and model weights originate from [https://huggingface.co/distilroberta-base](https://huggingface.co/distilroberta-base) and were converted to PaddlePaddle format for ease of use in PaddleNLP.\n"
"**Model Description:** GPT-2 Large is the **774M parameter** version of GPT-2, a transformer-based language model created and released by OpenAI. The model is a pretrained model on English language using a causal language modeling (CLM) objective.\n"
]
},
{
"cell_type": "markdown",
"id": "15b8f634",
"metadata": {},
"source": [
"- **Developed by:** OpenAI, see [associated research paper](https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf) and [GitHub repo](https://github.com/openai/gpt-2) for model developers.\n",
"- **Model Type:** Transformer-based language model\n",
"- **Language(s):** English\n",
"- **License:** [Modified MIT License](https://github.com/openai/gpt-2/blob/master/LICENSE)\n",
"- **Related Models:** [GPT-2](https://huggingface.co/gpt2), [GPT-Medium](https://huggingface.co/gpt2-medium) and [GPT-XL](https://huggingface.co/gpt2-xl)\n",
"**Model Description:** GPT-2 Large is the **774M parameter** version of GPT-2, a transformer-based language model created and released by OpenAI. The model is a pretrained model on English language using a causal language modeling (CLM) objective.\n"
]
},
{
"cell_type": "markdown",
"id": "0155f43f",
"metadata": {},
"source": [
"- **Developed by:** OpenAI, see [associated research paper](https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf) and [GitHub repo](https://github.com/openai/gpt-2) for model developers.\n",
"- **Model Type:** Transformer-based language model\n",
"- **Language(s):** English\n",
"- **License:** [Modified MIT License](https://github.com/openai/gpt-2/blob/master/LICENSE)\n",
"- **Related Models:** https://huggingface.co/gpt2, GPT-Medium and GPT-XL\n",
"title={Language models are unsupervised multitask learners},\n",
"author={Radford, Alec and Wu, Jeffrey and Child, Rewon and Luan, David and Amodei, Dario and Sutskever, Ilya and others},\n",
"journal={OpenAI blog},\n",
"volume={1},\n",
"number={8},\n",
"pages={9},\n",
"year={2019}\n",
"}\n",
"```"
]
},
{
"cell_type": "markdown",
"id": "7cded70d",
"metadata": {},
"source": [
"## Model Card Authors\n"
]
},
{
"cell_type": "markdown",
"id": "ff9ab2d4",
"metadata": {},
"source": [
"This model card was written by the Hugging Face team.\n",
"\n",
"> The model introduction and model weights originate from [https://huggingface.co/gpt2-large](https://huggingface.co/gpt2-large) and were converted to PaddlePaddle format for ease of use in PaddleNLP."
"详细内容请看[GPT2 in PaddleNLP](https://github.com/PaddlePaddle/PaddleNLP/blob/develop/model_zoo/gpt/README.md)。"
]
},
{
"cell_type": "markdown",
"id": "806177e3",
"metadata": {},
"source": [
"## Model Details\n"
]
},
{
"cell_type": "markdown",
"id": "dbcaecb0",
"metadata": {},
"source": [
"**Model Description:** GPT-2 Medium is the **355M parameter** version of GPT-2, a transformer-based language model created and released by OpenAI. The model is a pretrained model on English language using a causal language modeling (CLM) objective.\n"
]
},
{
"cell_type": "markdown",
"id": "ab73e9f0",
"metadata": {},
"source": [
"- **Developed by:** OpenAI, see [associated research paper](https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf) and [GitHub repo](https://github.com/openai/gpt-2) for model developers.\n",
"- **Model Type:** Transformer-based language model\n",
"- **Language(s):** English\n",
"- **License:** [Modified MIT License](https://github.com/openai/gpt-2/blob/master/LICENSE)\n",
"- **Related Models:** [GPT2](https://huggingface.co/gpt2), [GPT2-Large](https://huggingface.co/gpt2-large) and [GPT2-XL](https://huggingface.co/gpt2-xl)\n",
"You can get more details from [GPT2 in PaddleNLP](https://github.com/PaddlePaddle/PaddleNLP/blob/develop/model_zoo/gpt/README.md)."
]
},
{
"cell_type": "markdown",
"id": "f695ad73",
"metadata": {},
"source": [
"## Model Details\n"
]
},
{
"cell_type": "markdown",
"id": "5a8170d9",
"metadata": {},
"source": [
"**Model Description:** GPT-2 Medium is the **355M parameter** version of GPT-2, a transformer-based language model created and released by OpenAI. The model is a pretrained model on English language using a causal language modeling (CLM) objective.\n"
]
},
{
"cell_type": "markdown",
"id": "1d0dc244",
"metadata": {},
"source": [
"- **Developed by:** OpenAI, see [associated research paper](https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf) and [GitHub repo](https://github.com/openai/gpt-2) for model developers.\n",
"- **Model Type:** Transformer-based language model\n",
"- **Language(s):** English\n",
"- **License:** [Modified MIT License](https://github.com/openai/gpt-2/blob/master/LICENSE)\n",
"- **Related Models:** [GPT2](https://huggingface.co/gpt2), [GPT2-Large](https://huggingface.co/gpt2-large) and [GPT2-XL](https://huggingface.co/gpt2-xl)\n",
"title={Language models are unsupervised multitask learners},\n",
"author={Radford, Alec and Wu, Jeffrey and Child, Rewon and Luan, David and Amodei, Dario and Sutskever, Ilya and others},\n",
"journal={OpenAI blog},\n",
"volume={1},\n",
"number={8},\n",
"pages={9},\n",
"year={2019}\n",
"}\n",
"```"
]
},
{
"cell_type": "markdown",
"id": "eb327c10",
"metadata": {},
"source": [
"## Model Card Authors\n"
]
},
{
"cell_type": "markdown",
"id": "50fb7de8",
"metadata": {},
"source": [
"> The model introduction and model weights originate from https://huggingface.co/gpt2-medium and were converted to PaddlePaddle format for ease of use in PaddleNLP."
"> The model introduction and model weights originate from [https://huggingface.co/gpt2](https://huggingface.co/gpt2) and were converted to PaddlePaddle format for ease of use in PaddleNLP.\n"
"[IndoBERT](https://arxiv.org/abs/2009.05387) is a state-of-the-art language model for Indonesian based on the BERT model. The pretrained model is trained using a masked language modeling (MLM) objective and next sentence prediction (NSP) objective.\n"
"title={IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understanding},\n",
"author={Bryan Wilie and Karissa Vincentio and Genta Indra Winata and Samuel Cahyawijaya and X. Li and Zhi Yuan Lim and S. Soleman and R. Mahendra and Pascale Fung and Syafri Bahar and A. Purwarianti},\n",
"booktitle={Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing},\n",
"[IndoBERT](https://arxiv.org/abs/2009.05387) is a state-of-the-art language model for Indonesian based on the BERT model. The pretrained model is trained using a masked language modeling (MLM) objective and next sentence prediction (NSP) objective.\n"
"title={IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understanding},\n",
"author={Bryan Wilie and Karissa Vincentio and Genta Indra Winata and Samuel Cahyawijaya and X. Li and Zhi Yuan Lim and S. Soleman and R. Mahendra and Pascale Fung and Syafri Bahar and A. Purwarianti},\n",
"booktitle={Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing},\n",
"year={2020}\n",
"}\n",
"```"
]
},
{
"cell_type": "markdown",
"id": "e155d1ce",
"metadata": {},
"source": [
"> The model introduction and model weights originate from [https://huggingface.co/indobenchmark/indobert-base-p1](https://huggingface.co/indobenchmark/indobert-base-p1) and were converted to PaddlePaddle format for ease of use in PaddleNLP.\n"
