This dataset is for our paper: ARNOR: Attention Regularization based Noise Reduction for Distant Supervision Relation Classification. This test set is for sentence-level evaluation.
This dataset is used in our paper: "ARNOR: Attention Regularization based Noise Reduction for Distant Supervision Relation Classification". We release a new NYT test set for sentence-level evaluation of distant supervision relation extraction model. It increases almost 5 times positive instances than the previous one [2]. And it is carefully annotated to ensure accuracy.
The original data is from the dataset in the paper: Cotype: Joint extraction of typed entities and relations with knowledge bases. It is a distant supervision dataset from NYT (New York Time). And the test set is annotated by humans. However the number of positive instances in test set is small. We revise and annotate more test data based on it.
This dataset is based on Ren's [1] training set that is generated by distant supervision, and a manually annotated test set that contains 395 sentences from Hoffmann [2]. They are all from New York Times news articles [3]. However the number of positive instances in test set is small (only 396), and the quality is insufficient. We revise and annotate more test data based on it, and release two versions of datasets.
In a data file, each line is a json string. The content is like
...
...
@@ -18,7 +18,7 @@ In a data file, each line is a json string. The content is like
},
...
],
"entityMentions": [
"entityMentions": [
{
"text": "Entity words",
"label": "Entity type",
...
...
@@ -32,10 +32,74 @@ In a data file, each line is a json string. The content is like
Data version 1.0.0
=====
This version of dataset is the original one applied in our paper, which includes four files: train.json, test.json, dev_part.json, and test_part.json. Here dev_part.json and test_part.json are from test.json. This dataset can be downloaded here: https://baidu-nlp.bj.bcebos.com/arnor_dataset-1.0.0.tar.gz
This version of dataset is the original one applied in our paper, which includes four files: train.json, test.json, dev_part.json, and test_part.json. Here dev_part.json and test_part.json are from test.json. **This dataset can be downloaded here: https://baidu-nlp.bj.bcebos.com/arnor_dataset-1.0.0.tar.gz**
Data version 2.0.0
=====
More test date are coming soon ......
We strongly recommend to apply this dataset in later relation classification studies. This version contains more annotated test data comparing with version 1.0.0. We continuely annotated more data that is shown in the below table. What is more, we have removed the relation "/location/administrative_division/country" from the training set and changed "/location/country/administrative_divisions" into "/location/location/contains". Because we do not label these two relation types in test set.
| Test set | version 1.0.0 | version 2.0.0 |
| :-----| :-----| :-----|
| #Sentences | 1,024 | 3,192 |
| #Instances | 4,543 | 9,051 |
| #Positive instances | 671 | 2,224 |
**The download address is: http://baidu-nlp.bj.bcebos.com/arnor_dataset-2.0.0.tar.gz**
There are four files in it. Training set, dev set, and test set are all included. In addition, it also includes a "test_noise.json" file, which is for noise reduction evaluation.
Model Performances on version 2.0.0
-----
We reproduce experiments following our ARNOR paper. The results are listed below.
Main results:
| Method | Dev Prec. | Dev Rec. | Dev F1 | Test Prec. | Test Rec. | Test F1 |
title={ARNOR: Attention Regularization based Noise Reduction for Distant Supervision Relation Classification},
author={Jia, Wei and Dai, Dai and Xiao, Xinyan and Wu, Hua},
booktitle={Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
year={2019}
}
References
-----
[1] Xiang Ren, Zeqiu Wu, Wenqi He, Meng Qu, Clare R Voss, Heng Ji, Tarek F Abdelzaher, and Jiawei Han. 2017. Cotype: Joint extraction of typed entities and relations with knowledge bases. In Proceedings of the 26th International Conference on World Wide Web, pages 1015–1024. International World Wide Web Conferences Steering Committee.
[2] Raphael Hoffmann, Congle Zhang, Xiao Ling, Luke Zettlemoyer, and Daniel S Weld. 2011. Knowledge- based weak supervision for information extraction of overlapping relations. In Proceedings of the 49th Annual Meeting of the Association for Computa- tional Linguistics: Human Language Technologies- Volume 1, pages 541–550. Association for Compu- tational Linguistics.
[3] Sebastian Riedel, Limin Yao, and Andrew McCallum. 2010. Modeling relations and their mentions with- out labeled text. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 148–163. Springer.
