Geological Named Entity Recognition Based on MacBERT and R-Drop

NAVIGATE

Table of Contents

STATISTICS

Viewed290

Downloads495

Geological Named Entity Recognition Based on MacBERT and R-Drop

[HTML] PDF下载 (495)

[1]LIU Xin,XU Hongzhen,LIU Aihua,et al.Geological Named Entity Recognition Based on MacBERT and R-Drop[J].Journal of Zhengzhou University (Engineering Science),2024,45(03):89-95.[doi:10. 13705/ j. issn. 1671-6833. 2024. 03. 002]

Copy

Journal of Zhengzhou University (Engineering Science)[ISSN 1671-6833/CN 41-1339/T] Volume: 45 Number of periods: 2024 03 Page number: 89-95 Column: Public date: 2024-04-20

Title:: Geological Named Entity Recognition Based on MacBERT and R-Drop

Author(s):: LIU Xin¹; XU Hongzhen¹; 2; LIU Aihua²; DENG Dejun¹; 1. School of Information Engineering, East China University of Technology, Nanchang 330013, China; 2. School of Software, East China University of Technology, Nanchang 330013, China

Keywords:: named entity recognition; geology; MacBERT; BiGRU; R-Drop

CLC:: TP311

DOI:: 10. 13705/ j. issn. 1671-6833. 2024. 03. 002

Abstract:: The commonly used deep learning methods based on BERT pre-trained model in geological named entity recognition were character-based approaches, and could not utilize word-level information. Additionally, the dropout mechanism in neural networks might cause inconsistency between the training and inference stage. To address this issue, a geological named entity recognition model MBCR based on MacBERT and R-Drop was proposed. Firstly, MacBERT was used to learn text feature representations, which could fully utilize character and word information. Then, BiGRU was employed to encode context features, effectively extracting complete semantic information. Subsequently, CRF was adopted to capture dependencies between labels and generate the optimal label sequence. Moreover, R-Drop was introduced during the training process to further enhance the model′s generalization capabilities. Compared with BiLSTM-CRF, BERT-BiLSTM-CRF, and other models, the proposed MBCR model improved the F1-score on the NERdata dataset by 2. 08-4. 62 percentage points and on the Boson dataset by 1. 26-17. 54 percentage points.

References:: [1] WANG C S, HAZEN R M, CHENG Q M, et al. The deep-time digital earth program: data-driven discovery in geosciences[J]. National Science Review, 2021, 8(9): nwab027.
[2] 马凯, 田苗, 谭永健, 等. 基于四份区域地质调查报告构建的命名实体识别试验数据集研发[J]. 全球变化数据学报( 中英文), 2022, 6 ( 1 ): 78 - 84,237-243.
MA K, TIAN M, TAN Y J, et al. Development of a named entity recognition dataset based on four regional geological survey reports[ J]. Journal of Global Change Data & Discovery, 2022, 6(1): 78-84, 237-243.
[3] QIU Q J, XIE Z, WU L, et al. Automatic spatiotemporal and semantic information extraction from unstructured geosciences reports using text mining techniques[J]. Earth Science Informatics, 2020, 13(4): 1393-1410.
[4] 储德平, 万波, 李红, 等. 基于ELMO-CNN-BiLSTMCRF 模型的地质实体识别[J]. 地球科学, 2021, 46(8): 3039-3048.
CHU D P, WAN B, LI H, et al. Geological entity recognition based on ELMO-CNN-BiLSTM-CRF model [ J].Earth Science, 2021, 46(8): 3039-3048.
[5] DEVLIN J, CHANG M W, LEE K, et al. Bert: pretraining of deep bidirectional transformers for language understanding[EB/ OL]. (2018 - 10 - 11) [2023 - 03 -15]. https:∥arxiv. org/ abs/ 1810. 04805.
[6] ZOLNA K, ARPIT D, SUHUBDY D, et al. Fraternal dropout[EB/ OL]. (2017-10-31)[2023-03-15]. https:∥arxiv. org/ abs/ 1711. 00066.
[7] CUI Y M, CHE W X, LIU T, et al. Pre-training with whole word masking for Chinese BERT[J]. IEEE/ ACM Transactions on Audio, Speech, and Language Processing, 2021, 29: 3504-3514.
[8] LIANG X B, WU L J, LI J T, et al. R-drop: regularized dropout for neural networks [ J/ OL]. ( 2021 - 10 - 29) [2023-03-15]. https:∥arxiv. org/ abs/ 2106. 14448.
[9] LIU P, GUO Y M, WANG F L, et al. Chinese named entity recognition: the state of the art[J]. Neurocomputing, 2022, 473: 37-53.
[10] LI J, SUN A X, HAN J L, et al. A survey on deep learning for named entity recognition[J]. IEEE Transactionson Knowledge and Data Engineering, 2022, 34(1):50-70.
[11] ZHANG J, SHEN D, ZHOU G D, et al. Enhancing HMM-based biomedical named entity recognition by studying special phenomena[J]. Journal of Biomedical Informatics, 2004, 37(6): 411-422.
[12] SAHA S K, SARKAR S, MITRA P. Feature selection techniques for maximum entropy based biomedical named entity recognition[J]. Journal of Biomedical Informatics, 2009, 42(5): 905-911.
[13] SUN C J, GUAN Y, WANG X L, et al. Rich features based conditional random fields for biological named entities recognition[J]. Computers in Biology and Medicine,2007, 37(9): 1327-1333.
[14] 张雪英, 叶鹏, 王曙, 等. 基于深度信念网络的地质实体识别方法[J]. 岩石学报, 2018, 34(2): 343-351.
ZHANG X Y, YE P, WANG S, et al. Geological entity recognition method based on deep belief networks [ J]. Acta Petrologica Sinica, 2018, 34(2): 343-351.
[15] QIU Q J, XIE Z, WU L, et al. BiLSTM-CRF for geological named entity recognition from the geoscience literature[J]. Earth Science Informatics, 2019, 12(4): 565-579.
[16] MIKOLOV T, SUTSKEVER I, CHEN K, et al. Distributed representations of words and phrases and their compositionality[C]∥Proceedings of the 26th International Conferenceon Neural Information Processing Systems: Volume2. New York: ACM, 2013: 3111-3119.
[17] HUANG C, WANG Y Z, YU Y Q, et al. Chinese named entity recognition of geological news based on BERT model[J]. Applied Sciences, 2022, 12(15): 7708.
[18] 王权于, 李振华, 涂志鹏, 等. 基于BERT-BiGRUCRF 模型的岩土工程实体识别[J]. 地球科学, 2023,48(8): 3137-3150.
WANG Q Y, LI Z H, TU Z P, et al. Geotechnical named entity recognition based on BERT-BiGRU-CRF model[J]. Earth Science, 2023, 48(8): 3137-3150.
[19] YU Y Q, WANG Y Z, MU J Q, et al. Chinese mineral named entity recognition based on BERT model[J]. Expert Systems with Applications, 2022, 206: 117727.
[20] LIU H, QIU Q J, WU L, et al. Few-shot learning for name entity recognition in geological text based on GeoBERT[J]. Earth Science Informatics, 2022, 15(2):979-991.
[21] CHO K, VAN M B, GULCEHRE C, et al. Learning phrase representations using RNN encoder-decoder for statistical machine translation[EB/ OL]. (2014-06-03)[2023-03-15]. https:∥arxiv. org/ abs/ 1406. 1078.
[22] STRUBELL E, VERGA P, BELANGER D, et al. Fast and accurate entity recognition with iterated dilated convolutions[EB/ OL]. (2017-02-07) [2023-03-15]. https:∥arxiv. org/ abs/ 1702. 02098.
[23] HUANG Z H, XU W, YU K. Bidirectional LSTM-CRF models for sequence tagging[EB/ OL]. (2015-08-09)[2023-03-15]. https:∥arxiv. org/ abs/ 1508. 01991.
[24] CUI Z J, YUAN Z M, WU Y F, et al. Intelligent recommendation for departments based on medical knowledge graph[J]. IEEE Access, 2023, 11: 25372-25385.

Similar References:

Memo

Last Update: 2024-04-29