Ask Your Question
1

What is the method of obtaining the original word-level entity instead of wordpiece tokens in BERT NER?

asked 2023-07-12 22:25:12 +0000

qstack gravatar image

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted
0

answered 2023-07-12 22:48:02 +0000

lalupa gravatar image

One method to obtain the original word-level entity instead of wordpiece tokens in BERT NER is to align the wordpiece tokens to the original text by keeping track of the start and end characters of each token. Once the alignment is done, the labels assigned to the wordpiece tokens can be extended to the original words by considering only the label assigned to the first token of each word and propagating it to the remaining tokens of the word. This approach is called "token-level to word-level inference" and can be performed using heuristics or machine learning models.

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account. This space is reserved only for answers. If you would like to engage in a discussion, please instead post a comment under the question or an answer that you would like to discuss

Add Answer


Question Tools

Stats

Asked: 2023-07-12 22:25:12 +0000

Seen: 11 times

Last updated: Jul 12 '23