Revision history [back]

One method to obtain the original word-level entity instead of wordpiece tokens in BERT NER is to align the wordpiece tokens to the original text by keeping track of the start and end characters of each token. Once the alignment is done, the labels assigned to the wordpiece tokens can be extended to the original words by considering only the label assigned to the first token of each word and propagating it to the remaining tokens of the word. This approach is called "token-level to word-level inference" and can be performed using heuristics or machine learning models.