tokenizer#

class BertJapaneseTokenizer(vocab_file, do_lower_case=False, do_word_tokenize=True, do_subword_tokenize=True, word_tokenizer_type='mecab', subword_tokenizer_type='wordpiece', never_split=None, mecab_kwargs=None, unk_token='[UNK]', sep_token='[SEP]', pad_token='[PAD]', cls_token='[CLS]', mask_token='[MASK]', **kwargs)[source]#

Bases: BertTokenizer

Construct a BERT tokenizer for Japanese text, based on a MecabTokenizer.

Parameters:

vocab_file (str) – The vocabulary file path (ends with ‘.txt’) required to instantiate a WordpieceTokenizer.
do_lower_case (bool, optional) – Whether or not to lowercase the input when tokenizing. Defaults to`False`.
do_word_tokenize (bool, optional) – Whether to do word tokenization. Defaults to`True`.
do_subword_tokenize (bool, optional) – Whether to do subword tokenization. Defaults to`True`.
word_tokenizer_type (str, optional) – Type of word tokenizer. Defaults to`basic`.
subword_tokenizer_type (str, optional) – Type of subword tokenizer. Defaults to`wordpiece`.
never_split (bool, optional) – Kept for backward compatibility purposes. Defaults to`None`.
mecab_kwargs (str, optional) – Dictionary passed to the MecabTokenizer constructor.
unk_token (str) – A special token representing the unknown (out-of-vocabulary) token. An unknown token is set to be unk_token inorder to be converted to an ID. Defaults to “[UNK]”.
sep_token (str) – A special token separating two different sentences in the same input. Defaults to “[SEP]”.
pad_token (str) – A special token used to make arrays of tokens the same size for batching purposes. Defaults to “[PAD]”.
cls_token (str) – A special token used for sequence classification. It is the last token of the sequence when built with special tokens. Defaults to “[CLS]”.
mask_token (str) – A special token representing a masked token. This is the token used in the masked language modeling task which the model tries to predict the original unmasked ones. Defaults to “[MASK]”.

Examples

from paddlenlp.transformers import BertJapaneseTokenizer
tokenizer = BertJapaneseTokenizer.from_pretrained('iverxin/bert-base-japanese/')

inputs = tokenizer('こんにちは')
print(inputs)

'''
{'input_ids': [2, 10350, 25746, 28450, 3], 'token_type_ids': [0, 0, 0, 0, 0]}
'''

class MecabTokenizer(do_lower_case=False, never_split=None, normalize_text=True, mecab_dic='ipadic', mecab_option=None)[source]#

Bases: object

Runs basic tokenization with MeCab morphological parser.

tokenize(text, never_split=None, **kwargs)[source]#: Tokenizes a piece of text.

class CharacterTokenizer(vocab, unk_token, normalize_text=True)[source]#

Bases: object

Runs Character tokenization.

tokenize(text)[source]#

Tokenizes a piece of text into characters.

For example, input = "apple"" wil return as output ["a", "p", "p", "l", "e"].

Parameters:: text – A single token or whitespace separated tokens. This should have already been passed through BasicTokenizer.
Returns:: A list of characters.

tokenizer

Contents

tokenizer#