faster_tokenizer

to_tensor(string_values, name='text')

Create the tensor that the value holds the list of string. NOTICE: The value will be holded in the cpu place.

参数:

• string_values (list[string]) -- The value will be setted to the tensor.

• name (string) -- The name of the tensor.

to_vocab_buffer(vocab_dict, name)

Create the tensor that the value holds the map, the type of key is the string. NOTICE: The value will be holded in the cpu place.

参数:

• vocab_dict (dict) -- The value will be setted to the tensor. The key is token and the value is the token index.

• name (string) -- The name of the tensor.

class FasterTokenizer(vocab, do_lower_case=False, is_split_into_words=False)

基类：Layer

forward(text, text_pair=None, max_seq_len=0, pad_to_max_seq_len=False)

Defines the computation performed at every call. Should be overridden by all subclasses.

参数:

• *inputs (tuple) -- unpacked tuple arguments

• **kwargs (dict) -- unpacked dict arguments