modeling

Modeling classes for UNIMO model.

class UNIMOPretrainedModel(*args, **kwargs)

基类：paddlenlp.transformers.model_utils.PretrainedModel

An abstract class for pretrained UNIMO models. It provides UNIMO related model_config_file, pretrained_init_configuration, resource_files_names, pretrained_resource_files_map, base_model_prefix for downloading and loading pretrained models. See PretrainedModel for more details.

base_model_class

alias of paddlenlp.transformers.unimo.modeling.UNIMOModel

class UNIMOModel(vocab_size, hidden_size=768, num_hidden_layers=12, num_attention_heads=12, intermediate_size=3072, hidden_act='relu', hidden_dropout_prob=0.1, attention_probs_dropout_prob=0.1, normalize_before=False, max_position_embeddings=513, type_vocab_size=4, initializer_range=0.02, unk_token_id=17963, pad_token_id=0, bos_token_id=1, eos_token_id=3, mask_token_id=3)

基类：paddlenlp.transformers.unimo.modeling.UNIMOPretrainedModel

The bare UNIMO Model outputting raw hidden-states.

This model inherits from PretrainedModel. Refer to the superclass documentation for the generic methods.

This model is also a paddle.nn.Layer subclass. Use it as a regular Paddle Layer and refer to the Paddle documentation for all matter related to general usage and behavior.

参数

• vocab_size (int) -- Vocabulary size of inputs_ids in UNIMOModel. Also is the vocab size of token embedding matrix. Defines the number of different tokens that can be represented by the inputs_ids passed when calling UNIMOModel.

• hidden_size (int, optional) -- Dimensionality of the embedding layers and encoder layers. Defaults to 768.

• num_hidden_layers (int, optional) -- The number of hidden layers in the Transformer encoder. Defaults to 12.

• num_attention_heads (int, optional) -- Number of attention heads for each attention layer in the Transformer encoder. Defaults to 12.

• intermediate_size (int, optional) -- Dimensionality of the feed-forward (ff) layer in the encoder. Input tensors to ff layers are firstly projected from hidden_size to intermediate_size, and then projected back to hidden_size. Typically intermediate_size is larger than hidden_size. Defaults to 3072.

• hidden_act (str, optional) -- The non-linear activation function in the feed-forward layer. "gelu", "relu" and any other paddle supported activation functions are supported. Defaults to "gelu".

• hidden_dropout_prob (float, optional) -- The dropout probability used in pre-process and post-precess of MHA and FFN sub-layer. Defaults to 0.1.

• attention_probs_dropout_prob (float, optional) -- The dropout probability used in MultiHeadAttention in all encoder layers to drop some attention target. Defaults to 0.1.

• normalize_before (bool, optional) -- Indicate whether to put layer normalization into preprocessing of MHA and FFN sub-layers. If True, pre-process is layer normalization and post-precess includes dropout, residual connection. Otherwise, no pre-process and post-precess includes dropout, residual connection, layer normalization. Defaults to True.

• max_position_embeddings (int, optional) -- The maximum value of the dimensionality of position encoding, which dictates the maximum supported length of an input sequence. Defaults to 512.

• type_vocab_size (int, optional) -- The vocabulary size of the token_type_ids passed when calling UNIMOModel. Defaults to 2.

• initializer_range (float, optional) --

  The standard deviation of the normal initializer. Defaults to 0.02.

  注解

  A normal_initializer initializes weight matrices as normal distributions. See UNIMOPretrainedModel._init_weights() for how weights are initialized in UNIMOModel.

• unk_token_id (int, optional) -- A special token representing the unknown (out-of-vocabulary) token. An unknown token is set to be unk_token in order to be converted to an ID. Defaults to 17963.

• pad_token_id (int, optional) -- A special token used to make arrays of tokens the same size for batching purposes. Defaults to 0.

• bos_token_id (int, optional) -- A special token representing the beginning of a sequence that was used during pretraining. Defaults to 1.

• eos_token_id (int, optional) -- A special token representing the end of a sequence that was used during pretraining. Defaults to 3.

• mask_token_id (int, optional) -- A special token representing a masked token. This is the token used in the masked language modeling task which the model tries to predict the original unmasked ones. Defaults to 3.

get_input_embeddings()

get input embedding of model

返回

embedding of model

返回类型

nn.Embedding

set_input_embeddings(value)

set new input embedding for model

参数

value (Embedding) -- the new embedding of model

引发

NotImplementedError -- Model has not implement set_input_embeddings method

forward(input_ids, token_type_ids=None, position_ids=None, attention_mask=None, use_cache=False, cache=None, output_attentions=False, output_hidden_states=False, return_dict=False)

The UNIMOModel forward method, overrides the special __call__() method.

参数

• input_ids (Tensor) -- Indices of input sequence tokens in the vocabulary. They are numerical representations of tokens that build the input sequence. It's data type should be int64 and has a shape of [batch_size, sequence_length].

• token_type_ids (Tensor) --

  Segment token indices to indicate first and second portions of the inputs. Indices can be either 0 or 1:

  • 0 corresponds to a sentence A token,

  • 1 corresponds to a sentence B token.

  It's data type should be int64 and has a shape of [batch_size, sequence_length]. Defaults to None, which means no segment embeddings is added to token embeddings.

• position_ids (Tensor) -- Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, max_position_embeddings - 1]. It's data type should be int64 and has a shape of [batch_size, sequence_length]. Defaults to None.

• attention_mask (Tensor) -- Mask used in multi-head attention to avoid performing attention to some unwanted positions, usually the paddings or the subsequent positions. Its data type can be int, float and bool. When the data type is bool, the masked tokens have False values and the others have True values. When the data type is int, the masked tokens have 0 values and the others have 1 values. When the data type is float, the masked tokens have -INF values and the others have 0 values. It is a tensor with shape broadcasted to [batch_size, num_attention_heads, sequence_length, sequence_length]. For example, its shape can be [batch_size, sequence_length], [batch_size, sequence_length, sequence_length], [batch_size, num_attention_heads, sequence_length, sequence_length]. Defaults to None, which means nothing needed to be prevented attention to.

• use_cache -- (bool, optional): Whether or not use the model cache to speed up decoding. Defaults to False.

• cache (list, optional) -- It is a list, and each element in the list is incremental_cache produced by paddle.nn.TransformerEncoderLayer.gen_cache() method. See paddle.nn.TransformerEncoder.gen_cache() method for more details. It is only used for inference and should be None for training. Defaults to None.

• output_attentions (bool, optional) -- Whether or not to return the attentions tensors of all attention layers. See attentions under returned tensors for more detail. Defaults to False.

• output_hidden_states (bool, optional) -- Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for more detail. Defaults to False.

• return_dict (bool, optional) -- Whether to return a BaseModelOutputWithPastAndCrossAttentions object. If False, the output will be a tuple of tensors. Defaults to False.

返回

An instance of BaseModelOutputWithPastAndCrossAttentions if return_dict=True. Otherwise it returns a tuple of tensors corresponding to ordered and not None (depending on the input arguments) fields of BaseModelOutputWithPastAndCrossAttentions. Especially, When return_dict=output_hidden_states=output_attentions=False and cache=None, returns tensor Sequence_output of shape [batch_size, sequence_length, hidden_size], which is the output at the last layer of the model.

示例

from paddlenlp.transformers import UNIMOModel
from paddlenlp.transformers import UNIMOTokenizer

model = UNIMOModel.from_pretrained('unimo-text-1.0')
tokenizer = UNIMOTokenizer.from_pretrained('unimo-text-1.0')

inputs = tokenizer.gen_encode("Welcome to use PaddlePaddle and PaddleNLP!", return_tensors=True)
outputs = model(**inputs)

class UNIMOLMHeadModel(unimo)

基类：paddlenlp.transformers.unimo.modeling.UNIMOPretrainedModel

The UNIMO Model with a language modeling head on top designed for generation tasks.

参数

unimo (UNIMOModel) -- An instance of UNIMOModel.

forward(input_ids, token_type_ids=None, position_ids=None, attention_mask=None, masked_positions=None, use_cache=False, cache=None, labels=None, output_attentions=False, output_hidden_states=False, return_dict=False)

The UNIMOLMHeadModel forward method, overrides the special __call__() method.

参数

• input_ids (Tensor) -- See UNIMOModel.

• token_type_ids (Tensor) -- See UNIMOModel.

• position_ids (Tensor) -- See UNIMOModel.

• attention_mask (Tensor) -- See UNIMOModel.

• use_cache -- (bool, optional): See UNIMOModel.

• cache (list, optional) -- See UNIMOModel.

• labels (Tensor, optional) -- Labels for computing the left-to-right language modeling loss. Indices should be in [-100, 0, ..., vocab_size] (see input_ids docstring) Tokens with indices set to -100 are ignored (masked), the loss is only computed for the tokens with labels n [0, ..., vocab_size]

• output_attentions (bool, optional) -- See UNIMOModel.

• output_hidden_states (bool, optional) -- See UNIMOModel.

• return_dict (bool, optional) -- Whether to return a CausalLMOutputWithPastAndCrossAttentions object. If False, the output will be a tuple of tensors. Defaults to False.

返回

An instance of CausalLMOutputWithPastAndCrossAttentions if return_dict=True. Otherwise it returns a tuple of tensors corresponding to ordered and not None (depending on the input arguments) fields of CausalLMOutputWithPastAndCrossAttentions. Especially, When return_dict=output_hidden_states=output_attentions=False and cache=labels=None, returns tensor logits of shape [batch_size, sequence_length, hidden_size], which is the output at the last layer of the model.

示例

from paddlenlp.transformers import UNIMOLMHeadModel
from paddlenlp.transformers import UNIMOTokenizer

model = UNIMOLMHeadModel.from_pretrained('unimo-text-1.0')
tokenizer = UNIMOTokenizer.from_pretrained('unimo-text-1.0')

inputs = tokenizer.gen_encode(
    "Welcome to use PaddlePaddle and PaddleNLP!",
    return_tensors=True,
    is_split_into_words=False)
logits = model(**inputs)

UNIMOForMaskedLM

alias of paddlenlp.transformers.unimo.modeling.UNIMOLMHeadModel

UNIMOForConditionalGeneration

alias of paddlenlp.transformers.unimo.modeling.UNIMOLMHeadModel