modeling¶
-
class
BigBirdModel
(config: paddlenlp.transformers.bigbird.configuration.BigBirdConfig)[source]¶ Bases:
paddlenlp.transformers.bigbird.modeling.BigBirdPretrainedModel
The bare BigBird Model outputting raw hidden-states.
This model inherits from
PretrainedModel
. Refer to the superclass documentation for the generic methods.This model is also a Paddle paddle.nn.Layer subclass. Use it as a regular Paddle Layer and refer to the Paddle documentation for all matter related to general usage and behavior.
- Parameters
num_layers (int) – Number of hidden layers in the Transformer encoder.
vocab_size (int) – Vocabulary size of
inputs_ids
inBigBirdModel
. Also is the vocab size of token embedding matrix. Defines the number of different tokens that can be represented by theinputs_ids
passed when callingBigBirdModel
.nhead (int) – Number of attention heads for each attention layer in the Transformer encoder.
attn_dropout (float, optional) – The dropout probability used in MultiHeadAttention in all encoder layers to drop some attention target. Defaults to
0.1
.dim_feedforward (int, optional) – Dimensionality of the feed-forward (ff) layer in the Transformer encoder. Input tensors to ff layers are firstly projected from
hidden_size
tointermediate_size
, and then projected back tohidden_size
. Typicallyintermediate_size
is larger thanhidden_size
. Defaults to3072
.activation (str, optional) – The non-linear activation function in the feed-forward layer.
"gelu"
,"relu"
,"silu"
and"gelu_new"
are supported. Defaults to"gelu"
.normalize_before (bool, optional) – Indicates whether to put layer normalization into preprocessing of MHA and FFN sub-layers. If True, pre-process is layer normalization and post-precess includes dropout, residual connection. Otherwise, no pre-process and post-precess includes dropout, residual connection, layer normalization. Defaults to
False
.block_size (int, optional) – The block size for the attention mask. Defaults to
1
.window_size (int, optional) – The number of block in a window. Defaults to
3
.num_global_blocks (int, optional) – Number of global blocks per sequence. Defaults to
1
.num_rand_blocks (int, optional) – Number of random blocks per row. Defaults to
2
.seed (int, optional) – The random seed for generating random block id. Defaults to
None
.pad_token_id (int, optional) – The index of padding token for BigBird embedding. Defaults to
0
.hidden_size (int, optional) – Dimensionality of the embedding layer, encoder layer and pooler layer. Defaults to
768
.hidden_dropout_prob (float, optional) – The dropout probability for all fully connected layers in the embeddings and encoder. Defaults to
0.1
.max_position_embeddings (int, optional) – The maximum value of the dimensionality of position encoding, which dictates the maximum supported length of an input sequence. Defaults to
512
.type_vocab_size (int, optional) – The vocabulary size of the
token_type_ids
. Defaults to2
.
-
forward
(input_ids: Optional[paddle.Tensor] = None, token_type_ids: Optional[paddle.Tensor] = None, attention_mask: Optional[paddle.Tensor] = None, rand_mask_idx_list: Optional[List] = None, inputs_embeds: Optional[paddle.Tensor] = None, output_hidden_states: Optional[bool] = None, output_attentions: Optional[bool] = None, return_dict: Optional[bool] = None)[source]¶ The BigBirdModel forward method, overrides the __call__() special method.
- Parameters
input_ids (
Tensor
) – Indices of input sequence tokens in the vocabulary. Its data type should beint64
and it has a shape of [batch_size, sequence_length].token_type_ids (
Tensor
, optional) –Segment token indices to indicate first and second portions of the inputs. Indices can either be 0 or 1:
0 corresponds to a sentence A token,
1 corresponds to a sentence B token.
Its data type should be
int64
and it has a shape of [batch_size, sequence_length]. Defaults toNone
, which means we don’t add segment embeddings.inputs_embeds (Tensor, optional) – If you want to control how to convert
inputs_ids
indices into associated vectors, you can pass an embedded representation directly instead of passinginputs_ids
.attention_mask (Tensor, optional) – Mask used in multi-head attention to avoid performing attention on to some unwanted positions, usually the paddings or the subsequent positions. Its data type can be int, float and bool. When the data type is bool, the
masked
tokens haveFalse
values and the others haveTrue
values. When the data type is int, themasked
tokens have0
values and the others have1
values. When the data type is float, themasked
tokens have-INF
values and the others have0
values. It is a tensor with shape broadcasted to[batch_size, num_attention_heads, sequence_length, sequence_length]
. For example, its shape can be [batch_size, sequence_length], [batch_size, sequence_length, sequence_length], [batch_size, num_attention_heads, sequence_length, sequence_length]. We use whole-word-mask in ERNIE, so the whole word will have the same value. For example, “使用” as a word, “使” and “用” will have the same value. Defaults toNone
, which means nothing needed to be prevented attention to.rand_mask_idx_list (
list
, optional) – A list which contains some tensors used in bigbird random block.return_dict (bool, optional) – Whether to return a
ModelOutput
object. IfFalse
, the output will be a tuple of tensors. Defaults toFalse
.
- Returns
An instance of
BaseModelOutputWithPoolingAndCrossAttentions
ifreturn_dict=True
. Otherwise it returns a tuple of tensors corresponding to ordered and not None (depending on the input arguments) fields ofBaseModelOutputWithPoolingAndCrossAttentions
.
Examples
import paddle from paddlenlp.transformers import BigBirdModel, BigBirdTokenizer from paddlenlp.transformers import create_bigbird_rand_mask_idx_list tokenizer = BigBirdTokenizer.from_pretrained('bigbird-base-uncased') model = BigBirdModel.from_pretrained('bigbird-base-uncased') config = model.config max_seq_len = 512 input_ids = tokenizer.convert_tokens_to_ids( tokenizer( "This is a docudrama story on the Lindy Chamberlain case and a look at " "its impact on Australian society It especially looks at the problem of " "innuendo gossip and expectation when dealing with reallife dramasbr br " "One issue the story deals with is the way it is expected people will all " "give the same emotional response to similar situations Not everyone goes " "into wild melodramatic hysterics to every major crisis Just because the " "characters in the movies and on TV act in a certain way is no reason to " "expect real people to do so" )) input_ids.extend([0] * (max_seq_len - len(input_ids))) seq_len = len(input_ids) input_ids = paddle.to_tensor([input_ids]) rand_mask_idx_list = create_bigbird_rand_mask_idx_list( config["num_layers"], seq_len, seq_len, config["nhead"], config["block_size"], config["window_size"], config["num_global_blocks"], config["num_rand_blocks"], config["seed"]) rand_mask_idx_list = [ paddle.to_tensor(rand_mask_idx) for rand_mask_idx in rand_mask_idx_list ] output = model(input_ids, rand_mask_idx_list=rand_mask_idx_list)
-
get_input_embeddings
()[source]¶ get input embedding of model
- Returns
embedding of model
- Return type
nn.Embedding
-
set_input_embeddings
(value)[source]¶ set new input embedding for model
- Parameters
value (Embedding) – the new embedding of model
- Raises
NotImplementedError – Model has not implement
set_input_embeddings
method
-
class
BigBirdPretrainedModel
(*args, **kwargs)[source]¶ Bases:
paddlenlp.transformers.model_utils.PretrainedModel
An abstract class for pretrained BigBird models. It provides BigBird related
model_config_file
,pretrained_init_configuration
,resource_files_names
,pretrained_resource_files_map
,base_model_prefix
for downloading and loading pretrained models. SeePretrainedModel
for more details.-
config_class
¶ alias of
paddlenlp.transformers.bigbird.configuration.BigBirdConfig
-
base_model_class
¶ alias of
paddlenlp.transformers.bigbird.modeling.BigBirdModel
-
-
class
BigBirdForPretraining
(config: paddlenlp.transformers.bigbird.configuration.BigBirdConfig)[source]¶ Bases:
paddlenlp.transformers.bigbird.modeling.BigBirdPretrainedModel
BigBird Model with pretraining tasks on top.
- Parameters
bigbird (
BigBirdModel
) – An instance ofBigBirdModel
.
-
forward
(input_ids: Optional[paddle.Tensor] = None, token_type_ids: Optional[paddle.Tensor] = None, position_ids: Optional[paddle.Tensor] = None, rand_mask_idx_list: Optional[List] = None, masked_positions: Optional[paddle.Tensor] = None, attention_mask: Optional[paddle.Tensor] = None, rand_mask: Optional[paddle.Tensor] = None, inputs_embeds: Optional[paddle.Tensor] = None, labels: Optional[paddle.Tensor] = None, next_sentence_label: Optional[paddle.Tensor] = None, output_hidden_states: Optional[bool] = None, output_attentions: Optional[bool] = None, return_dict: Optional[bool] = None)[source]¶ The BigBirdForPretraining forward method, overrides the __call__() special method.
- Parameters
input_ids (Tensor) – See
BigBirdModel
.token_type_ids (Tensor) – See
BigBirdModel
.attention_mask (Tensor) – See
BigBirdModel
.rand_mask_idx_list (list) – See
BigBirdModel
.masked_positions (list) – A tensor indicates positions to be masked in the position embedding. Its data type should be int64 and its shape is [batch_size, mask_token_num].
mask_token_num
is the number of masked tokens. It should be no bigger thansequence_length
. Defaults toNone
, which means we output hidden-states of all tokens in masked token prediction.inputs_embeds (Tensor, optional) – See
BigBirdModel
.output_hidden_states (bool, optional) – Whether to return the hidden states of all layers. Defaults to
None
.output_attentions (bool, optional) – Whether to return the attentions tensors of all attention layers. Defaults to
None
.return_dict (bool, optional) – Whether to return a
BertForPreTrainingOutput
object. IfFalse
, the output will be a tuple of tensors. Defaults toNone
.
- Returns
An instance of
BertForPreTrainingOutput
ifreturn_dict=True
. Otherwise it returns a tuple of tensors corresponding to ordered and not None (depending on the input arguments) fields ofBertForPreTrainingOutput
.
Examples
import paddle from paddlenlp.transformers import BigBirdForPretraining, BigBirdTokenizer from paddlenlp.transformers import create_bigbird_rand_mask_idx_list tokenizer = BigBirdTokenizer.from_pretrained('bigbird-base-uncased') model = BigBirdForPretraining.from_pretrained('bigbird-base-uncased') config = model.bigbird.config max_seq_len = 512 input_ids, masked_lm_positions, masked_lm_ids, masked_lm_weights = tokenizer.encode( "This is a docudrama story on the Lindy Chamberlain case and a look at " "its impact on Australian society It especially looks at the problem of " "innuendo gossip and expectation when dealing with reallife dramasbr br " "One issue the story deals with is the way it is expected people will all " "give the same emotional response to similar situations Not everyone goes " "into wild melodramatic hysterics to every major crisis Just because the " "characters in the movies and on TV act in a certain way is no reason to " "expect real people to do so", max_seq_len=max_seq_len) seq_len = len(input_ids) input_ids = paddle.to_tensor([input_ids]) rand_mask_idx_list = create_bigbird_rand_mask_idx_list( config["num_layers"], seq_len, seq_len, config["nhead"], config["block_size"], config["window_size"], config["num_global_blocks"], config["num_rand_blocks"], config["seed"]) rand_mask_idx_list = [ paddle.to_tensor(rand_mask_idx) for rand_mask_idx in rand_mask_idx_list ] output = model(input_ids, rand_mask_idx_list=rand_mask_idx_list) print(output)
-
class
BigBirdPretrainingCriterion
(config: paddlenlp.transformers.bigbird.configuration.BigBirdConfig, use_nsp=False, ignore_index=0)[source]¶ Bases:
paddle.fluid.dygraph.layers.Layer
BigBird Criterion for a pretraining task on top.
- Parameters
vocab_size (int) – See
BigBirdModel
.use_nsp (bool, optional) – It decides whether it considers Next Sentence Prediction loss. Defaults to
False
.ignore_index (int) – Specifies a target value that is ignored and does not contribute to the input gradient. Only valid if
soft_label
is set toFalse
. Defaults to0
.
-
forward
(prediction_scores, seq_relationship_score, masked_lm_labels, next_sentence_labels, masked_lm_scale, masked_lm_weights)[source]¶ The BigBirdPretrainingCriterion forward method, overrides the __call__() special method.
- Parameters
prediction_scores (Tensor) – The scores of masked token prediction. Its data type should be float32. If
masked_positions
is None, its shape is [batch_size, sequence_length, vocab_size]. Otherwise, its shape is [batch_size, mask_token_num, vocab_size].seq_relationship_score (Tensor) – The scores of next sentence prediction. Its data type should be float32 and its shape is [batch_size, 2].
masked_lm_labels (Tensor) – The labels of the masked language modeling, its dimensionality is equal to
prediction_scores
. Its data type should be int64. Ifmasked_positions
is None, its shape is [batch_size, sequence_length, 1]. Otherwise, its shape is [batch_size, mask_token_num, 1].next_sentence_labels (Tensor) – The labels of the next sentence prediction task, the dimensionality of
next_sentence_labels
is equal toseq_relation_labels
. Its data type should be int64 and its shape is [batch_size, 1].masked_lm_scale (Tensor or int) – The scale of masked tokens. Used for the normalization of masked language modeling loss. If it is a
Tensor
, its data type should be int64 and its shape is equal toprediction_scores
.masked_lm_weights (Tensor) – The weight of masked tokens. Its data type should be float32 and its shape is [mask_token_num, 1].
- Returns
The pretraining loss, equals to the sum of
masked_lm_loss
plus the mean ofnext_sentence_loss
. Its data type should be float32 and its shape is [1].- Return type
Tensor
Example
import numpy as np import paddle from paddlenlp.transformers import BigBirdForPretraining, BigBirdTokenizer, BigBirdPretrainingCriterion from paddlenlp.transformers import create_bigbird_rand_mask_idx_list tokenizer = BigBirdTokenizer.from_pretrained('bigbird-base-uncased') model = BigBirdForPretraining.from_pretrained('bigbird-base-uncased') config = model.bigbird.config criterion = BigBirdPretrainingCriterion(config["vocab_size"], False) max_seq_len = 512 max_pred_length=75 input_ids, masked_lm_positions, masked_lm_ids, masked_lm_weights = tokenizer.encode( "This is a docudrama story on the Lindy Chamberlain case and a look at " "its impact on Australian society It especially looks at the problem of " "innuendo gossip and expectation when dealing with reallife dramasbr br " "One issue the story deals with is the way it is expected people will all " "give the same emotional response to similar situations Not everyone goes " "into wild melodramatic hysterics to every major crisis Just because the " "characters in the movies and on TV act in a certain way is no reason to " "expect real people to do so", max_seq_len=max_seq_len, max_pred_len=max_pred_length) seq_len = len(input_ids) masked_lm_positions_tmp = np.full(seq_len, 0, dtype=np.int32) masked_lm_ids_tmp = np.full([seq_len, 1], -1, dtype=np.int64) masked_lm_weights_tmp = np.full([seq_len], 0, dtype="float32") mask_token_num = 0 for i, x in enumerate([input_ids]): for j, pos in enumerate(masked_lm_positions): masked_lm_positions_tmp[mask_token_num] = i * seq_len + pos masked_lm_ids_tmp[mask_token_num] = masked_lm_ids[j] masked_lm_weights_tmp[mask_token_num] = masked_lm_weights[j] masked_lm_positions = masked_lm_positions_tmp masked_lm_ids = masked_lm_ids_tmp masked_lm_weights = masked_lm_weights_tmp print(masked_lm_ids.shape) input_ids = paddle.to_tensor([input_ids]) masked_lm_positions = paddle.to_tensor(masked_lm_positions) masked_lm_ids = paddle.to_tensor(masked_lm_ids, dtype='int64') masked_lm_weights = paddle.to_tensor(masked_lm_weights) masked_lm_scale = 1.0 next_sentence_labels = paddle.zeros(shape=(1, 1), dtype='int64') rand_mask_idx_list = create_bigbird_rand_mask_idx_list( config["num_layers"], seq_len, seq_len, config["nhead"], config["block_size"], config["window_size"], config["num_global_blocks"], config["num_rand_blocks"], config["seed"]) rand_mask_idx_list = [ paddle.to_tensor(rand_mask_idx) for rand_mask_idx in rand_mask_idx_list ] prediction_scores, seq_relationship_score = model(input_ids, rand_mask_idx_list=rand_mask_idx_list, masked_positions=masked_lm_positions) loss = criterion(prediction_scores, seq_relationship_score, masked_lm_ids, next_sentence_labels, masked_lm_scale, masked_lm_weights) print(loss)
-
class
BigBirdForSequenceClassification
(config: paddlenlp.transformers.bigbird.configuration.BigBirdConfig)[source]¶ Bases:
paddlenlp.transformers.bigbird.modeling.BigBirdPretrainedModel
BigBird Model with a linear layer on top of the output layer, designed for sequence classification/regression tasks like GLUE tasks.
- Parameters
bigbird (
BigBirdModel
) – An instance ofBigBirdModel
.num_classes (int, optional) – The number of classes. Defaults to
None
.
-
forward
(input_ids: Optional[paddle.Tensor] = None, token_type_ids: Optional[paddle.Tensor] = None, attention_mask: Optional[paddle.Tensor] = None, rand_mask_idx_list: Optional[List] = None, inputs_embeds: Optional[paddle.Tensor] = None, labels: Optional[paddle.Tensor] = None, output_hidden_states: Optional[bool] = None, output_attentions: Optional[bool] = None, return_dict: Optional[bool] = None)[source]¶ The BigBirdForSequenceClassification forward method, overrides the __call__() special method.
- Parameters
input_ids (Tensor) – See
BigBirdModel
.token_type_ids (Tensor) – See
BigBirdModel
.attention_mask (Tensor) – See
BigBirdModel
.rand_mask_idx_list (list) – See
BigBirdModel
.inputs_embeds (Tensor, optional) – See
BigBirdModel
.labels (Tensor of shape
(batch_size,)
, optional) – Labels for computing the sequence classification/regression loss. Indices should be in[0, ..., num_labels - 1]
. Ifnum_labels == 1
a regression loss is computed (Mean-Square loss), Ifnum_labels > 1
a classification loss is computed (Cross-Entropy).output_hidden_states (bool, optional) – Whether to return the hidden states of all layers. Defaults to
None
.output_attentions (bool, optional) – Whether to return the attentions tensors of all attention layers. Defaults to
None
.return_dict (bool, optional) – Whether to return a
SequenceClassifierOutput
object. IfFalse
, the output will be a tuple of tensors. Defaults toNone
.
- Returns
Returns tensor
output
, a tensor of the input text classification logits. Its data type should be float32 and it has a shape of [batch_size, num_classes].- Return type
Tensor
Examples
import paddle from paddlenlp.transformers import BigBirdForSequenceClassification, BigBirdTokenizer from paddlenlp.transformers import create_bigbird_rand_mask_idx_list tokenizer = BigBirdTokenizer.from_pretrained('bigbird-base-uncased') model = BigBirdForSequenceClassification.from_pretrained('bigbird-base-uncased') config = model.bigbird.config max_seq_len = 512 input_ids = tokenizer.convert_tokens_to_ids( tokenizer( "This is a docudrama story on the Lindy Chamberlain case and a look at " "its impact on Australian society It especially looks at the problem of " "innuendo gossip and expectation when dealing with reallife dramasbr br " "One issue the story deals with is the way it is expected people will all " "give the same emotional response to similar situations Not everyone goes " "into wild melodramatic hysterics to every major crisis Just because the " "characters in the movies and on TV act in a certain way is no reason to " "expect real people to do so" )) input_ids.extend([0] * (max_seq_len - len(input_ids))) seq_len = len(input_ids) input_ids = paddle.to_tensor([input_ids]) rand_mask_idx_list = create_bigbird_rand_mask_idx_list( config["num_layers"], seq_len, seq_len, config["nhead"], config["block_size"], config["window_size"], config["num_global_blocks"], config["num_rand_blocks"], config["seed"]) rand_mask_idx_list = [ paddle.to_tensor(rand_mask_idx) for rand_mask_idx in rand_mask_idx_list ] output = model(input_ids, rand_mask_idx_list=rand_mask_idx_list) print(output)
-
class
BigBirdPretrainingHeads
(config: paddlenlp.transformers.bigbird.configuration.BigBirdConfig)[source]¶ Bases:
paddle.fluid.dygraph.layers.Layer
The BigBird pretraining heads for a pretraining task.
- Parameters
hidden_size (int) – See
BigBirdModel
.vocab_size (int) – See
BigBirdModel
.activation (str) – See
BigBirdModel
.embedding_weights (Tensor, optional) – The weight of pretraining embedding layer. Its data type should be float32 and its shape is [hidden_size, vocab_size]. If set to
None
, use normal distribution to initialize weight. Defaults toNone
.
-
forward
(sequence_output, pooled_output, masked_positions=None)[source]¶ The BigBirdPretrainingHeads forward method, overrides the __call__() special method.
- Parameters
sequence_output (Tensor) – The sequence output of BigBirdModel. Its data type should be float32 and has a shape of [batch_size, sequence_length, hidden_size].
pooled_output (Tensor) – The pooled output of BigBirdModel. Its data type should be float32 and has a shape of [batch_size, hidden_size].
masked_positions (Tensor) – A tensor indicates positions to be masked in the position embedding. Its data type should be int64 and its shape is [batch_size, mask_token_num].
mask_token_num
is the number of masked tokens. It should be no bigger thansequence_length
. Defaults toNone
, which means we output hidden-states of all tokens in masked token prediction.
- Returns
(
prediction_scores
,seq_relationship_score
).With the fields:
- prediction_scores (Tensor):
The scores of masked token prediction. Its data type should be float32. If
masked_positions
is None, its shape is [batch_size, sequence_length, vocab_size]. Otherwise, its shape is [batch_size, mask_token_num, vocab_size].
- seq_relationship_score (Tensor):
The logits whether 2 sequences are NSP relationship. Its data type should be float32 and has a shape of [batch_size, 2].
- Return type
tuple
-
class
BigBirdForQuestionAnswering
(config: paddlenlp.transformers.bigbird.configuration.BigBirdConfig)[source]¶ Bases:
paddlenlp.transformers.bigbird.modeling.BigBirdPretrainedModel
BigBird Model with a linear layer on top of the hidden-states output to compute
span_start_logits
andspan_end_logits
, designed for question-answering tasks like SQuAD.- Parameters
bigbird (
BigBirdModel
) – An instance of BigBirdModel.dropout (float, optional) – The dropout probability for output of BigBirdModel. If None, use the same value as
hidden_dropout_prob
ofBigBirdModel
instancebigbird
. Defaults toNone
.
-
forward
(input_ids: Optional[paddle.Tensor] = None, token_type_ids: Optional[paddle.Tensor] = None, attention_mask: Optional[paddle.Tensor] = None, start_positions: Optional[paddle.Tensor] = None, end_positions: Optional[paddle.Tensor] = None, rand_mask_idx_list: Optional[List] = None, inputs_embeds: Optional[paddle.Tensor] = None, output_hidden_states: Optional[bool] = None, output_attentions: Optional[bool] = None, return_dict: Optional[bool] = None)[source]¶ The BigBirdForQuestionAnswering forward method, overrides the __call__() special method.
- Parameters
input_ids (Tensor) – See
BigBirdModel
.token_type_ids (Tensor, optional) – See
BigBirdModel
.attention_mask (Tensor) – See
BigBirdModel
.rand_mask_idx_list (
List
) – SeeBigBirdModel
.inputs_embeds (Tensor, optional) – See
BigBirdModel
.output_hidden_states (bool, optional) – Whether to return the hidden states of all layers. Defaults to
None
.output_attentions (bool, optional) – Whether to return the attentions tensors of all attention layers. Defaults to
None
.return_dict (bool, optional) – Whether to return a
QuestionAnsweringModelOutput
object. IfFalse
, the output will be a tuple of tensors. Defaults toNone
.
- Returns
An instance of
QuestionAnsweringModelOutput
ifreturn_dict=True
. Otherwise it returns a tuple of tensors corresponding to ordered and not None (depending on the input arguments) fields ofQuestionAnsweringModelOutput
.
Example
import paddle from paddlenlp.transformers.bigbird.modeling import BigBirdForQuestionAnswering from paddlenlp.transformers.bigbird.tokenizer import BigBirdTokenizer tokenizer = BigBirdTokenizer.from_pretrained('bigbird-base-uncased') model = BigBirdForQuestionAnswering.from_pretrained('bigbird-base-uncased') inputs = tokenizer("Welcome to use PaddlePaddle and PaddleNLP!", return_tensors='pd') inputs = {k:paddle.to_tensor([v]) for (k, v) in inputs.items()} outputs = model(**inputs) start_logits = outputs[0] end_logits =outputs[1]
-
class
BigBirdForTokenClassification
(config: paddlenlp.transformers.bigbird.configuration.BigBirdConfig)[source]¶ Bases:
paddlenlp.transformers.bigbird.modeling.BigBirdPretrainedModel
BigBird Model with a linear layer on top of the hidden-states output layer, designed for token classification tasks like NER tasks.
- Parameters
bigbird (
BigBirdModel
) – An instance of BigBirdModel.num_classes (int, optional) – The number of classes. Defaults to
2
.dropout (float, optional) – The dropout probability for output of BIGBIRD. If None, use the same value as
hidden_dropout_prob
ofBigBirdModel
instancebigbird
. Defaults to None.
-
forward
(input_ids: Optional[paddle.Tensor] = None, token_type_ids: Optional[paddle.Tensor] = None, attention_mask: Optional[paddle.Tensor] = None, rand_mask_idx_list: Optional[List] = None, inputs_embeds: Optional[paddle.Tensor] = None, labels: Optional[paddle.Tensor] = None, output_hidden_states: Optional[bool] = None, output_attentions: Optional[bool] = None, return_dict: Optional[bool] = None)[source]¶ The BigBirdForSequenceClassification forward method, overrides the __call__() special method.
- Parameters
input_ids (Tensor) – See
BigBirdModel
.token_type_ids (Tensor, optional) – See
BigBirdModel
.attention_mask (Tensor) – See
BigBirdModel
.rand_mask_idx_list (
List
) – SeeBigBirdModel
.labels (Tensor of shape
(batch_size, sequence_length)
, optional) – Labels for computing the token classification loss. Indices should be in[0, ..., num_labels - 1]
.inputs_embeds (Tensor, optional) – See
BigBirdModel
.output_hidden_states (bool, optional) – Whether to return the hidden states of all layers. Defaults to
None
.output_attentions (bool, optional) – Whether to return the attentions tensors of all attention layers. Defaults to
None
.return_dict (bool, optional) – Whether to return a
TokenClassifierOutput
object. If
- Returns
An instance of
TokenClassifierOutput
ifreturn_dict=True
. Otherwise it returns a tuple of tensors corresponding to ordered and not None (depending on the input arguments) fields ofTokenClassifierOutput
.
Example
import paddle from paddlenlp.transformers.bigbird.modeling import BigBirdForTokenClassification from paddlenlp.transformers.bigbird.tokenizer import BigBirdTokenizer tokenizer = BigBirdTokenizer.from_pretrained('bigbird-base-uncased') model = BigBirdForTokenClassification.from_pretrained('bigbird-base-uncased') inputs = tokenizer("Welcome to use PaddlePaddle and PaddleNLP!", return_tensors='pd') inputs = {k:paddle.to_tensor([v]) for (k, v) in inputs.items()} outputs = model(**inputs) logits = outputs
-
class
BigBirdForMultipleChoice
(config: paddlenlp.transformers.bigbird.configuration.BigBirdConfig)[source]¶ Bases:
paddlenlp.transformers.bigbird.modeling.BigBirdPretrainedModel
BigBird Model with a linear layer on top of the hidden-states output layer, designed for multiple choice tasks like RocStories/SWAG tasks .
- Parameters
bigbird (
BigBirdModel
) – An instance of BigBirdModel.num_choices (int, optional) – The number of choices. Defaults to
2
.dropout (float, optional) – The dropout probability for output of BIGBIRD. If None, use the same value as
hidden_dropout_prob
ofBigBirdModel
instancebigbird
. Defaults to None.
-
forward
(input_ids: Optional[paddle.Tensor] = None, attention_mask: Optional[paddle.Tensor] = None, rand_mask_idx_list: Optional[List] = None, token_type_ids: Optional[paddle.Tensor] = None, labels: Optional[paddle.Tensor] = None, inputs_embeds: Optional[paddle.Tensor] = None, output_hidden_states: Optional[bool] = None, output_attentions: Optional[bool] = None, return_dict: Optional[bool] = None)[source]¶ The BigBirdForMultipleChoice forward method, overrides the __call__() special method.
- Parameters
input_ids (Tensor) – See
BigBirdModel
and shape as [batch_size, num_choice, sequence_length].attention_mask (Tensor) – See
BigBirdModel
and shape as [batch_size, num_choice, n_head, sequence_length, sequence_length].rand_mask_idx_list (
List
) – SeeBigBirdModel
.labels (Tensor of shape
(batch_size, )
, optional) – Labels for computing the multiple choice classification loss. Indices should be in[0, ..., num_choices-1]
wherenum_choices
is the size of the second dimension of the input tensors. (Seeinput_ids
above)inputs_embeds (Tensor, optional) – See
BigBirdModel
.output_hidden_states (bool, optional) – Whether to return the hidden states of all layers. Defaults to
None
.output_attentions (bool, optional) – Whether to return the attentions tensors of all attention layers. Defaults to
None
.return_dict (bool, optional) – Whether to return a
MultipleChoiceModelOutput
object. IfFalse
, the output will be a tuple of tensors. Defaults toNone
.
- Returns
An instance of
MultipleChoiceModelOutput
ifreturn_dict=True
. Otherwise it returns a tuple of tensors corresponding to ordered and not None (depending on the input arguments) fields ofMultipleChoiceModelOutput
.
Example
import paddle from paddlenlp.transformers.bigbird.modeling import BigBirdForMultipleChoice from paddlenlp.transformers.bigbird.tokenizer import BigBirdTokenizer tokenizer = BigBirdTokenizer.from_pretrained('bigbird-base-uncased') model = BigBirdForTokenClassification.from_pretrained('bigbird-base-uncased') inputs = tokenizer("Welcome to use PaddlePaddle and PaddleNLP!", return_tensors='pd') inputs = {k:paddle.to_tensor([v]) for (k, v) in inputs.items()} outputs = model(**inputs) logits = outputs
-
class
BigBirdForMaskedLM
(config: paddlenlp.transformers.bigbird.configuration.BigBirdConfig)[source]¶ Bases:
paddlenlp.transformers.bigbird.modeling.BigBirdPretrainedModel
BigBird Model with a
language modeling
head on top.- Parameters
BigBird (
BigBirdModel
) – An instance ofBigBirdModel
.
-
forward
(input_ids: Optional[paddle.Tensor] = None, attention_mask: Optional[paddle.Tensor] = None, rand_mask_idx_list: Optional[List] = None, inputs_embeds: Optional[paddle.Tensor] = None, labels: Optional[paddle.Tensor] = None, output_hidden_states: Optional[bool] = None, output_attentions: Optional[bool] = None, return_dict: Optional[bool] = None)[source]¶ - Parameters
input_ids (Tensor) – See
BigBirdModel
.attention_mask (Tensor) – See
BigBirdModel
.rand_mask_idx_list (
List
) – SeeBigBirdModel
.inputs_embeds (Tensor, optional) – See
BigBirdModel
.labels (Tensor, optional) – The Labels for computing the masked language modeling loss. Indices should be in
[-100, 0, ..., vocab_size]
Tokens with indices set to-100
are ignored (masked), the loss is only computed for the tokens with labels in[0, ..., vocab_size]
Its shape is [batch_size, sequence_length].output_hidden_states (bool, optional) – Whether to return the hidden states of all layers. Defaults to
None
.output_attentions (bool, optional) – Whether to return the attentions tensors of all attention layers. Defaults to
None
.return_dict (bool, optional) – Whether to return a
MaskedLMOutput
object. IfFalse
, the output will be a tuple of tensors. Defaults toNone
.
- Returns
Returns tuple (
masked_lm_loss
,prediction_scores
, ``sequence_output`).With the fields:
masked_lm_loss
(Tensor):The masked lm loss. Its data type should be float32 and its shape is [1].
prediction_scores
(Tensor):The scores of masked token prediction. Its data type should be float32. Its shape is [batch_size, sequence_length, vocab_size].
sequence_output
(Tensor):Sequence of hidden-states at the last layer of the model. Its data type should be float32. Its shape is
[batch_size, sequence_length, hidden_size]
.
- Return type
tuple
-
class
BigBirdForCausalLM
(config: paddlenlp.transformers.bigbird.configuration.BigBirdConfig)[source]¶ Bases:
paddlenlp.transformers.bigbird.modeling.BigBirdPretrainedModel
BigBird Model for casual language model tasks.
- Parameters
BigBird (
BigBirdModel
) – An instance ofBigBirdModel
.
-
forward
(input_ids: Optional[paddle.Tensor] = None, attention_mask: Optional[paddle.Tensor] = None, rand_mask_idx_list: Optional[List] = None, inputs_embeds: Optional[paddle.Tensor] = None, labels: Optional[paddle.Tensor] = None, output_hidden_states: Optional[bool] = None, output_attentions: Optional[bool] = None, return_dict: Optional[bool] = None)[source]¶ - Parameters
input_ids (Tensor) – See
BigBirdModel
.attention_mask (Tensor) – See
BigBirdModel
.rand_mask_idx_list (
List
) – SeeBigBirdModel
.inputs_embeds (Tensor, optional) – See
BigBirdModel
.labels (Tensor, optional) – The Labels for computing the masked language modeling loss. Indices should be in
[-100, 0, ..., vocab_size]
Tokens with indices set to-100
are ignored (masked), the loss is only computed for the tokens with labels in[0, ..., vocab_size]
Its shape is [batch_size, sequence_length].output_hidden_states (bool, optional) – Whether to return the hidden states of all layers. Defaults to
False
.output_attentions (bool, optional) – Whether to return the attentions tensors of all attention layers. Defaults to
False
.return_dict (bool, optional) – Whether to return a
TokenClassifierOutput
object. IfFalse
, the output will be a tuple of tensors. Defaults toFalse
.
- Returns
Returns tuple (
masked_lm_loss
,prediction_scores
, ``sequence_output`).With the fields:
masked_lm_loss
(Tensor):The masked lm loss. Its data type should be float32 and its shape is [1].
prediction_scores
(Tensor):The scores of masked token prediction. Its data type should be float32. Its shape is [batch_size, sequence_length, vocab_size].
sequence_output
(Tensor):Sequence of hidden-states at the last layer of the model. Its data type should be float32. Its shape is
[batch_size, sequence_length, hidden_size]
.
- Return type
tuple