modeling¶

class BigBirdModel(config: paddlenlp.transformers.bigbird.configuration.BigBirdConfig)[source]¶

Bases: paddlenlp.transformers.bigbird.modeling.BigBirdPretrainedModel

The bare BigBird Model outputting raw hidden-states.

This model inherits from PretrainedModel. Refer to the superclass documentation for the generic methods.

This model is also a Paddle paddle.nn.Layer subclass. Use it as a regular Paddle Layer and refer to the Paddle documentation for all matter related to general usage and behavior.

Parameters

num_layers (int) – Number of hidden layers in the Transformer encoder.
vocab_size (int) – Vocabulary size of inputs_ids in BigBirdModel. Also is the vocab size of token embedding matrix. Defines the number of different tokens that can be represented by the inputs_ids passed when calling BigBirdModel.
nhead (int) – Number of attention heads for each attention layer in the Transformer encoder.
attn_dropout (float, optional) – The dropout probability used in MultiHeadAttention in all encoder layers to drop some attention target. Defaults to 0.1.
dim_feedforward (int, optional) – Dimensionality of the feed-forward (ff) layer in the Transformer encoder. Input tensors to ff layers are firstly projected from hidden_size to intermediate_size, and then projected back to hidden_size. Typically intermediate_size is larger than hidden_size. Defaults to 3072.
activation (str, optional) – The non-linear activation function in the feed-forward layer. "gelu", "relu", "silu" and "gelu_new" are supported. Defaults to "gelu".
normalize_before (bool, optional) – Indicates whether to put layer normalization into preprocessing of MHA and FFN sub-layers. If True, pre-process is layer normalization and post-precess includes dropout, residual connection. Otherwise, no pre-process and post-precess includes dropout, residual connection, layer normalization. Defaults to False.
block_size (int, optional) – The block size for the attention mask. Defaults to 1.
window_size (int, optional) – The number of block in a window. Defaults to 3.
num_global_blocks (int, optional) – Number of global blocks per sequence. Defaults to 1.
num_rand_blocks (int, optional) – Number of random blocks per row. Defaults to 2.
seed (int, optional) – The random seed for generating random block id. Defaults to None.
pad_token_id (int, optional) – The index of padding token for BigBird embedding. Defaults to 0.
hidden_size (int, optional) – Dimensionality of the embedding layer, encoder layer and pooler layer. Defaults to 768.
hidden_dropout_prob (float, optional) – The dropout probability for all fully connected layers in the embeddings and encoder. Defaults to 0.1.
max_position_embeddings (int, optional) – The maximum value of the dimensionality of position encoding, which dictates the maximum supported length of an input sequence. Defaults to 512.
type_vocab_size (int, optional) – The vocabulary size of the token_type_ids. Defaults to 2.

forward(input_ids: Optional[paddle.Tensor] = None, token_type_ids: Optional[paddle.Tensor] = None, attention_mask: Optional[paddle.Tensor] = None, rand_mask_idx_list: Optional[List] = None, inputs_embeds: Optional[paddle.Tensor] = None, output_hidden_states: Optional[bool] = None, output_attentions: Optional[bool] = None, return_dict: Optional[bool] = None)[source]¶

The BigBirdModel forward method, overrides the __call__() special method.

Parameters

input_ids (Tensor) – Indices of input sequence tokens in the vocabulary. Its data type should be int64 and it has a shape of [batch_size, sequence_length].
token_type_ids (Tensor, optional) –
Segment token indices to indicate first and second portions of the inputs. Indices can either be 0 or 1:
- 0 corresponds to a sentence A token,
- 1 corresponds to a sentence B token.
Its data type should be int64 and it has a shape of [batch_size, sequence_length]. Defaults to None, which means we don’t add segment embeddings.
inputs_embeds (Tensor, optional) – If you want to control how to convert inputs_ids indices into associated vectors, you can pass an embedded representation directly instead of passing inputs_ids.
attention_mask (Tensor, optional) – Mask used in multi-head attention to avoid performing attention on to some unwanted positions, usually the paddings or the subsequent positions. Its data type can be int, float and bool. When the data type is bool, the masked tokens have False values and the others have True values. When the data type is int, the masked tokens have 0 values and the others have 1 values. When the data type is float, the masked tokens have -INF values and the others have 0 values. It is a tensor with shape broadcasted to [batch_size, num_attention_heads, sequence_length, sequence_length]. For example, its shape can be [batch_size, sequence_length], [batch_size, sequence_length, sequence_length], [batch_size, num_attention_heads, sequence_length, sequence_length]. We use whole-word-mask in ERNIE, so the whole word will have the same value. For example, “使用” as a word, “使” and “用” will have the same value. Defaults to None, which means nothing needed to be prevented attention to.
rand_mask_idx_list (list, optional) – A list which contains some tensors used in bigbird random block.
return_dict (bool, optional) – Whether to return a ModelOutput object. If False, the output will be a tuple of tensors. Defaults to False.

Returns

An instance of BaseModelOutputWithPoolingAndCrossAttentions if return_dict=True. Otherwise it returns a tuple of tensors corresponding to ordered and not None (depending on the input arguments) fields of BaseModelOutputWithPoolingAndCrossAttentions.

Examples

import paddle
from paddlenlp.transformers import BigBirdModel, BigBirdTokenizer
from paddlenlp.transformers import create_bigbird_rand_mask_idx_list

tokenizer = BigBirdTokenizer.from_pretrained('bigbird-base-uncased')
model = BigBirdModel.from_pretrained('bigbird-base-uncased')
config = model.config
max_seq_len = 512
input_ids = tokenizer.convert_tokens_to_ids(
    tokenizer(
        "This is a docudrama story on the Lindy Chamberlain case and a look at "
        "its impact on Australian society It especially looks at the problem of "
        "innuendo gossip and expectation when dealing with reallife dramasbr br "
        "One issue the story deals with is the way it is expected people will all "
        "give the same emotional response to similar situations Not everyone goes "
        "into wild melodramatic hysterics to every major crisis Just because the "
        "characters in the movies and on TV act in a certain way is no reason to "
        "expect real people to do so"
    ))
input_ids.extend([0] * (max_seq_len - len(input_ids)))
seq_len = len(input_ids)
input_ids = paddle.to_tensor([input_ids])
rand_mask_idx_list = create_bigbird_rand_mask_idx_list(
    config["num_layers"], seq_len, seq_len, config["nhead"],
    config["block_size"], config["window_size"], config["num_global_blocks"],
    config["num_rand_blocks"], config["seed"])
rand_mask_idx_list = [
    paddle.to_tensor(rand_mask_idx) for rand_mask_idx in rand_mask_idx_list
]
output = model(input_ids, rand_mask_idx_list=rand_mask_idx_list)

get_input_embeddings()[source]¶

get input embedding of model

Returns: embedding of model
Return type: nn.Embedding

set_input_embeddings(value)[source]¶

set new input embedding for model

Parameters: value (Embedding) – the new embedding of model
Raises: NotImplementedError – Model has not implement set_input_embeddings method

class BigBirdPretrainedModel(*args, **kwargs)[source]¶

Bases: paddlenlp.transformers.model_utils.PretrainedModel

An abstract class for pretrained BigBird models. It provides BigBird related model_config_file, pretrained_init_configuration, resource_files_names, pretrained_resource_files_map, base_model_prefix for downloading and loading pretrained models. See PretrainedModel for more details.

config_class¶: alias of paddlenlp.transformers.bigbird.configuration.BigBirdConfig

base_model_class¶: alias of paddlenlp.transformers.bigbird.modeling.BigBirdModel

class BigBirdForPretraining(config: paddlenlp.transformers.bigbird.configuration.BigBirdConfig)[source]¶

Bases: paddlenlp.transformers.bigbird.modeling.BigBirdPretrainedModel

BigBird Model with pretraining tasks on top.

Parameters: bigbird (BigBirdModel) – An instance of BigBirdModel.

forward(input_ids: Optional[paddle.Tensor] = None, token_type_ids: Optional[paddle.Tensor] = None, position_ids: Optional[paddle.Tensor] = None, rand_mask_idx_list: Optional[List] = None, masked_positions: Optional[paddle.Tensor] = None, attention_mask: Optional[paddle.Tensor] = None, rand_mask: Optional[paddle.Tensor] = None, inputs_embeds: Optional[paddle.Tensor] = None, labels: Optional[paddle.Tensor] = None, next_sentence_label: Optional[paddle.Tensor] = None, output_hidden_states: Optional[bool] = None, output_attentions: Optional[bool] = None, return_dict: Optional[bool] = None)[source]¶

The BigBirdForPretraining forward method, overrides the __call__() special method.

Parameters

input_ids (Tensor) – See BigBirdModel.
token_type_ids (Tensor) – See BigBirdModel.
attention_mask (Tensor) – See BigBirdModel.
rand_mask_idx_list (list) – See BigBirdModel.
masked_positions (list) – A tensor indicates positions to be masked in the position embedding. Its data type should be int64 and its shape is [batch_size, mask_token_num]. mask_token_num is the number of masked tokens. It should be no bigger than sequence_length. Defaults to None, which means we output hidden-states of all tokens in masked token prediction.
inputs_embeds (Tensor, optional) – See BigBirdModel.
output_hidden_states (bool, optional) – Whether to return the hidden states of all layers. Defaults to None.
output_attentions (bool, optional) – Whether to return the attentions tensors of all attention layers. Defaults to None.
return_dict (bool, optional) – Whether to return a BertForPreTrainingOutput object. If False, the output will be a tuple of tensors. Defaults to None.

Returns

An instance of BertForPreTrainingOutput if return_dict=True. Otherwise it returns a tuple of tensors corresponding to ordered and not None (depending on the input arguments) fields of BertForPreTrainingOutput.

Examples

import paddle
from paddlenlp.transformers import BigBirdForPretraining, BigBirdTokenizer
from paddlenlp.transformers import create_bigbird_rand_mask_idx_list

tokenizer = BigBirdTokenizer.from_pretrained('bigbird-base-uncased')
model = BigBirdForPretraining.from_pretrained('bigbird-base-uncased')
config = model.bigbird.config
max_seq_len = 512
input_ids, masked_lm_positions, masked_lm_ids, masked_lm_weights = tokenizer.encode(
        "This is a docudrama story on the Lindy Chamberlain case and a look at "
        "its impact on Australian society It especially looks at the problem of "
        "innuendo gossip and expectation when dealing with reallife dramasbr br "
        "One issue the story deals with is the way it is expected people will all "
        "give the same emotional response to similar situations Not everyone goes "
        "into wild melodramatic hysterics to every major crisis Just because the "
        "characters in the movies and on TV act in a certain way is no reason to "
        "expect real people to do so", max_seq_len=max_seq_len)

seq_len = len(input_ids)
input_ids = paddle.to_tensor([input_ids])
rand_mask_idx_list = create_bigbird_rand_mask_idx_list(
    config["num_layers"], seq_len, seq_len, config["nhead"],
    config["block_size"], config["window_size"], config["num_global_blocks"],
    config["num_rand_blocks"], config["seed"])
rand_mask_idx_list = [
    paddle.to_tensor(rand_mask_idx) for rand_mask_idx in rand_mask_idx_list
]
output = model(input_ids, rand_mask_idx_list=rand_mask_idx_list)
print(output)

class BigBirdPretrainingCriterion(config: paddlenlp.transformers.bigbird.configuration.BigBirdConfig, use_nsp=False, ignore_index=0)[source]¶

Bases: paddle.fluid.dygraph.layers.Layer

BigBird Criterion for a pretraining task on top.

Parameters

vocab_size (int) – See BigBirdModel.
use_nsp (bool, optional) – It decides whether it considers Next Sentence Prediction loss. Defaults to False.
ignore_index (int) – Specifies a target value that is ignored and does not contribute to the input gradient. Only valid if soft_label is set to False. Defaults to 0.

forward(prediction_scores, seq_relationship_score, masked_lm_labels, next_sentence_labels, masked_lm_scale, masked_lm_weights)[source]¶

The BigBirdPretrainingCriterion forward method, overrides the __call__() special method.

Parameters

prediction_scores (Tensor) – The scores of masked token prediction. Its data type should be float32. If masked_positions is None, its shape is [batch_size, sequence_length, vocab_size]. Otherwise, its shape is [batch_size, mask_token_num, vocab_size].
seq_relationship_score (Tensor) – The scores of next sentence prediction. Its data type should be float32 and its shape is [batch_size, 2].
masked_lm_labels (Tensor) – The labels of the masked language modeling, its dimensionality is equal to prediction_scores. Its data type should be int64. If masked_positions is None, its shape is [batch_size, sequence_length, 1]. Otherwise, its shape is [batch_size, mask_token_num, 1].
next_sentence_labels (Tensor) – The labels of the next sentence prediction task, the dimensionality of next_sentence_labels is equal to seq_relation_labels. Its data type should be int64 and its shape is [batch_size, 1].
masked_lm_scale (Tensor or int) – The scale of masked tokens. Used for the normalization of masked language modeling loss. If it is a Tensor, its data type should be int64 and its shape is equal to prediction_scores.
masked_lm_weights (Tensor) – The weight of masked tokens. Its data type should be float32 and its shape is [mask_token_num, 1].

Returns

The pretraining loss, equals to the sum of masked_lm_loss plus the mean of next_sentence_loss. Its data type should be float32 and its shape is [1].

Return type

Tensor

Example

import numpy as np
import paddle
from paddlenlp.transformers import BigBirdForPretraining, BigBirdTokenizer, BigBirdPretrainingCriterion
from paddlenlp.transformers import create_bigbird_rand_mask_idx_list

tokenizer = BigBirdTokenizer.from_pretrained('bigbird-base-uncased')
model = BigBirdForPretraining.from_pretrained('bigbird-base-uncased')
config = model.bigbird.config
criterion = BigBirdPretrainingCriterion(config["vocab_size"], False)
max_seq_len = 512
max_pred_length=75
input_ids, masked_lm_positions, masked_lm_ids, masked_lm_weights = tokenizer.encode(
        "This is a docudrama story on the Lindy Chamberlain case and a look at "
        "its impact on Australian society It especially looks at the problem of "
        "innuendo gossip and expectation when dealing with reallife dramasbr br "
        "One issue the story deals with is the way it is expected people will all "
        "give the same emotional response to similar situations Not everyone goes "
        "into wild melodramatic hysterics to every major crisis Just because the "
        "characters in the movies and on TV act in a certain way is no reason to "
        "expect real people to do so", max_seq_len=max_seq_len, max_pred_len=max_pred_length)

seq_len = len(input_ids)
masked_lm_positions_tmp = np.full(seq_len, 0, dtype=np.int32)
masked_lm_ids_tmp = np.full([seq_len, 1], -1, dtype=np.int64)
masked_lm_weights_tmp = np.full([seq_len], 0, dtype="float32")

mask_token_num = 0
for i, x in enumerate([input_ids]):
    for j, pos in enumerate(masked_lm_positions):
        masked_lm_positions_tmp[mask_token_num] = i * seq_len + pos
        masked_lm_ids_tmp[mask_token_num] = masked_lm_ids[j]
        masked_lm_weights_tmp[mask_token_num] = masked_lm_weights[j]

masked_lm_positions = masked_lm_positions_tmp
masked_lm_ids = masked_lm_ids_tmp
masked_lm_weights = masked_lm_weights_tmp
print(masked_lm_ids.shape)
input_ids = paddle.to_tensor([input_ids])
masked_lm_positions = paddle.to_tensor(masked_lm_positions)
masked_lm_ids = paddle.to_tensor(masked_lm_ids, dtype='int64')
masked_lm_weights = paddle.to_tensor(masked_lm_weights)
masked_lm_scale = 1.0
next_sentence_labels = paddle.zeros(shape=(1, 1), dtype='int64')

rand_mask_idx_list = create_bigbird_rand_mask_idx_list(
    config["num_layers"], seq_len, seq_len, config["nhead"],
    config["block_size"], config["window_size"], config["num_global_blocks"],
    config["num_rand_blocks"], config["seed"])
rand_mask_idx_list = [
    paddle.to_tensor(rand_mask_idx) for rand_mask_idx in rand_mask_idx_list
]
prediction_scores, seq_relationship_score = model(input_ids, rand_mask_idx_list=rand_mask_idx_list, masked_positions=masked_lm_positions)

loss = criterion(prediction_scores, seq_relationship_score,
                masked_lm_ids, next_sentence_labels,
                masked_lm_scale, masked_lm_weights)
print(loss)

class BigBirdForSequenceClassification(config: paddlenlp.transformers.bigbird.configuration.BigBirdConfig)[source]¶

Bases: paddlenlp.transformers.bigbird.modeling.BigBirdPretrainedModel

BigBird Model with a linear layer on top of the output layer, designed for sequence classification/regression tasks like GLUE tasks.

Parameters

bigbird (BigBirdModel) – An instance of BigBirdModel.
num_classes (int, optional) – The number of classes. Defaults to None.

forward(input_ids: Optional[paddle.Tensor] = None, token_type_ids: Optional[paddle.Tensor] = None, attention_mask: Optional[paddle.Tensor] = None, rand_mask_idx_list: Optional[List] = None, inputs_embeds: Optional[paddle.Tensor] = None, labels: Optional[paddle.Tensor] = None, output_hidden_states: Optional[bool] = None, output_attentions: Optional[bool] = None, return_dict: Optional[bool] = None)[source]¶

The BigBirdForSequenceClassification forward method, overrides the __call__() special method.

Parameters

input_ids (Tensor) – See BigBirdModel.
token_type_ids (Tensor) – See BigBirdModel.
attention_mask (Tensor) – See BigBirdModel.
rand_mask_idx_list (list) – See BigBirdModel.
inputs_embeds (Tensor, optional) – See BigBirdModel.
labels (Tensor of shape (batch_size,), optional) – Labels for computing the sequence classification/regression loss. Indices should be in [0, ..., num_labels - 1]. If num_labels == 1 a regression loss is computed (Mean-Square loss), If num_labels > 1 a classification loss is computed (Cross-Entropy).
output_hidden_states (bool, optional) – Whether to return the hidden states of all layers. Defaults to None.
output_attentions (bool, optional) – Whether to return the attentions tensors of all attention layers. Defaults to None.
return_dict (bool, optional) – Whether to return a SequenceClassifierOutput object. If False, the output will be a tuple of tensors. Defaults to None.

Returns

Returns tensor output, a tensor of the input text classification logits. Its data type should be float32 and it has a shape of [batch_size, num_classes].

Return type

Tensor

Examples

import paddle
from paddlenlp.transformers import BigBirdForSequenceClassification, BigBirdTokenizer
from paddlenlp.transformers import create_bigbird_rand_mask_idx_list

tokenizer = BigBirdTokenizer.from_pretrained('bigbird-base-uncased')
model = BigBirdForSequenceClassification.from_pretrained('bigbird-base-uncased')
config = model.bigbird.config
max_seq_len = 512
input_ids = tokenizer.convert_tokens_to_ids(
    tokenizer(
        "This is a docudrama story on the Lindy Chamberlain case and a look at "
        "its impact on Australian society It especially looks at the problem of "
        "innuendo gossip and expectation when dealing with reallife dramasbr br "
        "One issue the story deals with is the way it is expected people will all "
        "give the same emotional response to similar situations Not everyone goes "
        "into wild melodramatic hysterics to every major crisis Just because the "
        "characters in the movies and on TV act in a certain way is no reason to "
        "expect real people to do so"
    ))
input_ids.extend([0] * (max_seq_len - len(input_ids)))
seq_len = len(input_ids)
input_ids = paddle.to_tensor([input_ids])
rand_mask_idx_list = create_bigbird_rand_mask_idx_list(
    config["num_layers"], seq_len, seq_len, config["nhead"],
    config["block_size"], config["window_size"], config["num_global_blocks"],
    config["num_rand_blocks"], config["seed"])
rand_mask_idx_list = [
    paddle.to_tensor(rand_mask_idx) for rand_mask_idx in rand_mask_idx_list
]
output = model(input_ids, rand_mask_idx_list=rand_mask_idx_list)
print(output)

class BigBirdPretrainingHeads(config: paddlenlp.transformers.bigbird.configuration.BigBirdConfig)[source]¶

Bases: paddle.fluid.dygraph.layers.Layer

The BigBird pretraining heads for a pretraining task.

Parameters

hidden_size (int) – See BigBirdModel.
vocab_size (int) – See BigBirdModel.
activation (str) – See BigBirdModel.
embedding_weights (Tensor, optional) – The weight of pretraining embedding layer. Its data type should be float32 and its shape is [hidden_size, vocab_size]. If set to None, use normal distribution to initialize weight. Defaults to None.

forward(sequence_output, pooled_output, masked_positions=None)[source]¶

The BigBirdPretrainingHeads forward method, overrides the __call__() special method.

Parameters

sequence_output (Tensor) – The sequence output of BigBirdModel. Its data type should be float32 and has a shape of [batch_size, sequence_length, hidden_size].
pooled_output (Tensor) – The pooled output of BigBirdModel. Its data type should be float32 and has a shape of [batch_size, hidden_size].
masked_positions (Tensor) – A tensor indicates positions to be masked in the position embedding. Its data type should be int64 and its shape is [batch_size, mask_token_num]. mask_token_num is the number of masked tokens. It should be no bigger than sequence_length. Defaults to None, which means we output hidden-states of all tokens in masked token prediction.

Returns

(prediction_scores, seq_relationship_score).

With the fields:

prediction_scores (Tensor):
The scores of masked token prediction. Its data type should be float32. If masked_positions is None, its shape is [batch_size, sequence_length, vocab_size]. Otherwise, its shape is [batch_size, mask_token_num, vocab_size].
seq_relationship_score (Tensor):
The logits whether 2 sequences are NSP relationship. Its data type should be float32 and has a shape of [batch_size, 2].

Return type

tuple

class BigBirdForQuestionAnswering(config: paddlenlp.transformers.bigbird.configuration.BigBirdConfig)[source]¶

Bases: paddlenlp.transformers.bigbird.modeling.BigBirdPretrainedModel

BigBird Model with a linear layer on top of the hidden-states output to compute span_start_logits and span_end_logits, designed for question-answering tasks like SQuAD.

Parameters

bigbird (BigBirdModel) – An instance of BigBirdModel.
dropout (float, optional) – The dropout probability for output of BigBirdModel. If None, use the same value as hidden_dropout_prob of BigBirdModel instance bigbird. Defaults to None.

forward(input_ids: Optional[paddle.Tensor] = None, token_type_ids: Optional[paddle.Tensor] = None, attention_mask: Optional[paddle.Tensor] = None, start_positions: Optional[paddle.Tensor] = None, end_positions: Optional[paddle.Tensor] = None, rand_mask_idx_list: Optional[List] = None, inputs_embeds: Optional[paddle.Tensor] = None, output_hidden_states: Optional[bool] = None, output_attentions: Optional[bool] = None, return_dict: Optional[bool] = None)[source]¶

The BigBirdForQuestionAnswering forward method, overrides the __call__() special method.

Parameters

input_ids (Tensor) – See BigBirdModel.
token_type_ids (Tensor, optional) – See BigBirdModel.
attention_mask (Tensor) – See BigBirdModel.
rand_mask_idx_list (List) – See BigBirdModel.
inputs_embeds (Tensor, optional) – See BigBirdModel.
output_hidden_states (bool, optional) – Whether to return the hidden states of all layers. Defaults to None.
output_attentions (bool, optional) – Whether to return the attentions tensors of all attention layers. Defaults to None.
return_dict (bool, optional) – Whether to return a QuestionAnsweringModelOutput object. If False, the output will be a tuple of tensors. Defaults to None.

Returns

An instance of QuestionAnsweringModelOutput if return_dict=True. Otherwise it returns a tuple of tensors corresponding to ordered and not None (depending on the input arguments) fields of QuestionAnsweringModelOutput.

Example

import paddle
from paddlenlp.transformers.bigbird.modeling import BigBirdForQuestionAnswering
from paddlenlp.transformers.bigbird.tokenizer import BigBirdTokenizer

tokenizer = BigBirdTokenizer.from_pretrained('bigbird-base-uncased')
model = BigBirdForQuestionAnswering.from_pretrained('bigbird-base-uncased')

inputs = tokenizer("Welcome to use PaddlePaddle and PaddleNLP!", return_tensors='pd')
inputs = {k:paddle.to_tensor([v]) for (k, v) in inputs.items()}
outputs = model(**inputs)

start_logits = outputs[0]
end_logits  =outputs[1]

class BigBirdForTokenClassification(config: paddlenlp.transformers.bigbird.configuration.BigBirdConfig)[source]¶

Bases: paddlenlp.transformers.bigbird.modeling.BigBirdPretrainedModel

BigBird Model with a linear layer on top of the hidden-states output layer, designed for token classification tasks like NER tasks.

Parameters

bigbird (BigBirdModel) – An instance of BigBirdModel.
num_classes (int, optional) – The number of classes. Defaults to 2.
dropout (float, optional) – The dropout probability for output of BIGBIRD. If None, use the same value as hidden_dropout_prob of BigBirdModel instance bigbird. Defaults to None.

forward(input_ids: Optional[paddle.Tensor] = None, token_type_ids: Optional[paddle.Tensor] = None, attention_mask: Optional[paddle.Tensor] = None, rand_mask_idx_list: Optional[List] = None, inputs_embeds: Optional[paddle.Tensor] = None, labels: Optional[paddle.Tensor] = None, output_hidden_states: Optional[bool] = None, output_attentions: Optional[bool] = None, return_dict: Optional[bool] = None)[source]¶

The BigBirdForSequenceClassification forward method, overrides the __call__() special method.

Parameters

input_ids (Tensor) – See BigBirdModel.
token_type_ids (Tensor, optional) – See BigBirdModel.
attention_mask (Tensor) – See BigBirdModel.
rand_mask_idx_list (List) – See BigBirdModel.
labels (Tensor of shape (batch_size, sequence_length), optional) – Labels for computing the token classification loss. Indices should be in [0, ..., num_labels - 1].
inputs_embeds (Tensor, optional) – See BigBirdModel.
output_hidden_states (bool, optional) – Whether to return the hidden states of all layers. Defaults to None.
output_attentions (bool, optional) – Whether to return the attentions tensors of all attention layers. Defaults to None.
return_dict (bool, optional) – Whether to return a TokenClassifierOutput object. If

Returns

An instance of TokenClassifierOutput if return_dict=True. Otherwise it returns a tuple of tensors corresponding to ordered and not None (depending on the input arguments) fields of TokenClassifierOutput.

Example

import paddle
from paddlenlp.transformers.bigbird.modeling import BigBirdForTokenClassification
from paddlenlp.transformers.bigbird.tokenizer import BigBirdTokenizer

tokenizer = BigBirdTokenizer.from_pretrained('bigbird-base-uncased')
model = BigBirdForTokenClassification.from_pretrained('bigbird-base-uncased')

inputs = tokenizer("Welcome to use PaddlePaddle and PaddleNLP!", return_tensors='pd')
inputs = {k:paddle.to_tensor([v]) for (k, v) in inputs.items()}
outputs = model(**inputs)

logits = outputs

class BigBirdForMultipleChoice(config: paddlenlp.transformers.bigbird.configuration.BigBirdConfig)[source]¶

Bases: paddlenlp.transformers.bigbird.modeling.BigBirdPretrainedModel

BigBird Model with a linear layer on top of the hidden-states output layer, designed for multiple choice tasks like RocStories/SWAG tasks .

Parameters

bigbird (BigBirdModel) – An instance of BigBirdModel.
num_choices (int, optional) – The number of choices. Defaults to 2.
dropout (float, optional) – The dropout probability for output of BIGBIRD. If None, use the same value as hidden_dropout_prob of BigBirdModel instance bigbird. Defaults to None.

forward(input_ids: Optional[paddle.Tensor] = None, attention_mask: Optional[paddle.Tensor] = None, rand_mask_idx_list: Optional[List] = None, token_type_ids: Optional[paddle.Tensor] = None, labels: Optional[paddle.Tensor] = None, inputs_embeds: Optional[paddle.Tensor] = None, output_hidden_states: Optional[bool] = None, output_attentions: Optional[bool] = None, return_dict: Optional[bool] = None)[source]¶

The BigBirdForMultipleChoice forward method, overrides the __call__() special method.

Parameters

input_ids (Tensor) – See BigBirdModel and shape as [batch_size, num_choice, sequence_length].
attention_mask (Tensor) – See BigBirdModel and shape as [batch_size, num_choice, n_head, sequence_length, sequence_length].
rand_mask_idx_list (List) – See BigBirdModel.
labels (Tensor of shape (batch_size, ), optional) – Labels for computing the multiple choice classification loss. Indices should be in [0, ..., num_choices-1] where num_choices is the size of the second dimension of the input tensors. (See input_ids above)
inputs_embeds (Tensor, optional) – See BigBirdModel.
output_hidden_states (bool, optional) – Whether to return the hidden states of all layers. Defaults to None.
output_attentions (bool, optional) – Whether to return the attentions tensors of all attention layers. Defaults to None.
return_dict (bool, optional) – Whether to return a MultipleChoiceModelOutput object. If False, the output will be a tuple of tensors. Defaults to None.

Returns

An instance of MultipleChoiceModelOutput if return_dict=True. Otherwise it returns a tuple of tensors corresponding to ordered and not None (depending on the input arguments) fields of MultipleChoiceModelOutput.

Example

import paddle
from paddlenlp.transformers.bigbird.modeling import BigBirdForMultipleChoice
from paddlenlp.transformers.bigbird.tokenizer import BigBirdTokenizer

tokenizer = BigBirdTokenizer.from_pretrained('bigbird-base-uncased')
model = BigBirdForTokenClassification.from_pretrained('bigbird-base-uncased')

inputs = tokenizer("Welcome to use PaddlePaddle and PaddleNLP!", return_tensors='pd')
inputs = {k:paddle.to_tensor([v]) for (k, v) in inputs.items()}
outputs = model(**inputs)

logits = outputs

class BigBirdForMaskedLM(config: paddlenlp.transformers.bigbird.configuration.BigBirdConfig)[source]¶

Bases: paddlenlp.transformers.bigbird.modeling.BigBirdPretrainedModel

BigBird Model with a language modeling head on top.

Parameters: BigBird (BigBirdModel) – An instance of BigBirdModel.

forward(input_ids: Optional[paddle.Tensor] = None, attention_mask: Optional[paddle.Tensor] = None, rand_mask_idx_list: Optional[List] = None, inputs_embeds: Optional[paddle.Tensor] = None, labels: Optional[paddle.Tensor] = None, output_hidden_states: Optional[bool] = None, output_attentions: Optional[bool] = None, return_dict: Optional[bool] = None)[source]¶

Parameters

input_ids (Tensor) – See BigBirdModel.
attention_mask (Tensor) – See BigBirdModel.
rand_mask_idx_list (List) – See BigBirdModel.
inputs_embeds (Tensor, optional) – See BigBirdModel.
labels (Tensor, optional) – The Labels for computing the masked language modeling loss. Indices should be in [-100, 0, ..., vocab_size] Tokens with indices set to -100 are ignored (masked), the loss is only computed for the tokens with labels in [0, ..., vocab_size] Its shape is [batch_size, sequence_length].
output_hidden_states (bool, optional) – Whether to return the hidden states of all layers. Defaults to None.
output_attentions (bool, optional) – Whether to return the attentions tensors of all attention layers. Defaults to None.
return_dict (bool, optional) – Whether to return a MaskedLMOutput object. If False, the output will be a tuple of tensors. Defaults to None.

Returns

Returns tuple (masked_lm_loss, prediction_scores, ``sequence_output`).

With the fields:

masked_lm_loss (Tensor):
The masked lm loss. Its data type should be float32 and its shape is [1].
prediction_scores (Tensor):
The scores of masked token prediction. Its data type should be float32. Its shape is [batch_size, sequence_length, vocab_size].
sequence_output (Tensor):
Sequence of hidden-states at the last layer of the model. Its data type should be float32. Its shape is [batch_size, sequence_length, hidden_size].

Return type

tuple

class BigBirdForCausalLM(config: paddlenlp.transformers.bigbird.configuration.BigBirdConfig)[source]¶

Bases: paddlenlp.transformers.bigbird.modeling.BigBirdPretrainedModel

BigBird Model for casual language model tasks.

Parameters: BigBird (BigBirdModel) – An instance of BigBirdModel.

forward(input_ids: Optional[paddle.Tensor] = None, attention_mask: Optional[paddle.Tensor] = None, rand_mask_idx_list: Optional[List] = None, inputs_embeds: Optional[paddle.Tensor] = None, labels: Optional[paddle.Tensor] = None, output_hidden_states: Optional[bool] = None, output_attentions: Optional[bool] = None, return_dict: Optional[bool] = None)[source]¶

Parameters

input_ids (Tensor) – See BigBirdModel.
attention_mask (Tensor) – See BigBirdModel.
rand_mask_idx_list (List) – See BigBirdModel.
inputs_embeds (Tensor, optional) – See BigBirdModel.
labels (Tensor, optional) – The Labels for computing the masked language modeling loss. Indices should be in [-100, 0, ..., vocab_size] Tokens with indices set to -100 are ignored (masked), the loss is only computed for the tokens with labels in [0, ..., vocab_size] Its shape is [batch_size, sequence_length].
output_hidden_states (bool, optional) – Whether to return the hidden states of all layers. Defaults to False.
output_attentions (bool, optional) – Whether to return the attentions tensors of all attention layers. Defaults to False.
return_dict (bool, optional) – Whether to return a TokenClassifierOutput object. If False, the output will be a tuple of tensors. Defaults to False.

Returns

Returns tuple (masked_lm_loss, prediction_scores, ``sequence_output`).

With the fields:

masked_lm_loss (Tensor):
The masked lm loss. Its data type should be float32 and its shape is [1].
prediction_scores (Tensor):
The scores of masked token prediction. Its data type should be float32. Its shape is [batch_size, sequence_length, vocab_size].
sequence_output (Tensor):
Sequence of hidden-states at the last layer of the model. Its data type should be float32. Its shape is [batch_size, sequence_length, hidden_size].

Return type

tuple