modeling#

Modeling classes for FNet model.

class FNetPretrainedModel(*args, **kwargs)[source]#

Bases: PretrainedModel

An abstract class for pretrained FNet models. It provides FNet related model_config_file, pretrained_init_configuration, resource_files_names, pretrained_resource_files_map, base_model_prefix for downloading and loading pretrained models. See PretrainedModel for more details.

config_class#

alias of FNetConfig

base_model_class#

alias of FNetModel

class FNetModel(config: FNetConfig)[source]#

Bases: FNetPretrainedModel

The model can behave as an encoder, following the architecture described in FNet: Mixing Tokens with Fourier Transforms by James Lee-Thorp, Joshua Ainslie, Ilya Eckstein, Santiago Ontanon.

get_input_embeddings()[source]#

get input embedding of model

Returns:

embedding of model

Return type:

nn.Embedding

set_input_embeddings(value)[source]#

set new input embedding for model

Parameters:

value (Embedding) – the new embedding of model

Raises:

NotImplementedError – Model has not implement set_input_embeddings method

forward(input_ids=None, token_type_ids=None, position_ids=None, inputs_embeds=None, output_hidden_states=None, return_dict=None)[source]#

The FNetModel forward method.

Parameters:

  • input_ids (Tensor) – Indices of input sequence tokens in the vocabulary. They are numerical representations of tokens that build the input sequence. Its data type should be int64 and it has a shape of [batch_size, sequence_length].

  • token_type_ids (Tensor, optional) –

    Segment token indices to indicate different portions of the inputs. Selected in the range [0, type_vocab_size - 1]. If type_vocab_size is 2, which means the inputs have two portions. Indices can either be 0 or 1:

    • 0 corresponds to a sentence A token,

    • 1 corresponds to a sentence B token.

    Its data type should be int64 and it has a shape of [batch_size, sequence_length]. Defaults to None, which means we don’t add segment embeddings.

  • position_ids (Tensor, optional) – Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, max_position_embeddings - 1]. Shape as (batch_size, num_tokens) and dtype as int64. Defaults to None.

  • inputs_embeds – If you want to control how to convert inputs_ids indices into associated vectors, you can pass an embedded representation directly instead of passing inputs_ids.

Returns:

Returns tuple (sequence_output, pooled_output, encoder_outputs[1:]) or a dict with last_hidden_state`, pooled_output, all_hidden_states, fields.

With the fields:

  • sequence_output (Tensor):

    Sequence of hidden-states at the last layer of the model. It’s data type should be float32 and has a shape of [batch_size, sequence_length, hidden_size].

  • pooled_output (Tensor):

    The output of first token ([CLS]) in sequence. We “pool” the model by simply taking the hidden state corresponding to the first token. Its data type should be float32 and has a shape of [batch_size, hidden_size].

  • last_hidden_state (Tensor):

    The output of the last encoder layer, it is also the sequence_output. It’s data type should be float32 and has a shape of [batch_size, sequence_length, hidden_size].

  • all_hidden_states (Tensor):

    Hidden_states of all layers in the Transformer encoder. The length of all_hidden_states is num_hidden_layers + 1. For all element in the tuple, its data type should be float32 and its shape is [batch_size, sequence_length, hidden_size].

Return type:

tuple or Dict

Example

import paddle
from paddlenlp.transformers.fnet.modeling import FNetModel
from paddlenlp.transformers.fnet.tokenizer import FNetTokenizer

tokenizer = FNetTokenizer.from_pretrained('fnet-base')
model = FNetModel.from_pretrained('fnet-base')

inputs = tokenizer("Welcome to use PaddlePaddle and PaddleNLP!")
inputs = {k:paddle.to_tensor([v]) for (k, v) in inputs.items()}
output = model(**inputs)
class FNetForSequenceClassification(config: FNetConfig, num_classes=2)[source]#

Bases: FNetPretrainedModel

FNet Model with a linear layer on top of the output layer, designed for sequence classification/regression tasks like GLUE tasks.

Parameters:

  • fnet (FNetModel) – An instance of FNetModel.

  • num_classes (int, optional) – The number of classes. Defaults to 2.

forward(input_ids=None, token_type_ids=None, position_ids=None, inputs_embeds=None, labels=None, output_hidden_states=None, return_dict=None)[source]#

The FNetForSequenceClassification forward method.

Parameters:

  • input_ids (Tensor) – Indices of input sequence tokens in the vocabulary. They are numerical representations of tokens that build the input sequence. Its data type should be int64 and it has a shape of [batch_size, sequence_length].

  • token_type_ids (Tensor, optional) –

    Segment token indices to indicate different portions of the inputs. Selected in the range [0, type_vocab_size - 1]. If type_vocab_size is 2, which means the inputs have two portions. Indices can either be 0 or 1:

    • 0 corresponds to a sentence A token,

    • 1 corresponds to a sentence B token.

    Its data type should be int64 and it has a shape of [batch_size, sequence_length]. Defaults to None, which means we don’t add segment embeddings.

  • position_ids (Tensor, optional) – Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, max_position_embeddings - 1]. Shape as (batch_size, num_tokens) and dtype as int64. Defaults to None.

  • inputs_embeds – If you want to control how to convert inputs_ids indices into associated vectors, you can pass an embedded representation directly instead of passing inputs_ids.

Returns:

Returns tensor logits, or a dict with logits, hidden_states, attentions fields.

With the fields:

  • logits (Tensor):

    A tensor of the input text classification logits. Shape as [batch_size, num_classes] and dtype as float32.

  • hidden_states (Tensor):

    Hidden_states of all layers in the Transformer encoder. The length of hidden_states is num_hidden_layers + 1. For all element in the tuple, its data type should be float32 and its shape is [batch_size, sequence_length, hidden_size].

Return type:

Tensor or Dict

Example

import paddle
from paddlenlp.transformers.fnet.modeling import FNetForSequenceClassification
from paddlenlp.transformers.fnet.tokenizer import FNetTokenizer

tokenizer = FNetTokenizer.from_pretrained('fnet-base')
model = FNetModel.from_pretrained('fnet-base')

inputs = tokenizer("Welcome to use PaddlePaddle and PaddleNLP!")
inputs = {k:paddle.to_tensor([v]) for (k, v) in inputs.items()}
output = model(**inputs)
class FNetForPreTraining(config: FNetConfig)[source]#

Bases: FNetPretrainedModel

FNet Model with two heads on top as done during the pretraining: a masked language modeling head and a next sentence prediction (classification) head.

get_output_embeddings()[source]#

To be overwrited for models with output embeddings

Returns:

the otuput embedding of model

Return type:

Optional[Embedding]

get_input_embeddings()[source]#

get input embedding of model

Returns:

embedding of model

Return type:

nn.Embedding

forward(input_ids=None, token_type_ids=None, position_ids=None, inputs_embeds=None, labels=None, next_sentence_label=None, output_hidden_states=None, return_dict=None)[source]#

The FNetForPretraining forward method.

Parameters:

  • input_ids (Tensor) – See FNetModel.

  • token_type_ids (Tensor, optional) – See FNetModel.

  • position_ids (Tensor, optional) – See FNetModel.

  • labels (LongTensor of shape (batch_size, sequence_length), optional) – Labels for computing the masked language modeling loss.

  • inputs_embeds (Tensor, optional) – See FNetModel.

  • next_sentence_labels (Tensor) – The labels of the next sentence prediction task, the dimensionality of next_sentence_labels is equal to seq_relation_labels. Its data type should be int64 and its shape is [batch_size, 1]

  • output_hidden_states (bool, optional) – See FNetModel.

  • return_dict (bool, optional) – See FNetModel.

Returns:

Returns tuple (prediction_scores, seq_relationship_score) or a dict with prediction_logits, seq_relationship_logits, hidden_states fields.

Return type:

tuple or Dict

class FNetForMaskedLM(config: FNetConfig)[source]#

Bases: FNetPretrainedModel

FNet Model with a masked language modeling head on top.

Parameters:

fnet (FNetModel) – An instance of FNetModel.

get_output_embeddings()[source]#

To be overwrited for models with output embeddings

Returns:

the otuput embedding of model

Return type:

Optional[Embedding]

get_input_embeddings()[source]#

get input embedding of model

Returns:

embedding of model

Return type:

nn.Embedding

forward(input_ids=None, token_type_ids=None, position_ids=None, inputs_embeds=None, labels=None, next_sentence_label=None, output_hidden_states=None, return_dict=None)[source]#

The FNetForMaskedLM forward method.

Parameters:
Returns:

Returns tensor prediction_scores or a dict with prediction_logits, hidden_states fields.

With the fields:

  • prediction_scores (Tensor):

    The scores of masked token prediction. Its data type should be float32. and its shape is [batch_size, sequence_length, vocab_size].

  • hidden_states (Tensor):

    Hidden_states of all layers in the Transformer encoder. The length of hidden_states is num_hidden_layers + 1. For all element in the tuple, its data type should be float32 and its shape is [batch_size, sequence_length, hidden_size].

Return type:

Tensor or Dict

class FNetForNextSentencePrediction(config: FNetConfig)[source]#

Bases: FNetPretrainedModel

FNet Model with a next sentence prediction head on top.

Parameters:

fnet (FNetModel) – An instance of FNetModel.

get_output_embeddings()[source]#

To be overwrited for models with output embeddings

Returns:

the otuput embedding of model

Return type:

Optional[Embedding]

get_input_embeddings()[source]#

get input embedding of model

Returns:

embedding of model

Return type:

nn.Embedding

forward(input_ids=None, token_type_ids=None, position_ids=None, inputs_embeds=None, labels=None, next_sentence_label=None, output_hidden_states=None, return_dict=None)[source]#

Defines the computation performed at every call. Should be overridden by all subclasses.

Parameters:

  • *inputs (tuple) – unpacked tuple arguments

  • **kwargs (dict) – unpacked dict arguments

class FNetForMultipleChoice(config: FNetConfig)[source]#

Bases: FNetPretrainedModel

FNet Model with a linear layer on top of the hidden-states output layer, designed for multiple choice tasks like SWAG tasks .

Parameters:

fnet (FNetModel) – An instance of FNetModel.

forward(input_ids=None, token_type_ids=None, position_ids=None, inputs_embeds=None, labels=None, output_hidden_states=None, return_dict=None)[source]#

Defines the computation performed at every call. Should be overridden by all subclasses.

Parameters:

  • *inputs (tuple) – unpacked tuple arguments

  • **kwargs (dict) – unpacked dict arguments

class FNetForTokenClassification(config: FNetConfig, num_classes=2)[source]#

Bases: FNetPretrainedModel

FNet Model with a linear layer on top of the hidden-states output layer, designed for token classification tasks like NER tasks.

Parameters:

  • fnet (FNetModel) – An instance of FNetModel.

  • num_classes (int, optional) – The number of classes. Defaults to 2.

forward(input_ids=None, token_type_ids=None, position_ids=None, inputs_embeds=None, labels=None, output_hidden_states=None, return_dict=None)[source]#

Defines the computation performed at every call. Should be overridden by all subclasses.

Parameters:

  • *inputs (tuple) – unpacked tuple arguments

  • **kwargs (dict) – unpacked dict arguments

class FNetForQuestionAnswering(config: FNetConfig, num_labels)[source]#

Bases: FNetPretrainedModel

FNet Model with a linear layer on top of the hidden-states output to compute span_start_logits and span_end_logits, designed for question-answering tasks like SQuAD.

Parameters:

  • fnet (FNetModel) – An instance of FNetModel.

  • num_labels (int) – The number of labels.

forward(input_ids=None, token_type_ids=None, position_ids=None, inputs_embeds=None, start_positions=None, end_positions=None, output_hidden_states=None, return_dict=None)[source]#

Defines the computation performed at every call. Should be overridden by all subclasses.

Parameters:

  • *inputs (tuple) – unpacked tuple arguments

  • **kwargs (dict) – unpacked dict arguments