The base class for all pretrained models. It mainly provides common methods for loading (construction and loading) and saving pretrained models. Loading and saving also rely on the following class attributes which should be overridden by derived classes accordingly:
model_config_file (str): Represents the file name of model configuration for configuration saving and loading in local file system. The value is
resource_files_names (dict): Name of local file where the model configuration can be saved and loaded locally. Currently, resources only include the model state, thus the dict only includes
'model_state'as key with corresponding value
'model_state.pdparams'for model weights saving and loading.
pretrained_init_configuration (dict): Provides the model configurations of built-in pretrained models (contrasts to models in local file system). It has pretrained model names as keys (such as
bert-base-uncased), and the values are dict preserving corresponding configuration for model initialization.
pretrained_resource_files_map (dict): Provides resource URLs of built-in pretrained models (contrasts to models in local file system). It has the same key as resource_files_names (that is "model_state"), and the corresponding value is a dict with specific model name to model weights URL mapping (such as "bert-base-uncased" -> "https://bj.bcebos.com/paddlenlp/models/transformers/bert-base-uncased.pdparams").
base_model_prefix (str): Represents the attribute associated to the base model in derived classes of the same architecture adding layers on top of the base model. Note: A base model class is pretrained model class decorated by
register_base_model, such as
BertModel; A derived model class is a pretrained model class adding layers on top of the base model, and it has a base model as attribute, such as
Methods common to models for text generation are defined in
GenerationMixinand also inherited here.
InitTrackerMetais used to create
PretrainedModel, by which subclasses can track arguments for initialization automatically.
The body of the same model architecture. It is the base model itself for base model or the base model attribute for derived model.
Contains all supported built-in pretrained model names of the current PretrainedModel class.
from_pretrained(pretrained_model_name_or_path, *args, **kwargs)¶
Creates an instance of
PretrainedModel. Model weights are loaded by specifying name of a built-in pretrained model, or a community contributed model, or a local file directory path.
pretrained_model_name_or_path (str) --
Name of pretrained model or dir path to load from. The string can be:
Name of a built-in pretrained model
Name of a community-contributed pretrained model.
Local directory path which contains model weights file("model_state.pdparams") and model config file ("model_config.json").
*args (tuple) -- Position arguments for model
__init__. If provided, use these as position argument values for model initialization.
**kwargs (dict) -- Keyword arguments for model
__init__. If provided, use these to update pre-defined keyword argument values for model initialization. If the keyword is in
__init__argument names of base model, update argument values of the base model; else update argument values of derived model.
load_state_as_np (bool, optional) -- The weights read in can be choosed to place on CPU or GPU though the model is on the default device. If
True, load the model weights as
numpy.ndarrayon CPU. Otherwise, weights would be loaded as tensors on the default device. Note that if on GPU, the latter would creates extra temporary tensors in addition to the model weights, which doubles the memory usage . Thus it is suggested to use
Truefor big models on GPU. Default to
An instance of
from paddlenlp.transformers import BertForSequenceClassification # Name of built-in pretrained model model = BertForSequenceClassification.from_pretrained('bert-base-uncased') # Name of community-contributed pretrained model model = BertForSequenceClassification.from_pretrained('yingyibiao/bert-base-uncased-sst-2-finetuned') # Load from local directory path model = BertForSequenceClassification.from_pretrained('./my_bert/')
Get model configuration.
The config of the model.
Saves model configuration to a file named "model_config.json" under
save_dir (str) -- Directory to save model_config file into.
Saves model configuration and related resources (model state) as files under
save_dir. The model configuration would be saved into a file named "model_config.json", and model state would be saved into a file named "model_state.pdparams".
save_dircan be used in
from_pretrainedas argument value of
pretrained_model_name_or_pathto re-load the trained model.
save_dir (str) -- Directory to save files into.
from paddlenlp.transformers import BertForSequenceClassification model = BertForSequenceClassification.from_pretrained('bert-base-uncased') model.save_pretrained('./trained_model/') # reload from save_directory model = BertForSequenceClassification.from_pretrained('./trained_model/')
Resizes input token embeddings matrix of the model according to new_num_tokens.
new_num_tokens (int) -- The number of new tokens in the embedding matrix. Increasing the size will add newly initialized vectors at the end. Reducing the size will remove vectors from the end. If not provided or None, just returns a pointer to the input tokens embedding module of the model without doing anything.
The input tokens Embeddings Module of the model.
A decorator for
PretrainedModelclass. It first retrieves the parent class of the class being decorated, then sets the
base_model_classattribute of that parent class to be the class being decorated. In summary, the decorator registers the decorated class as the base model class in all derived classes under the same architecture.
cls (PretrainedModel) -- The class (inherited from PretrainedModel) to be decorated .
The input class
from paddlenlp.transformers import BertModel, register_base_model BertModel = register_base_model(BertModel) assert BertModel.base_model_class == BertModel