trainer_base¶
-
class
Trainer
(model: Optional[Union[paddlenlp.transformers.model_utils.PretrainedModel, paddle.fluid.dygraph.layers.Layer]] = None, criterion: Optional[paddle.fluid.dygraph.layers.Layer] = None, args: Optional[paddlenlp.trainer.training_args.TrainingArguments] = None, data_collator: Optional[DataCollator] = None, train_dataset: Optional[paddle.fluid.dataloader.dataset.Dataset] = None, eval_dataset: Optional[paddle.fluid.dataloader.dataset.Dataset] = None, tokenizer: Optional[paddlenlp.transformers.tokenizer_utils.PretrainedTokenizer] = None, compute_metrics: Optional[Callable[[paddlenlp.trainer.trainer_utils.EvalPrediction], Dict]] = None, callbacks: Optional[List[paddlenlp.trainer.trainer_callback.TrainerCallback]] = None, optimizers: Tuple[paddle.optimizer.optimizer.Optimizer, paddle.optimizer.lr.LRScheduler] = (None, None))[source]¶ Bases:
object
Trainer is a simple but feature-complete training and eval loop for PaddlePaddle, optimized for PaddleNLP.
- Parameters
model ([
PretrainedModel
] orpaddle.nn.Layer
, optional) –The model to train, evaluate or use for predictions.
[
Trainer
] is optimized to work with the [PretrainedModel
] provided by the library. You can still use your own models defined aspaddle.nn.Layer
as long as they work the same way as the PaddleNLP models.criterion (
paddle.nn.Layer
, optional) – The model may only output the loggit, if you want do more computation for the output of model, you can add the criterion Layer.args ([
TrainingArguments
], optional) – The arguments to tweak for training. Will default to a basic instance of [TrainingArguments
] with theoutput_dir
set to a directory named tmp_trainer in the current directory if not provided.data_collator (
DataCollator
, optional) – The function to use to form a batch from a list of elements oftrain_dataset
oreval_dataset
. Will default to [default_data_collator
] if notokenizer
is provided, an instance of [DataCollatorWithPadding
] otherwise.train_dataset (
paddle.io.Dataset
orpaddle.io.IterableDataset
, optional) – The dataset to use for training. If it is andatasets.Dataset
, columns not accepted by themodel.forward()
method are automatically removed.eval_dataset (
paddle.io.Dataset
, optional) – The dataset to use for evaluation. If it is andatasets.Dataset
, columns not accepted by themodel.forward()
method are automatically removed.tokenizer ([
PretrainedTokenizer
], optional) – The tokenizer used to preprocess the data. If provided, will be used to automatically pad the inputs the maximum length when batching inputs, and it will be saved along the model to make it easier to rerun an interrupted training or reuse the fine-tuned model.compute_metrics (
Callable[[EvalPrediction], Dict]
, optional) – The function that will be used to compute metrics at evaluation. Must take a [EvalPrediction
] and return a dictionary string to metric values.callbacks (List of [
TrainerCallback
], optional) – A list of callbacks to customize the training loop. Will add those to the list of default callbacks. If you want to remove one of the default callbacks used, use the [Trainer.remove_callback
] method.optimizers (
Tuple[paddle.optimizer.Optimizer, paddle.optimizer.lr.LRScheduler]
, optional) – A tuple containing the optimizer and the scheduler to use. Will default to an instance of [AdamW
] on your model and a scheduler given by [get_linear_schedule_with_warmup
] controlled byargs
.
Important attributes:
model – Always points to the core model. If using a transformers model, it will be a [
PretrainedModel
] subclass.model_wrapped – Always points to the most external model in case one or more other modules wrap the original model. This is the model that should be used for the forward pass. For example, the inner model is wrapped in
paddle.DataParallel
. If model hasn’t been wrapped, thenself.model_wrapped
is the same asself.model
.
-
log_metrics
(split, metrics)¶ Log metrics in a specially formatted way Under distributed environment this is done only for a process with rank 0. :param split: Mode/split name: one of
train
,eval
,test
:type split:str
:param metrics: The metrics returned from train/evaluate/predictmetrics: metrics dict :type metrics:Dict[str, float]
-
metrics_format
(metrics: Dict[str, float]) → Dict[str, float]¶ Reformat Trainer metrics values to a human-readable format :param metrics: The metrics returned from train/evaluate/predict :type metrics:
Dict[str, float]
- Returns
The reformatted metrics
- Return type
metrics (
Dict[str, float]
)
-
save_metrics
(split, metrics, combined=True)¶ Save metrics into a json file for that split, e.g.
train_results.json
. Under distributed environment this is done only for a process with rank 0. :param split: Mode/split name: one oftrain
,eval
,test
,all
:type split:str
:param metrics: The metrics returned from train/evaluate/predict :type metrics:Dict[str, float]
:param combined: Creates combined metrics by updatingall_results.json
with metrics of this call :type combined:bool
, optional, defaults toTrue
To understand the metrics please read the docstring of [
log_metrics
]. The only difference is that raw unformatted numbers are saved in the current method.
-
save_state
()¶ Saves the Trainer state, since Trainer.save_model saves only the tokenizer with the model Under distributed environment this is done only for a process with rank 0.
-
add_callback
(callback)[source]¶ Add a callback to the current list of [
TrainerCallback
].- Parameters
callback (
type
or [TrainerCallback
]) – A [TrainerCallback
] class or an instance of a [TrainerCallback
]. In the first case, will instantiate a member of that class.
-
pop_callback
(callback)[source]¶ Remove a callback from the current list of [
TrainerCallback
] and returns it. If the callback is not found, returnsNone
(and no error is raised). :param callback: A [TrainerCallback
] class or an instance of a [TrainerCallback
]. In thefirst case, will pop the first member of that class found in the list of callbacks.
- Returns
The callback removed, if found.
- Return type
[
TrainerCallback
]
-
remove_callback
(callback)[source]¶ Remove a callback from the current list of [
TrainerCallback
]. :param callback: A [TrainerCallback
] class or an instance of a [TrainerCallback
]. In thefirst case, will remove the first member of that class found in the list of callbacks.
-
load_state_dict_from_checkpoint
(resume_from_checkpoint=None)[source]¶ load state_dict from_checkpoint, Only load model state dict.
-
train
(resume_from_checkpoint: Optional[Union[bool, str]] = None, ignore_keys_for_eval: Optional[List[str]] = None)[source]¶ Main training entry point.
- Parameters
resume_from_checkpoint (
str
orbool
, optional) – If astr
, local path to a saved checkpoint as saved by a previous instance of [Trainer
]. If abool
and equalsTrue
, load the last checkpoint in args.output_dir as saved by a previous instance of [Trainer
]. If present, training will resume from the model/optimizer/scheduler states loaded here.ignore_keys_for_eval (
List[str]
, optional) – A list of keys in the output of your model (if it is a dictionary) that should be ignored when gathering predictions for evaluation during the training.
-
get_train_dataloader
()[source]¶ Returns the training [
DataLoader
].Will use no sampler if
self.train_dataset
does not implement__len__
, a random sampler (adapted to distributed training if necessary) otherwise.Subclass and override this method if you want to inject some custom behavior.
-
get_eval_dataloader
(eval_dataset: Optional[paddle.fluid.dataloader.dataset.Dataset] = None) → paddle.fluid.reader.DataLoader[source]¶ Returns the evaluation [
DataLoader
].Subclass and override this method if you want to inject some custom behavior.
- Parameters
eval_dataset (
paddle.io.Dataset
, optional) – If provided, will overrideself.eval_dataset
. If it is andatasets.Dataset
, columns not accepted by themodel.forward()
method are automatically removed. It must implement__len__
.
-
get_test_dataloader
(test_dataset: paddle.fluid.dataloader.dataset.Dataset) → paddle.fluid.reader.DataLoader[source]¶ Returns the test [
DataLoader
].Subclass and override this method if you want to inject some custom behavior.
- Parameters
test_dataset (
paddle.io.Dataset
, optional) – The test dataset to use. If it is andatasets.Dataset
, columns not accepted by themodel.forward()
method are automatically removed. It must implement__len__
.
-
create_optimizer_and_scheduler
(num_training_steps: int)[source]¶ Setup the optimizer and the learning rate scheduler.
We provide a reasonable default that works well. If you want to use something else, you can pass a tuple in the Trainer’s init through
optimizers
, or subclass and override this method (orcreate_optimizer
and/orcreate_scheduler
) in a subclass.
-
create_optimizer
(lr_scheduler=None)[source]¶ Setup the optimizer.
We provide a reasonable default that works well. If you want to use something else, you can pass a tuple in the Trainer’s init through
optimizers
, or subclass and override this method in a subclass.
-
static
get_optimizer_cls_and_kwargs
(args: paddlenlp.trainer.training_args.TrainingArguments) → Tuple[Any, Any][source]¶ Returns the optimizer class and optimizer parameters based on the training arguments.
- Parameters
args (
paddlenlp.training_args.TrainingArguments
) – The training arguments for the training session.
-
create_scheduler
(num_training_steps: int)[source]¶ Setup the scheduler. The optimizer of the trainer must have been set up either before this method is called or passed as an argument.
- Parameters
num_training_steps (int) – The number of training steps to do.
-
autocast_smart_context_manager
()[source]¶ A helper wrapper that creates an appropriate context manager for
autocast
while feeding it the desired arguments, depending on the situation.
-
compute_loss
(model, inputs, return_outputs=False)[source]¶ How the loss is computed by Trainer. By default, all models return the loss in the first element. Subclass and override for custom behavior.
-
training_step
(model: paddle.fluid.dygraph.layers.Layer, inputs: Dict[str, Union[paddle.Tensor, Any]]) → paddle.Tensor[source]¶ Perform a training step on a batch of inputs.
Subclass and override to inject custom behavior.
- Parameters
model (
nn.Layer
) – The model to train.inputs (
Dict[str, Union[paddle.Tensor, Any]]
) –The inputs and targets of the model.
The dictionary will be unpacked before being fed to the model. Most models expect the targets under the argument
labels
. Check your model’s documentation for all accepted arguments.
- Returns
The tensor with training loss on this batch.
- Return type
paddle.Tensor
-
save_model
(output_dir: Optional[str] = None)[source]¶ Will save the model, so you can reload it using
from_pretrained()
.Will only save from the main process.
-
log
(logs: Dict[str, float]) → None[source]¶ Log
logs
on the various objects watching training.Subclass and override this method to inject custom behavior.
- Parameters
logs (
Dict[str, float]
) – The values to log.
-
evaluate
(eval_dataset: Optional[paddle.fluid.dataloader.dataset.Dataset] = None, ignore_keys: Optional[List[str]] = None, metric_key_prefix: str = 'eval') → Dict[str, float][source]¶ Run evaluation and returns metrics.
The calling script will be responsible for providing a method to compute metrics, as they are task-dependent (pass it to the init
compute_metrics
argument).You can also subclass and override this method to inject custom behavior.
- Parameters
eval_dataset (
Dataset
, optional) – Pass a dataset if you wish to overrideself.eval_dataset
. If it is andatasets.Dataset
, columns not accepted by themodel.forward()
method are automatically removed. It must implement the__len__
method.ignore_keys (
Lst[str]
, optional) – A list of keys in the output of your model (if it is a dictionary) that should be ignored when gathering predictions.metric_key_prefix (
str
, optional, defaults to"eval"
) – An optional prefix to be used as the metrics key prefix. For example the metrics “bleu” will be named “eval_bleu” if the prefix is “eval” (default)
- Returns
A dictionary containing the evaluation loss and the potential metrics computed from the predictions. The dictionary also contains the epoch number which comes from the training state.
-
evaluation_loop
(dataloader: paddle.fluid.reader.DataLoader, description: str, prediction_loss_only: Optional[bool] = None, ignore_keys: Optional[List[str]] = None, metric_key_prefix: str = 'eval', max_eval_iters: Optional[int] = - 1) → paddlenlp.trainer.trainer_utils.EvalLoopOutput[source]¶ Prediction/evaluation loop, shared by
Trainer.evaluate()
andTrainer.predict()
.Works both with or without labels.
-
predict
(test_dataset: paddle.fluid.dataloader.dataset.Dataset, ignore_keys: Optional[List[str]] = None, metric_key_prefix: str = 'test') → paddlenlp.trainer.trainer_utils.PredictionOutput[source]¶ Run prediction and returns predictions and potential metrics. Depending on the dataset and your use case, your test dataset may contain labels. In that case, this method will also return metrics, like in
evaluate()
. :param test_dataset: Dataset to run the predictions on. If it is andatasets.Dataset
, columns not accepted by themodel.forward()
method are automatically removed. Has to implement the method__len__
- Parameters
ignore_keys (
Lst[str]
, optional) – A list of keys in the output of your model (if it is a dictionary) that should be ignored when gathering predictions.metric_key_prefix (
str
, optional, defaults to"test"
) – An optional prefix to be used as the metrics key prefix. For example the metrics “bleu” will be named “test_bleu” if the prefix is “test” (default)
<Tip> If your predictions or labels have different sequence length (for instance because you’re doing dynamic padding in a token classification task) the predictions will be padded (on the right) to allow for concatenation into one array. The padding index is -100. </Tip> Returns: NamedTuple A namedtuple with the following keys:
predictions (
np.ndarray
): The predictions ontest_dataset
.label_ids (
np.ndarray
, optional): The labels (if the dataset contained some).metrics (
Dict[str, float]
, optional): The potential dictionary of metrics (if the dataset contained labels).
-
prediction_step
(model: paddle.fluid.dygraph.layers.Layer, inputs: Dict[str, Union[paddle.Tensor, Any]], prediction_loss_only: bool, ignore_keys: Optional[List[str]] = None) → Tuple[Optional[paddle.Tensor], Optional[paddle.Tensor], Optional[paddle.Tensor]][source]¶ Perform an evaluation step on
model
usinginputs
.Subclass and override to inject custom behavior.
- Parameters
model (
nn.Layer
) – The model to evaluate.inputs (
Dict[str, Union[paddle.Tensor, Any]]
) –The inputs and targets of the model.
The dictionary will be unpacked before being fed to the model. Most models expect the targets under the argument
labels
. Check your model’s documentation for all accepted arguments.prediction_loss_only (
bool
) – Whether or not to return the loss only.ignore_keys (
Lst[str]
, optional) – A list of keys in the output of your model (if it is a dictionary) that should be ignored when gathering predictions.
- Returns
A tuple with the loss, logits and labels (each being optional).
- Return type
Tuple[Optional[paddle.Tensor], Optional[paddle.Tensor], Optional[paddle.Tensor]]
-
num_examples
(dataloader: paddle.fluid.reader.DataLoader) → int[source]¶ Helper to get number of samples in a [
DataLoader
] by accessing its dataset.Will raise an exception if the underlying dataset does not implement method
__len__
-
is_local_process_zero
() → bool[source]¶ Whether or not this process is the local (e.g., on one machine if training in a distributed fashion on several machines) main process.
-
is_world_process_zero
() → bool[source]¶ Whether or not this process is the global main process (when training in a distributed fashion on several machines, this is only going to be
True
for one process).
-
compress
()¶ Supports pruning DynaBERT and post-training quantization. If both are needed, pruning DynaBERT would be performed before quantizaton.
-
quant
(input_dir, strategy)¶ Supports Post-Training Quantization now.