.. _nlp_model:

Model NLP
=========

The config file for NLP models contain three main sections:

    - ``trainer``: contains the configs for PTL training. For more information, refer to :doc:`../core/core` and `PTL Trainer class API <https://pytorch-lightning.readthedocs.io/en/stable/common/trainer.html#trainer-class-api>`.
    - ``exp_manager``: the configs of the experiment manager. For more information, refer to :doc:`../core/core`.
    - ``model``: contains the configs of the datasets, model architecture, tokenizer, optimizer, scheduler, etc.

The following sub-sections of the model section are shared among most of the NLP models.

    - ``tokenizer``: specifies the tokenizer
    - ``language_model``: specifies the underlying model to be used as the encoder
    - ``optim``: the configs of the optimizer and scheduler :doc:`../core/core`

The ``tokenizer`` and ``language_model`` sections have the following parameters:

+------------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **Parameter**                                  | **Data Type**   |  **Description**                                                                                             |
+------------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **model.tokenizer.tokenizer_name**             | string          | Tokenizer name will be filled automatically based on ``model.language_model.pretrained_model_name``.         |
+------------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **model.tokenizer.vocab_file**                 | string          | Path to tokenizer vocabulary.                                                                                |
+------------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **model.tokenizer.tokenizer_model**            | string          | Path to tokenizer model (only for sentencepiece tokenizer).                                                  |
+------------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **model.language_model.pretrained_model_name** | string          | Pre-trained language model name, for example: ``bert-base-cased`` or ``bert-base-uncased``.                  |
+------------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **model.language_model.lm_checkpoint**         | string          | Path to the pre-trained language model checkpoint.                                                           |
+------------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **model.language_model.config_file**           | string          | Path to the pre-trained language model config file.                                                          |
+------------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **model.language_model.config**                | dictionary      | Config of the pre-trained language model.                                                                    |
+------------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+

The parameter **model.language_model.pretrained_model_name** can be one of the following:

    - ``megatron-bert-345m-uncased``, ``megatron-bert-345m-cased``, ``biomegatron-bert-345m-uncased``, ``biomegatron-bert-345m-cased``, ``bert-base-uncased``, ``bert-large-uncased``, ``bert-base-cased``, ``bert-large-cased``
    - ``distilbert-base-uncased``, ``distilbert-base-cased``
    - ``roberta-base``, ``roberta-large``, ``distilroberta-base``
    - ``albert-base-v1``, ``albert-large-v1``, ``albert-xlarge-v1``, ``albert-xxlarge-v1``, ``albert-base-v2``, ``albert-large-v2``, ``albert-xlarge-v2``, ``albert-xxlarge-v2``