---
license: apache-2.0
language: pt
tags:
- bert
- pytorch
- safetensors
- tensorflow 2
- onnx
datasets:
- brBanking
---

# BERTimbau Base Finance ("bert-base-portuguese-cased-finance")

![image/png](https://cdn-uploads.huggingface.co/production/uploads/63fe1a380c1bbe8e29d3c401/VPIefkyEgAtUNV5Q55D-f.png)

## Introduction

BERTimbau Finance Base is a model fine-tuned using two major Brazilian banking datasets, placing it within the finance domain.

It was developed by fine-tuning the BERTimbau Base language model.

BERTimbau Finance Base is available in multiple formats: Pytorch, Safetensors, Tensorflow 2, Tensorflow 1 checkpoints, ONNX.



BERTimbau Base is a pretrained BERT model for Brazilian Portuguese that achieves state-of-the-art performances on three downstream NLP tasks: Named Entity Recognition, Sentence Textual Similarity and Recognizing Textual Entailment. It is available in two sizes: Base and Large.

For further information, please direct to [BERTimbau repository](https://github.com/neuralmind-ai/portuguese-bert/).

## Available models

| Model                                        | Arch.      | #Layers | #Params |
| -------------------------------------------- | ---------- | ------- | ------- |
| `ai-agi/bert-base-portuguese-cased-finance`  | BERT-Large |   12    |  110M   |
| `ai-agi/bert-large-portuguese-cased-finance` | BERT-Large |   24    |  335M   |

## Usage

```python
from transformers import AutoTokenizer  # Or BertTokenizer
from transformers import AutoModelForPreTraining  # Or BertForPreTraining for loading pretraining heads
from transformers import AutoModel  # or BertModel, for BERT without pretraining heads

model = AutoModelForPreTraining.from_pretrained('ai-agi/bert-base-portuguese-cased-finance')
tokenizer = AutoTokenizer.from_pretrained('ai-agi/bert-base-portuguese-cased-finance', do_lower_case=False)
```

### Masked language modeling prediction example

```python
from transformers import pipeline

pipe = pipeline('fill-mask', model=model, tokenizer=tokenizer)

pipe('Tinha uma [MASK] no meio do caminho.')
# [{'score': 0.5054386258125305,
#   'sequence': '[CLS] Tinha uma pedra no meio do caminho. [SEP]',
#   'token': 5028,
#   'token_str': 'pedra'},
#  {'score': 0.05616172030568123,
#   'sequence': '[CLS] Tinha uma curva no meio do caminho. [SEP]',
#   'token': 9562,
#   'token_str': 'curva'},
#  {'score': 0.02348282001912594,
#   'sequence': '[CLS] Tinha uma parada no meio do caminho. [SEP]',
#   'token': 6655,
#   'token_str': 'parada'},
#  {'score': 0.01795753836631775,
#   'sequence': '[CLS] Tinha uma mulher no meio do caminho. [SEP]',
#   'token': 2606,
#   'token_str': 'mulher'},
#  {'score': 0.015246033668518066,
#   'sequence': '[CLS] Tinha uma luz no meio do caminho. [SEP]',
#   'token': 3377,
#   'token_str': 'luz'}]

```

### For BERT embeddings

```python

import torch

model = AutoModel.from_pretrained('ai-agi/bert-base-portuguese-cased-finance')
input_ids = tokenizer.encode('Tinha uma pedra no meio do caminho.', return_tensors='pt')

with torch.no_grad():
    outs = model(input_ids)
    encoded = outs[0][0, 1:-1]  # Ignore [CLS] and [SEP] special tokens

# encoded.shape: (8, 1024)
# tensor([[ 1.1872,  0.5606, -0.2264,  ...,  0.0117, -0.1618, -0.2286],
#         [ 1.3562,  0.1026,  0.1732,  ..., -0.3855, -0.0832, -0.1052],
#         [ 0.2988,  0.2528,  0.4431,  ...,  0.2684, -0.5584,  0.6524],
#         ...,
#         [ 0.3405, -0.0140, -0.0748,  ...,  0.6649, -0.8983,  0.5802],
#         [ 0.1011,  0.8782,  0.1545,  ..., -0.1768, -0.8880, -0.1095],
#         [ 0.7912,  0.9637, -0.3859,  ...,  0.2050, -0.1350,  0.0432]])