Merge branch 'main' of hf.co:kirp/kosmos2_5 into main

README.md

@@ -1,50 +1,62 @@
 ---
-license: apache-2.0
+language: en
+license: mit
 ---
-# Test vesion
+# Under testing
+
+# Kosmos-2.5
+
+[Microsoft Document AI](https://www.microsoft.com/en-us/research/project/document-ai/) | [GitHub](https://github.com/microsoft/unilm/tree/master/kosmos-2.5)
+
+## Model description
+
+Kosmos-2.5 is a multimodal literate model for machine reading of text-intensive images. Pre-trained on large-scale text-intensive images, Kosmos-2.5 excels in two distinct yet cooperative transcription tasks: (1) generating spatially-aware text blocks, where each block of text is assigned its spatial coordinates within the image, and (2) producing structured text output that captures styles and structures into the markdown format. This unified multimodal literate capability is achieved through a shared decoder-only auto-regressive Transformer architecture, task-specific prompts, and flexible text representations. We evaluate Kosmos-2.5 on end-to-end document-level text recognition and image-to-markdown text generation. Furthermore, the model can be readily adapted for any text-intensive image understanding task with different prompts through supervised fine-tuning, making it a general-purpose tool for real-world applications involving text-rich images. This work also paves the way for the future scaling of multimodal large language models.
+
+[Kosmos-2.5: A Multimodal Literate Model](https://arxiv.org/abs/2309.11419)
+
+## NOTE:
+Since this is a generative model, there is a risk of **hallucination** during the generation process, and it **CAN NOT** guarantee the accuracy of all OCR/Markdown results in the images.
+
+## Use with transformers：
 ```bash
 pip install git+https://github.com/tic-top/transformers.git
 ```
-
 ```python
-from transformers import AutoModelForVision2Seq, AutoProcessor
 from PIL import Image
+import requests
 import torch
-device = "cuda:2"
+from transformers import AutoProcessor, AutoModelForVision2Seq
+import re
+
 repo = "kirp/kosmos2_5"
-dtype = torch.float16
-# dtype = torch.bfloat16
-model = AutoModelForVision2Seq.from_pretrained(repo, device_map = device, torch_dtype=dtype)
-# print(model)
-# exit(0)
+device = "cuda:0"
+dtype = torch.bfloat16
+model = AutoModelForVision2Seq.from_pretrained(repo, device_map=device, torch_dtype=dtype)
 processor = AutoProcessor.from_pretrained(repo)
 
-path = "receipt_00008.png"
-image = Image.open(path)
-prompt = "<ocr>" 
-# prompt = "<md>"
-inputs = processor(text=prompt, images=image, return_tensors="pt", max_patches=4096)
+url = "https://huggingface.co/kirp/kosmos2_5/resolve/main/receipt_00008.png"
+image = Image.open(requests.get(url, stream=True).raw)
+prompt = "<ocr>" # <md>
 
-raw_width, raw_height = image.size
+inputs = processor(text=prompt, images=image, return_tensors="pt")
 height, width = inputs.pop("height"), inputs.pop("width")
+raw_width, raw_height = image.size
 scale_height = raw_height / height
 scale_width = raw_width / width
 
 inputs = {k: v.to(device) if v is not None else None for k, v in inputs.items()}
 inputs["flattened_patches"] = inputs["flattened_patches"].to(dtype)
-with torch.no_grad():
-    generated_text = model.generate(**inputs, max_length=4096)
-generated_text = processor.batch_decode(generated_text)
-import re, os
-def postprocess(y, scale_height, scale_width, result_path=None):
-    y = (
-        y.replace("<s>", "")
-        .replace("</s>", "")
-        .replace("<image>", "")
-        .replace("</image>", "")
-        .replace(prompt, "")
-    )
-    print(y)
+
+generated_ids = model.generate(
+    **inputs,
+    max_new_tokens=1024,
+)
+generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)
+
+def postprocess(y, scale_height, scale_width):
+    y = y.replace(prompt, "")
+    if "<md>" in prompt:
+        return y
     pattern = r"<bbox><x_\d+><y_\d+><x_\d+><y_\d+></bbox>"
     bboxs_raw = re.findall(pattern, y)
     lines = re.split(pattern, y)[1:]
@@ -53,24 +65,54 @@ def postprocess(y, scale_height, scale_width, result_path=None):
     info = ""
     for i in range(len(lines)):
         box = bboxs[i]
-        # do we need to convert the size of the box?
-        # maybe yes
         x0, y0, x1, y1 = box
-        # maybe modify the order
         if not (x0 >= x1 or y0 >= y1):
             x0 = int(x0 * scale_width)
             y0 = int(y0 * scale_height)
             x1 = int(x1 * scale_width)
             y1 = int(y1 * scale_height)
             info += f"{x0},{y0},{x1},{y0},{x1},{y1},{x0},{y1},{lines[i]}"
-    
-    if result_path is not None:
-        os.makedirs(os.path.dirname(result_path), exist_ok=True)
-        # create and write in utf-8
-        with open(result_path, "w", encoding="utf-8") as f:
-            f.write(info)
-    else:
-        print(info)
-
-postprocess(generated_text[0], scale_height, scale_width)
-```
+    return info
+
+output_text = postprocess(generated_text[0], scale_height, scale_width)
+print(output_text)
+```
+```text
+55,595,71,595,71,629,55,629,1
+82,595,481,595,481,635,82,635,[REG] BLACK SAKURA
+716,590,841,590,841,629,716,629,45,455
+55,637,71,637,71,672,55,672,1
+82,637,486,637,486,675,82,675,COOKIE DOH SAUCES
+818,632,843,632,843,668,818,668,0
+51,683,71,683,71,719,51,719,1
+82,683,371,683,371,719,82,719,NATA DE COCO
+820,677,845,677,845,713,820,713,0
+32,770,851,770,851,811,32,811,Sub Total 45,455
+28,811,853,811,853,858,28,858,PB1 (10%) 4,545
+28,857,855,857,855,905,28,905,Rounding 0
+24,905,858,905,858,956,24,956,Total 50,000
+17,1096,868,1096,868,1150,17,1150,Card Payment 50,000
+```
+
+
+
+## Citation
+
+If you find Kosmos-2.5 useful in your research, please cite the following paper:
+
+```
+@article{lv2023kosmos,
+  title={Kosmos-2.5: A multimodal literate model},
+  author={Lv, Tengchao and Huang, Yupan and Chen, Jingye and Cui, Lei and Ma, Shuming and Chang, Yaoyao and Huang, Shaohan and Wang, Wenhui and Dong, Li and Luo, Weiyao and others},
+  journal={arXiv preprint arXiv:2309.11419},
+  year={2023}
+}
+```
+
+## License
+The content of this project itself is licensed under the [MIT](https://github.com/microsoft/unilm/blob/master/kosmos-2.5/LICENSE)
+
+[Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct)
+
+
+