Table processing (#72)

* First commit

* Table processing in document layout

* Platform x86_64 check'

* PaddleOCR integrated

* Deactivate show_log in paddleocr

* Utilize layout updates

* Formatting and linting

* Correct how linting ignores are accumulated

* Removed fitz

* Bug fixed in intersect_rect

* Bump to default 200 dpi

* Updated README with instructions to install paddleocr

* Fixed typo

* Deal with empty case

* Formatting

* Updates to pass flake8

* Added table test

* Fixed test

* Typing changes

* formatting for large fixture

* Up pixel to reflect new dpi

* Make table extraction opt-in

* Change content to check for

* Add install targets for paddleocr

* Add optional pip install for paddleocr

* Update README.md

Updated paddleocr installation instructions

* Remove unused functions

* New image for table testing

* Remove non-unique assignment case

* Correct slot_into_contains arguments

* Test for nms

* update fixtures

* Added test for nms

* fix for disable table extraction by default

* Revised test

* Update old tests

* reuse postprocess

* Additional tests

* More rect tests, extract_text_from_spans

* Remove unused code

* Align supercells test

* Updated removal supercell test

* header_supercell_tree test

* name change to forked paddleocr

* Updated installation and removel of a print statement

* tidied file

* Version update

* linting

* Updated test

* Changed Makefile

---------

Co-authored-by: Antonio Jimeno Yepes <[email protected]>
Co-authored-by: Alan Bertl <[email protected]>

Author: ajjimeno

CHANGELOG.md

@@ -1,5 +1,6 @@
-## 0.2.13-dev0
+## 0.2.13
 
+* Add table processing
 * Change OCR logic to be aware of PDF image elements
 
 ## 0.2.12

Makefile

@@ -20,7 +20,7 @@ install-base: install-base-pip-packages
 install: install-base-pip-packages install-dev install-detectron2 install-test
 
 .PHONY: install-ci
-install-ci: install-base-pip-packages install-test
+install-ci: install-base-pip-packages install-test install-paddleocr
 
 .PHONY: install-base-pip-packages
 install-base-pip-packages:
@@ -31,6 +31,10 @@ install-base-pip-packages:
 install-detectron2:
 	pip install "detectron2@git+https://github.com/facebookresearch/detectron2.git@78d5b4f335005091fe0364ce4775d711ec93566e"
 
+.PHONY: install-paddleocr
+install-paddleocr:
+	pip install "unstructured.PaddleOCR"
+
 .PHONY: install-test
 install-test:
 	pip install -r requirements/test.txt

README.md

@@ -34,6 +34,17 @@ Windows is not officially supported by Detectron2, but some users are able to in
 See discussion [here](https://layout-parser.github.io/tutorials/installation#for-windows-users) for 
 tips on installing Detectron2 on Windows.
 
+### PaddleOCR
+
+[PaddleOCR](https://github.com/Unstructured-IO/unstructured.PaddleOCR) is required for table processing for `x86_64` architectures.
+It should not be installed under MacOS with Apple Silicon cpu.
+
+PaddleOCR should be installed using the following instructions.
+
+```shell
+pip install "unstructured.PaddleOCR"
+```
+
 ### Repository
 
 To install the repository for development, clone the repo and run `make install` to install dependencies.

sample-docs/example_table.jpg

@@ -3,7 +3,7 @@ license_files = LICENSE.md
 
 [flake8]
 max-line-length = 100
-ignore = D100, D101, D104, D105, D107, D2, D4
+extend-ignore = D100, D101, D104, D105, D107, D2, D4
 per-file-ignores =
     test_*/**: D
 

setup.cfg

@@ -18,6 +18,7 @@
 limitations under the License.
 """
 from setuptools import setup, find_packages
+from platform import machine
 
 from unstructured_inference.__version__ import __version__
 
@@ -60,5 +61,5 @@
         "onnxruntime",
         "transformers",
     ],
-    extras_require={},
+    extras_require={"paddle-ocr": "unstructured.PaddleOCR"},
 )

setup.py

@@ -186,11 +186,12 @@ def points(self):
 
 
 class MockPageLayout(layout.PageLayout):
-    def __init__(self, layout=None, model=None, ocr_strategy="auto"):
+    def __init__(self, layout=None, model=None, ocr_strategy="auto", extract_tables=False):
         self.image = None
         self.layout = layout
         self.model = model
         self.ocr_strategy = ocr_strategy
+        self.extract_tables = extract_tables
 
     def ocr(self, text_block: MockTextRegion):
         return text_block.ocr_text