Merge pull request #39 from Probesys/calc

* Prepartion to upgrade to debian 13 and library upgrades

* Add usage of Calc (ods, xlsx, ..) in lotemplate
Add the possibility to launch multiple libreoffice and to cleanly loadbalance
between them (see doc)

* now not dic for json

* create unittest for  calc

* refactor code for starting multi office and select random connexion

* readd restart in case libreoffice died unexpectedly

* add exception for tmpfile

* fix import of function problem

* ADD possibity of looping on table

* ADD handle of deprecetion of dict in json

* Correction of starting libreoffice

* Add calc tests

* now use jsondiff to compare json  + bug correction in Calc Statement

* add cache system for json plus unitest for it

* refactor code for cache system

* add ruff linter in CI

* fixed code that was not closing the document when necessary and add function in cli and api for stat and clean wrong open lo doc

* fix doc cli parameters

* bugfix in the API when missing key in json

* MAXTIME default value

* fix bugs on error messages with the API

* Change the syntax of the named ranges : loop_down_xxx

* add usebruno tool

* fix tests after renaming table_r/d by loop_down/right

* Documentation

* fix ruff

---------

Co-authored-by: Philippe Le Van and Cyr

Author: philippe-levan

.dockerignore

@@ -0,0 +1,9 @@
+/usebruno
+/venv
+/uploads
+/exports
+/.vscode
+/.local
+/.github
+/.config
+/.cache

.env

@@ -1 +1,3 @@
-SECRET_KEY=DEFAULT_KEY
+NB_WORKERS=6
+MAXTIME=120
+SECRET_KEY=DEFAULT_KEY

.github/workflows/ruff.yml

@@ -0,0 +1,8 @@
+name: Ruff
+on: [ push, pull_request ]
+jobs:
+  ruff:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: astral-sh/ruff-action@v3

.github/workflows/unittest.yml

@@ -8,7 +8,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
     - name: Checkout
-      uses: actions/checkout@v3
+      uses: actions/checkout@v4
       with:
         fetch-depth: 0
 

.gitignore

@@ -1,5 +1,7 @@
 __pycache__
 .bash_history
+*.swp
+.viminfo
 /.cache/
 /.config/
 .local
@@ -10,16 +12,21 @@ uploads/
 .fleet/
 .vscode/
 .~lock.*#
+.python_history
 exports/
 /config*
 /lotemplate/unittest/files/content/*.unittest.txt
 /lotemplate/unittest/files/content/*.unittest.odt
+/lotemplate/unittest/files/content/*.unittest.html
 /lotemplate/unittest/files/content/debug.docx
 /lotemplate/unittest/files/content/debug.odt
 /lotemplate/unittest/files/content/debug.json
 /lotemplate/unittest/files/content/*.unittest.odt
 /lotemplate/unittest/files/content/*.unittest.pdf
 /venv
+/lotemplate/unittest/files/content/e89fbedb61af3994184da3e5340bd9e9-calc_variables.ods.json
 /output*.*
 .fontconfig/
 docker-compose.override.yml
+/tmpfile/**
+!/tmpfile/.keep

API/utils.py

@@ -2,51 +2,40 @@
 Copyright (C) 2023 Probesys
 """
 
-from flask import *
+from flask import Response, send_file
 
 import lotemplate as ot
 
-import configargparse as cparse
 import glob
 import os
 import sys
-import subprocess
-from time import sleep
 from typing import Union
-from zipfile import ZipFile
-
-p = cparse.ArgumentParser(default_config_files=['config.yml', 'config.ini', 'config'])
-p.add_argument('--config', '-c', is_config_file=True, help='Configuration file path')
-p.add_argument('--host', default="localhost", help='Host address to use for the libreoffice connection')
-p.add_argument('--port', default="2002", help='Port to use for the libreoffice connexion')
-args = p.parse_known_args()[0]
-
-os.makedirs("uploads", exist_ok=True)
-os.makedirs("exports", exist_ok=True)
-subprocess.call(
-    f'soffice "--accept=socket,host={args.host},port={args.port};urp;StarOffice.ServiceManager" &', shell=True)
-sleep(3)
-cnx = ot.Connexion(args.host, args.port)
 
+host='localhost'
+port='200'
+gworkers=0
+scannedjson=''
+maxtime=60
+def start_soffice(workers,jsondir,maxt=60):
+    global gworkers
+    global my_lo
+    global scannedjson
+    global maxtime
+    maxtime=maxt
+    scannedjson=jsondir
+    gworkers=workers
+    os.makedirs("uploads", exist_ok=True)
+    os.makedirs("exports", exist_ok=True)
+    os.makedirs(scannedjson, exist_ok=True)
+    clean_temp_files()
+    my_lo=ot.start_multi_office(nb_env=workers)
 
-def restart_soffice() -> None:
-    """
-    simply restart the soffice process
 
-    :return: None
-    """
-
-    clean_temp_files()
-    subprocess.call(
-        f'soffice "--accept=socket,host={cnx.host},port={cnx.port};urp;StarOffice.ServiceManager" &',
-        shell=True
-    )
-    sleep(2)
-    try:
-        cnx.restart()
-    except:
-        pass
+def connexion():
+   global my_lo 
+   cnx= ot.randomConnexion(my_lo)
 
+   return cnx
 
 def clean_temp_files():
     """
@@ -103,7 +92,6 @@ def error_format(exception: Exception, message: str = None) -> dict:
         {
             'error': type(exception).__name__,
             'code': exception.code if isinstance(exception, ot.errors.LotemplateError) else type(exception).__name__,
-            # 'message': message or str(exception),
             'message': message or exception_message,
             'variables': exception.infos if isinstance(exception, ot.errors.LotemplateError) else {}
         }
@@ -148,23 +136,16 @@ def save_file(directory: str, f, name: str, error_caught=False) -> Union[tuple[d
         i += 1
     f.stream.seek(0)
     f.save(f"uploads/{directory}/{name}")
+
+
+    cnx = connexion()
+    global scannedjson
     try:
-        with ot.Template(f"uploads/{directory}/{name}", cnx, True) as temp:
+        with ot.TemplateFromExt(f"uploads/{directory}/{name}", cnx, True,scannedjson) as temp:
             values = temp.variables
     except ot.errors.TemplateError as e:
         delete_file(directory, name)
         return error_format(e), 415
-    except ot.errors.UnoException as e:
-        delete_file(directory, name)
-        restart_soffice()
-        if error_caught:
-            return (
-                error_format(e, "Internal server error on file opening. Please checks the README file, section "
-                                "'Unsolvable problems' for more informations."),
-                500
-            )
-        else:
-            return save_file(directory, f, name, True)
     except Exception as e:
         delete_file(directory, name)
         return error_format(e), 500
@@ -180,24 +161,14 @@ def scan_file(directory: str, file: str, error_caught=False) -> Union[tuple[dict
     :param error_caught: specify if an error was already caught
     :return: a json and optionally an int which represent the status code to return
     """
-
-    try:
-        with ot.Template(f"uploads/{directory}/{file}", cnx, True) as temp:
+    cnx = connexion()
+    global scannedjson
+    with ot.TemplateFromExt(f"uploads/{directory}/{file}", cnx, True,scannedjson) as temp:
             variables = temp.variables
-    except ot.errors.UnoException as e:
-        restart_soffice()
-        if error_caught:
-            return (
-                error_format(e, "Internal server error on file opening. Please checks the README file, section "
-                                "'Unsolvable problems' for more informations."),
-                500
-            )
-        else:
-            return scan_file(directory, file, True)
     return {'file': file, 'message': "Successfully scanned", 'variables': variables}
 
 
-def fill_file(directory: str, file: str, json, error_caught=False) -> Union[tuple[dict, int], dict, Response]:
+def fill_file(directory: str, file: str, json, error_caught=False) -> Union[tuple[dict, int], dict, tuple[str,Response]]:
     """
     fill the specified file
 
@@ -208,61 +179,48 @@ def fill_file(directory: str, file: str, json, error_caught=False) -> Union[tupl
     :return: a json and optionally an int which represent the status code to return
     """
 
-    if type(json) != list or not json:
-        return error_sim("JsonSyntaxError", 'api_invalid_base_value_type', "The json should be a non-empty array"), 415
-
+    if  isinstance(json, list):
+        json=json[0]
+        print("####\nUsing a list of dict is DEPRECATED, you must directly send the dict.")
+        print("See documentation.\n#######")
+    cnx = connexion()
+    global scannedjson
     try:
-        with ot.Template(f"uploads/{directory}/{file}", cnx, True) as temp:
-
-            exports = []
-
-            for elem in json:
-
-                length = len(elem)
-                is_name_present = type(elem.get("name")) is str
-                is_variables_present = type(elem.get("variables")) is dict
-                is_page_break_present = type(elem.get("page_break")) is bool
-
-                if (
-                        not is_name_present
-                        or not is_variables_present
-                        or ((length > 2 and not is_page_break_present) or (length > 3 and is_page_break_present))
-                ):
-                    return error_sim(
-                        "JsonSyntaxError",
-                        'api_invalid_instance_syntax',
-                        "Each instance of the array in the json should be an object containing only 'name' - "
-                        "a non-empty string, 'variables' - a non-empty object, and, optionally, 'page_break' - "
-                        "a boolean."), 415
-
-                try:
-                    json_variables = ot.convert_to_datas_template(elem["variables"])
-                    temp.search_error(json_variables)
-                    temp.fill(elem["variables"])
-                    if elem.get('page_break', False):
-                        temp.page_break()
-                    exports.append(temp.export("exports/" + elem["name"], should_replace=(
-                        True if len(json) == 1 else False)))
-                except Exception as e:
-                    return error_format(e), 415
-
-            if len(exports) == 1:
-                return send_file(exports[0], download_name=exports[0].split("/")[-1])
-            else:
-                with ZipFile('exports/export.zip', 'w') as zipped:
-                    for elem2 in exports:
-                        zipped.write(elem2, elem2.split("/")[-1])
-                return send_file('exports/export.zip', 'export.zip')
-    except ot.errors.UnoException as e:
-        restart_soffice()
-        if error_caught:
-            return (
-                error_format(e, "Internal server error on file opening. Please checks the README file, section "
-                                "'Unsolvable problems' for more informations."),
-                500
-            )
-        else:
-            return fill_file(directory, file, json, True)
-
-
-clean_temp_files()
+        with ot.TemplateFromExt(f"uploads/{directory}/{file}", cnx, True,scannedjson) as temp:
+
+            length = len(json)
+            is_name_present = type(json.get("name")) is str
+            is_variables_present = type(json.get("variables")) is dict
+            is_page_break_present = type(json.get("page_break")) is bool
+
+            if (
+                    not is_name_present
+                    or not is_variables_present
+                    or ((length > 2 and not is_page_break_present) or (length > 3 and is_page_break_present))
+            ):
+                return 415, error_sim(
+                    "JsonSyntaxError",
+                    'api_invalid_instance_syntax',
+                    "Each instance of the array in the json should be an object containing only 'name' - "
+                    "a non-empty string, 'variables' - a non-empty object, and, optionally, 'page_break' - "
+                    "a boolean.")
+
+            try:
+                json_variables = ot.convert_to_datas_template(json["variables"])
+                temp.search_error(json_variables)
+                temp.fill(json["variables"])
+                if json.get('page_break', False):
+                    temp.page_break()
+                export_file=temp.export(json["name"],"exports")
+                export_name=json["name"]
+            except Exception as e:
+                if 'export_name' in locals():
+                    return ( export_file,error_format(e))
+                else:
+                    return ( "nofile",error_format(e))
+
+            return (export_file,send_file(export_file, export_name))
+
+    except Exception as e:
+            return error_format(e), 500
+

CHANGELOG.md

@@ -0,0 +1,75 @@
+Versions
+========
+
+Note : the upgrade from version 1.x to 2.x is easy. There is no reason to stay to version 1.x.
+
+The upgrade documentation is in the file [UPGRADE.md](UPGRADE.md).
+
+Versions 2.x
+------------
+
+- v2.0.0 : 01/01/2025 
+  - BC Break (easy to fix) : see [UPGRADE.md](UPGRADE.md)
+  - We can now generate Calc / Excel files (from Calc templates)
+  - Is multiThreaded : we can generate several files at the same time
+  - Performances improvements
+  - No BC Breaks for the templates
+  - upgrade debian, LibreOffice, Python libs versions
+  - for devs : added "use bruno" requests inside the repository
+
+Versions 1.x
+------------
+
+- v1.6.1 : 2024-04-12 : bugfix
+  - fix the issue https://github.com/Probesys/lotemplate/issues/34 : too many endif bugg
+- v1.6.0 : 2024-04-11
+  - allow put variables inside headers and footers
+  - fix a bug when a variable is both inside the text content and inside a table (it should not arrive, but it is fixed)
+  - a new unit test system based on PDF converted to text in order to test contents that are not converted to text with a simple saveAs 
+- v1.5.2 : 2024-02-24 : Better README
+  - Rewrite for a betterdocker DockerFile without bug 
+- v1.5.1 : 2024-02-16 : Better README
+  - Rewriting of the README file
+- v1.5.0 : 2024-02-12 : syntax error detection
+  - add syntax error detection in if statements
+  - add syntax error detection in for statements
+  - come back to default libreoffice of Debian Bookworm (removed backports, incompatibility)
+- v1.4.1 : 2023-11-20 : micro-feature for counter and fix possible bug
+  - use counters for counting elements of a list
+  - fix possible bug with reset and last.
+- v1.4.0, 2023-11-17 : counters
+  - add a counter system inside templates
+  - add better scan for if statement. Raises an error if there is too many endif in the template.
+  - speedup html statement replacement and scanning
+  - speedup for statement replacement and scanning
+  - tests of for scanning
+  - internal : add scan testing inside content unit tests
+- v1.3.0, 2023-11-16 :
+  - major refactoring. No evolution for the user.
+  - new unit tests on tables and images
+  - no BC Break (theoretically)
+- v1.2.8, 2023-09-01 :
+  - fix bug in TextShape var replacement
+- v1.2.7, 2023-08-30 :
+  - Upgrade to debian bookworm slim
+- v1.2.6, 2023-08-30 :
+  - new comparators for if statements : ===, !==, CONTAINS, NOT_CONTAINS
+  - variables of type "html" are now supported and copied as HTML
+- v1.2.5, 2023-07-17 : temporary fix for detecting endhtml and endfor
+- v1.2.4, 2023-07-09 : fix major bug in if statement scanning
+- v1.2.3, 2023-07-07 : no endif detection, performance improvement in if statement
+- v1.2.2, 2023-06-09 : bugfix html statement scan missing
+- v1.2.1, 2023-06-05 : little fix for CI
+- v1.2.0, 2023-06-04 : if statements inside for
+- v1.1.0, 2023-05-23 : recursive if statement
+- v1.0.1, 2023-05-05 : workaround, fix in html formatting
+- v1.0.0, 2023-05-03 : if statement, for statement, html statement
+- not numbered : about may 2022 : first version
+
+### Possible futur evolutions
+
+- Possibly to add dynamic images in tables
+- another way to make image variables that would be compatible with Microsoft Word and maybe other formats (example : set the variable name in the 'alternative text' field)
+- key system for each institution for security
+- handle bulleted lists using table like variables
+- use variable formatting instead of the one of the character before

Dockerfile

@@ -1,4 +1,7 @@
-FROM debian:bookworm-slim as prod
+FROM debian:trixie-slim as prod
+
+RUN rm -f /etc/apt/apt.conf.d/docker-clean; echo 'Binary::apt::APT::Keep-Downloaded-Packages "true";' > /etc/apt/apt.conf.d/keep-cache
+
 RUN --mount=type=cache,id=apt-cache,target=/var/cache/apt,sharing=locked \
     --mount=type=cache,id=apt-lib,target=/var/lib/apt,sharing=locked \
     --mount=type=cache,id=debconf,target=/var/cache/debconf,sharing=locked \