- enable tar compression for bigger files
- All notable changes to this project will be documented in this file.
- jacoco aggregate
- fix : webservice table images were deleted by batch in multinod mode
- mapping vertical optimization
- better balance of file batching between the nods available
- mapping optimization
- wal_segment_size to 1GB in postgres executor stateful kubernetes configuration
- mapping optimization
- removed deprecated methods in UtilitaireDao
- if the ARC process must export the files to the master nod, the data retrieval webservice will only select the files marked as EXPORTED
- revert vertical scaling on mapping phase
- unicity on structurize phase now sort data if case of duplicate : first record is picked
- performance. Vertical scaling for the operation that maps data to model table. For each model data table, one data insertion thread is created to lower the issue of having many model data tables.
- ephemeral pvc for temporary tablespace
- added jacoco aggregate for sonar accuracy
- non beta version
- postgres 18 support
- duckdb upgrade version 1.4
- minio upgrade version 1.6.0
- new parameter stop_on_backup to stop batch when an hot database backup has started. It checks the archive command to see if archive are disabled or redirected to bin. Production is stopped when archive mode is enabled ans not redirected to bin.
- spring security 6.5.4
- spring security 6.5.4
- export phase update
- when the export phase is activated, users may now set the target of mapping tables in the export data menu
- default export targets of mapping tables are parquet files
- user can also set target the the master nod
- mapping table data are copied from executors nod to master nod during the export phase
- changed postgres docker image from bitnami to standart postgres
- fix the non-copying of type_consolidation when importing family rules
- minio client version
- security fix
- minimal csp policy
- changed uploads because of tomcat 10.1.43 multipart files changes
- security fix
- most of the process queries now don't use autocommit in order to avoid database connection locks
- fix a crash bug when stopping and resuming batch pipeline
- quality
- added a configurable emptyDir usable to set a temporary tablespace for the kubernetes postgres executors
- set KUBERNETES_EXECUTOR_TEMPTABLESPACE_MEDIUM to "Memory"for a memory drive
- set KUBERNETES_EXECUTOR_TEMPTABLESPACE_MEDIUM to "" for a normal ephemeral drive
- spring security critical upgrade
- webjar for js dependencies
- fix in multiphase api : temporary data are deleted when no longer used
- new multiphase api : file are now fully proceed in separate threads
- fix filesystem maintenance problem when files no longer exists
- removed hotfix version-94.1.104
- spring security and zonky io maven dependencies upgrade
- hotfix : check if an instance of arc batch is already running
- sandbox list retrieved by data retrieval webservice are stored in mod_environnement_arc entity
- new library to read tar.gz as apache one throws crc32 errors randomly
- the data retrieval webservice now gives the sandbox meta table in metadata
- optimisation for pilotage dead tuples maintenance
- optimization : keep intermediate columns in gui batch mode
- optimization : single batch connection and vacuum strategy
- optimization : single batch connection
- fix bug in data retieval webservice
- fix bug in data retieval webservice
- fix java null pointer exception when crc error on reader
- fix parquet file export memory leak
- download data to s3 gui button works again
- if s3 is active, files which have triggered batch error are sent to s3
- fix executor ephemral pvc
- ignore independance rules when child tag doesn't exist
- add priority rules to sort incoming files
- common.io security version update
- postgres scanner version upgrade
- bug fix to catch when parquet export fail in order to interrupt batch correctly
- spring security update
- datawarehouse and archives inside datawarehouse can be loaded according to an order defined by user. Check "entrepot" gui.
- add kubernetes ephemeral volume management for executor pods
- fix web service not waiting for large data to be created
- spring update
- parquet support for data retrieval webservice
- enhanced logs for data retrieval webservice
- clear parquet export directory if s3 output bucket is defined
- jwt authentification for data retrieval webservice if keycloak is set
- user_restricted right fix
- pvc is deleted in volatile mode after batch ends
- fix file action (delete, replay, archive replay)
- fix batch exit when production is set to off
- fix the total file size limiter used by reception phase in batch mode
- Add a new database parameter LanceurIHM.tailleMaxReceptionEnMb to limit the total file size to be received in gui mode.
- bug fix for deirectory creation in kubernetes pvc
- add parameters KUBERNETES_EXECUTOR_CPU, KUBERNETES_EXECUTOR_RAM
- file retention period can be in set fr.insee.arc.files.retention.days
- KO will now be deleted from database after data retention period
- rework external table column red for file to be valid in database
- bug fix on parquet encryption key
- reduce privilege level for kubernetes executor databases
- sql constraints on the table parameter
- debug gui access can now be set in a distinct keycloak group (fr.insee.gui.debug.disable)
- exception handling for access denied
- docker images now have version. They are deployed through the "released on tag" workflow and handle same version number.
- rest webservice securisation for injection
- gui database securisation for injection (add gui constraints for all rules)
- new gui generic error message
- docker add : parquet encryption key for minio s3 export
- docker add : keycloak
- checkmarx fix
- databases ip caching to avoid dsn spam in batch mode
- deprecated spring security fix (rest webservice and web)
- action are now logged with idep
- ARC batch exports data to parquet (when fr.insee.arc.process.export property is not empty). Default is off
- ARC batch pops, uses and removes executor database (when fr.insee.arc.kubernetes.executor.volatile is not empty). Default is off
- bug fix for gz archive entry name
- parquet export
- bug fix with filter gui
- gui bug fix : import rules and update comment in nomenclature screen
- tomcat fix as a library was missing in new version of tomcat 10.8
- removed webservice entrypoint /execute/engine/{serviceName}/{serviceId}/{bas} as it looks unused
- webservice data retrieval servlet access fix
- kubernetess support
- disablegui property now working
- tomcat 10, java 17, spring 6
- fix : ws import wait for large data tables
- last version in tomcat 9, java 11
- avoid overflow when large file data are set on a single line
- batch failed with functional warning
- upload mapping rules now the model declared variables
- scalable data retrieval webservice
- csv_gzip format is available for the data retrieval webservice
- dependencies security fix
- fix performance issue in to_delete reception query
- bind variable in gererFamille
- production version
- deployment for version qualification
- initialization optimization
- the webservice for data retrieval deletes the client pending data tables at the start of a new invoke
- the webservice for data retrieval acts creates data table asynchronously to avoid timeout problems
- functionnal tests
- pg 14.8 bug fix
- bug fix in gui file report
- scalability connection management
- passage en production
- version beta avec suppression de la phase de filtrage
- github workflow java 11
- show sand box informations at the home page
- loader performance enhancement
- gui quality refactor
- wrong xsd location in webservice configuration
- unit test for technical and core class (part 1)
- upload data file via GUI is fixed to match with new security constraints
- web health-check is now public
- bug fix in normage view actions
- bug fix FileUtils.cleanDirectory with symbolic lock
- bug fix in duplicate detection
- gui refactor
- parallelism refactor
- healthcheck quality refactor and add-on for web module
- DDI model implementation
- bug fix in maintenance screen
- The batch logger default level was incorrect and fixed
- code quality
- secured the data retrieval service (https + whitelist)
- normage optimization for cartesian reduction with new independance rules
- model metadata can now be exported
- esane test for ci/cd
- deliverable name fix for internal ci/cd tested
- urls for hello, healthcheck and version pages are now public access
- batch optimization : better queuing for files processing to avoid downtime
- bug fix to download database output from the gui
- button to apply directly rules in production environments
- optimization of cartesian reduction algorithm
- spring and springboot security fix
- multiselection on export gui
- button to apply directly rules in production environments
- optimization of cartesian reduction algorithm
- spring and springboot security fix
- multiselection on export gui
- button to apply directly rules in production environments
- optimization of cartesian reduction algorithm
- add exclusion rules on structurize phase
- model tablename can now contain digits
- postgres driver security bump
- spring security bump
- h2 security fix
- git informations retrieval bug fix for batch
- fix version display message for batch
- revert phase ihm bugfix for reception phase
- files timestamp fix
- log4j 2.17.1
- log4j 2.17.0
- database h2 security fix
- fix #2 for log4j security jndi injection problem
- git-commit-id-plugin depandency update to get the tag number correctly
- sonar security fix
- for security purpose, ihm has now distinct public page index and a secured index
- keycloak.resource=keycloak.json settings is now default. ARC now detects if keycloak is activated by scanning the "realm" and "auth-server-url" value. No changes required for current installations
- log4j security fix
- the data retrieval webservice can now be used to retrieve specific data types ("mapping", "nomenclature", "metadata")
- maintenance parameters GUI to set the way ARC works is now available
- archive reader now works for tar.gz archives that contain several directories
- bug fix in user management interface due to a null attribute
- database and application version now directly picked from git metadata
- tabs in gui are now persitant
- new maintenance menu to set parameters and test loggers
- bugfixes in gui
- fixes bug preventing writing log to file (web-service)
- adds new property fr.insee.arc.log.level to configure ARC log level
- allows logging to file through property
- extends expressions to control rules
- fixes multithread bug on SimpleDateFormat causing NPE in mapping
- adds /healthcheck to webservice
- improves error message on key-value loading failing
- adds clarifying report to files marked KO because one other file in the archive could not be read