Skip to content

OP-1084 Make CSV export an RFC-4180 open format and offer it in the export dialog - #1617

Open
giuseppesorge wants to merge 1 commit into
informatici:developfrom
giuseppesorge:OP-1084-csv-open-format
Open

OP-1084 Make CSV export an RFC-4180 open format and offer it in the export dialog#1617
giuseppesorge wants to merge 1 commit into
informatici:developfrom
giuseppesorge:OP-1084-csv-open-format

Conversation

@giuseppesorge

Copy link
Copy Markdown
Contributor

OP-1084 — Export to an open file format (CSV)

JIRA: https://openhospital.atlassian.net/browse/OP-1084

Problem

Every user-facing data export currently produces only a proprietary Microsoft format (.xlsx via POI XSSF, or .xls via HSSF). The shared export file chooser (ExcelExporter.getJFileChooserExcel) offered no open format. The class already contained three exportToCSV methods, but they were unused/unwired and not RFC 4180 compliant:

  • semicolon separator with no quoting/escaping — a field containing ;, " or a line break corrupted the file;
  • inconsistent encoding — two methods wrote UTF-8 (+BOM), while exportDataToCSV used new FileWriter(...) (platform default charset);
  • the resultset writer dropped the separator before the last column and emitted a trailing one; exportDataToCSV emitted a trailing separator on every row.

Change (core)

  • Hardened the three CSV writers to RFC 4180: a shared escapeCSV (quote a field that contains the separator, a ", CR or LF; double any embedded "), shared writeCSVRow/formatValue, consistent UTF-8 with BOM, CRLF row terminator. Integral ids/counts are emitted verbatim (no locale grouping, so a code 12345 stays 12345, not 12,345); decimals use the locale format with grouping disabled.
  • Added a CSV - open format (*.csv) filter to getJFileChooserExcel, selected by default, and disabled the accept-all filter (this also removes a latent ClassCastException on the (FileNameExtensionFilter) getFileFilter() cast used by the report exporters).
  • Added dispatch helpers exportTable(JTable, File[, int]) and exportResultset(ResultSet, File) that select the format from the file extension.
  • JasperReportsManager now delegates its six report-export branches to exportResultset (which also handles .csv).
  • Strengthened TestExcelExporter with content/format assertions (BOM, separator, RFC 4180 quoting, no trailing separator, extension dispatch).

Notes for review

  • CSV is set as the default filter in the save dialog — a deliberate, user-visible change, in line with the GDPR open-format goal of this ticket. It is trivially reversible (one line: setFileFilter(csvFilter)excelFilter) if you would rather Excel stayed the default with CSV merely available.
  • Dates are left as ISO (java.sql.Date.toString()), which is more interoperable than the dd/MM/yyyy used for timestamps; not changed here to keep the diff focused.
  • This is the prerequisite for OP-1324 ("remove unused export-to-CSV code").

Verification

  • TestExcelExporter 12/12 and stat.Tests 4/4 green; companion GUI mvn package green.
  • Real run against a MariaDB demo database (415 pharmaceuticals): Pharmacy → Pharmaceuticals → Export defaults to the CSV filter; the produced file is valid UTF-8+BOM, CRLF, ;-separated, with no trailing separator, integer codes without grouping, special characters preserved, and fields quoted only when needed.

…xport dialog

Harden the three CSV writers in ExcelExporter to produce RFC-4180 output: quote/escape fields that contain the separator, a double quote or a line break; consistent UTF-8 BOM; CRLF row terminator. Fix latent bugs along the way - the resultset writer dropped the separator before the last column and emitted a trailing one, and exportDataToCSV wrote with the platform default charset and a trailing separator. Integral ids/counts are no longer run through locale grouping (e.g. 12345, not 12,345).

Add a 'CSV - open format' filter to the export file chooser (selected by default) and disable the accept-all filter; add exportTable/exportResultset dispatch helpers that pick the format from the file extension. JasperReportsManager now delegates report export to exportResultset, which also handles .csv.

Strengthen TestExcelExporter with content/format assertions (BOM, separator, RFC-4180 quoting, no trailing separator, extension dispatch).
@giuseppesorge

Copy link
Copy Markdown
Contributor Author

Related PRs (same branch OP-1084-csv-open-format):

JIRA: https://openhospital.atlassian.net/browse/OP-1084

@dbmalkovsky one design call I'd like your steer on: I set the CSV ("open format") filter as the default in the export Save dialog (Excel stays available), since that matches the GDPR open-format intent of the ticket. If you'd rather keep Excel as the default with CSV merely offered, it's a one-line change — happy to flip it either way.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant