Skip to content

refactor: Replace Gzip compression with LZ4 in file handling #1306

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
OmarBrbutovic wants to merge 1 commit into
base: main
Choose a base branch
from
+25 −26
Draft

refactor: Replace Gzip compression with LZ4 in file handling #1306

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 14 additions & 7 deletions frontend/package-lock.json
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 2 additions & 1 deletion frontend/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@
"@types/js-cookie": "^3.0.6",
"@types/leaflet": "^1.9.12",
"@types/lodash": "^4.17.6",
"@types/lz4js": "^0.2.2",
"@types/react": "^18.3.3",
"@types/react-dom": "^18.3.0",
"@types/react-relay": "^18.2.1",
Expand All @@ -34,13 +35,13 @@
"class-variance-authority": "^0.7.1",
"clsx": "^2.1.1",
"dayjs": "^1.11.19",
"fflate": "^0.8.2",
"graphql": "^16.9.0",
"history": "^5.1.0",
"js-cookie": "^3.0.5",
"leaflet": "^1.9.4",
"lodash": "^4.17.21",
"lucide-react": "^0.577.0",
"lz4js": "^0.2.0",
"react": "^18.3.1",
"react-apexcharts": "^2.1.0",
"react-bootstrap": "^2.10.4",
Expand Down
27 changes: 9 additions & 18 deletions frontend/src/lib/files.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@
* SPDX-License-Identifier: Apache-2.0
*/

import { gzip } from "fflate";
import { compress } from "lz4js";

const BLOCK_SIZE = 512;

Expand Down Expand Up @@ -112,15 +112,6 @@ const createTarArchive = async (files: File[]): Promise<Uint8Array> => {
const nameBytes = encoder.encode(fileName);
header.set(nameBytes.slice(0, 100), 0);

// File mode (offset 100, 8 bytes) - 0000644
header.set(encoder.encode("0000644\0"), 100);

// Owner ID (offset 108, 8 bytes) - 0000000
header.set(encoder.encode("0000000\0"), 108);

// Group ID (offset 116, 8 bytes) - 0000000
header.set(encoder.encode("0000000\0"), 116);

// File size in octal (offset 124, 12 bytes)
const sizeOctal = fileData.length.toString(8).padStart(11, "0");
header.set(encoder.encode(sizeOctal + "\0"), 124);
Expand Down Expand Up @@ -176,18 +167,18 @@ const createTarArchive = async (files: File[]): Promise<Uint8Array> => {
return tarData;
};

// Creates a tar.gz archive from multiple files.
// Creates an LZ4 compressed tar archive from multiple files.
// Returns a Blob of the compressed archive.
const createTarGzArchive = async (files: File[]): Promise<Blob> => {
Copy link
Copy Markdown
Contributor

@osmanhadzic osmanhadzic Mar 26, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
const createTarGzArchive = async (files: File[]): Promise<Blob> => {
const createTarLz4Archive = async (files: File[]): Promise<Blob> => {

const tarData = await createTarArchive(files);
const gzippedData = await new Promise<Uint8Array>((resolve, reject) => {
gzip(tarData, (err, data) => {
if (err) reject(err);
else resolve(data);
});

// Run compression on a later tick so callers can await it asynchronously.
const lz4Data = await new Promise<Uint8Array>((resolve) => {
setTimeout(() => resolve(compress(tarData)), 0);
Copy link
Copy Markdown
Contributor

@osmanhadzic osmanhadzic Mar 26, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove the setTimeout wrapper (it doesn’t make compression truly async).

});
return new Blob([gzippedData.buffer as ArrayBuffer], {
type: "application/gzip",

return new Blob([lz4Data.buffer as ArrayBuffer], {
type: "application/x-lz4",
});
};

Expand Down
Loading