Wasm binary compression & alternatives to bloating a git repo

[0pcom]

Concept: embedded wasm

The idea is to be able to embed the web sources compiled by core build web into another golang program - presumably the server side part of the same application.

The main motivation for doing this is file size limit for files uploaded to a git repo.

I see, now, that the file size limit is much lower for uploading files via the github web interface than via git add on the command line.

However, it is still beneficial to include the smallest binary possible in a git repo if that is being embedded into another golang program and served.

I've never come across a tool which can successfully compress a wasm binary - but I've not done an exhaustive search.

UPX

I can confirm from previous tests that UPX is not suitable for compressing wasm binaries, and will not produce a working wasm binary.

binaryen

It seems that wasm-opt provided by binaryen may function correctly for compressing wasm binaries. I'm compiling it now to test.

...

After installing binaryen I attempted to run the wasm-opt tool on the compiled app.wasm of the web sources for docs

wasm-opt --enable-bulk-memory-opt -Oz --fast-math --disable-simd -o app1.wasm app.wasm

with the most aggressive mode, I was running out of RAM before the operation completed. It consumed ~13GB of ram

Here are other optimization options from wasm-opt

Optimization options:
---------------------

   -O                                           execute default optimization 
                                                passes (equivalent to -Os) 
   -O0                                          execute no optimization passes 
   -O1                                          execute -O1 optimization passes
                                                (quick&useful opts, useful for 
                                                iteration builds) 
   -O2                                          execute -O2 optimization passes
                                                (most opts, generally gets most
                                                perf) 
   -O3                                          execute -O3 optimization passes
                                                (spends potentially a lot of 
                                                time optimizing) 
   -O4                                          execute -O4 optimization passes
                                                (also flatten the IR, which can
                                                take a lot more time and memory,
                                                but is useful on more nested / 
                                                complex / less-optimized input)
   -Os                                          execute default optimization 
                                                passes, focusing on code size 
   -Oz                                          execute default optimization 
                                                passes, super-focusing on code 
                                                size 

With -O2 the binary size shrinks by only a few megs, but the operation does complete.

$ du -h *.wasm
65M	app1.wasm
67M	app.wasm
$ du *.wasm
66196	app1.wasm
67752	app.wasm

With every setting above -O2 I had to kill the process at around 90% memory usage. Perhaps it will work better on a machine with more RAM,

Compression to transfer encoding formats

Instead of compressing the wasm binary such that it maintains execute-ability, perhaps it could be compressed via brotli or gzip or other browser supported compression formats. The reasoning for this is that one isn't typically executing the wasm binary on the disk. The browser gets them via network interface, so it's potentially a little tweak on the server side of just setting the compression type while not applying compression for the already-compressed wasm binary. The advantage of this is that it can possibly be coupled with other binary compression methods which result in executable wasm files.

Source inclusion + core compiler as runtime dependency

An alternative approach - assuming that I want to run an application with go run or go install - without having the source
would be to include (embed) the golang source code which is compiled by core build web into the server side of the application. Then, one can invoke the program to produce the files on the disk which are then compiled by core build web and served by this application. This would certainly avoid bloating a git repo with a wasm binary, but it would add the core tool (and golang) as a dependency for at least the subcommand which invokes this.

The advantage here is just avoiding a wasm binary in a git repo for application source code.

[0pcom]

I've elected to use the last alternative method of source inclusion with core as a runtime dependency on the server side.

A directory is created inside the system's temporary folder in order to extract the embedded sources (which, in this example, is everything inside the ui directory)

Here is some sample code to that effect. This is server-side code where the ui sources are embedded.
The files get written to /tmp/ui` and the vendor dir is left untouched on subsequent invocations. Here is a trimmed down example

Output:

2025/03/19 04:15:01 All files successfully extracted to: /tmp/ui
go: creating new go.mod: module fiber.skywire.dev/ui
go: to add module requirements and sums:
	go mod tidy
go: added github.com/skycoin/skycoin v0.28.1-0.20241105130348-39b49a2d0a7f
go: added github.com/skycoin/skywire v1.3.29-rc7.0.20250313050427-65ff378bfe94
go: finding module for package cogentcore.org/core/content
go: finding module for package cogentcore.org/core/core
go: finding module for package cogentcore.org/core/styles
go: finding module for package cogentcore.org/core/icons
go: finding module for package cogentcore.org/core/tree
go: finding module for package cogentcore.org/core/base/mergefs
go: finding module for package github.com/0magnet/calvin
go: finding module for package cogentcore.org/core/paint
go: found cogentcore.org/core/base/mergefs in cogentcore.org/core v0.3.8
go: found cogentcore.org/core/content in cogentcore.org/core v0.3.8
go: found cogentcore.org/core/core in cogentcore.org/core v0.3.8
go: found cogentcore.org/core/icons in cogentcore.org/core v0.3.8
go: found cogentcore.org/core/paint in cogentcore.org/core v0.3.8
go: found cogentcore.org/core/styles in cogentcore.org/core v0.3.8
go: found cogentcore.org/core/tree in cogentcore.org/core v0.3.8
go: found github.com/0magnet/calvin in github.com/0magnet/calvin v0.0.0-20241204203002-126d3bcd81d8
core build succeeded
[GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.
 - using env:	export GIN_MODE=release
 - using code:	gin.SetMode(gin.ReleaseMode)

[GIN-debug] GET    /app-worker.js            --> main.main.func1.2 (1 handlers)
[GIN-debug] GET    /app.css                  --> main.main.func1.2 (1 handlers)
[GIN-debug] GET    /app.js                   --> main.main.func1.2 (1 handlers)
[GIN-debug] GET    /app.wasm                 --> main.main.func1.2 (1 handlers)
[GIN-debug] GET    /icons/192.png            --> main.main.func1.2 (1 handlers)
[GIN-debug] GET    /icons/32.png             --> main.main.func1.2 (1 handlers)
[GIN-debug] GET    /icons/512.png            --> main.main.func1.2 (1 handlers)
[GIN-debug] GET    /icons/svg.svg            --> main.main.func1.2 (1 handlers)
[GIN-debug] GET    /                         --> main.main.func1.1 (1 handlers)
[GIN-debug] GET    /manifest.webmanifest     --> main.main.func1.2 (1 handlers)
[GIN-debug] GET    /wasm_exec.js             --> main.main.func1.2 (1 handlers)
listening on http://127.0.0.1:8082 using gin router
[GIN-debug] [WARNING] You trusted all proxies, this is NOT safe. We recommend you to set a value.
Please check https://pkg.go.dev/github.com/gin-gonic/gin#readme-don-t-trust-all-proxies for details.
[GIN-debug] Listening and serving HTTP on :8082

Code:

package main

import (
	"embed"
	"log"
	"path/filepath"
	"fmt"
	"io/fs"
	"os"
	"errors"
	"strings"
	"github.com/gin-gonic/gin"
	"github.com/bitfield/script"
)

const webPort = 8082

//go:embed ui/*
var embeddedFiles embed.FS

func extractFiles() (string, error) {
    // Create a temporary directory inside the system's temp folder
    tempDir := os.TempDir() + "/ui"
    err := os.Mkdir(tempDir, 0755)
    if err != nil {
        // If the directory already exists or error in creating, clean up the directory manually
				toRemove, err := script.FindFiles(tempDir).Reject(tempDir+"/vendor").Slice()
				if err != nil {
						return "", err
				}
				for i, _ := range toRemove {
					_, err := os.Stat(toRemove[i])
					if err != nil && !errors.Is(err, os.ErrNotExist) {
						return "", err
					} else {
					err = os.RemoveAll(toRemove[i])
					if err != nil {
						return "", err
					}
				}
			}
			log.Println("Omitted vendor dir when cleaning directory:", tempDir)
			log.Println("Directory contents:")
			_, err = script.FindFiles(tempDir).Stdout()
    }


    // Walk through embedded files and extract them
    if err := fs.WalkDir(embeddedFiles, "ui", func(path string, d fs.DirEntry, err error) error {
        if err != nil {
            return fmt.Errorf("failed to access path %s: %w", path, err)
        }
        relPath, err := filepath.Rel("ui", path)
        if err != nil {
            return fmt.Errorf("failed to calculate relative path for %s: %w", path, err)
        }
        outputPath := filepath.Join(tempDir, relPath)
        if d.IsDir() {
            return os.MkdirAll(outputPath, 0755)
        }
        content, err := embeddedFiles.ReadFile(path)
        if err != nil {
            return fmt.Errorf("failed to read file %s: %w", path, err)
        }
        if err := os.WriteFile(outputPath, content, 0644); err != nil {
            return fmt.Errorf("failed to write file %s: %w", outputPath, err)
        }
        return nil
    }); err != nil {
        return "", fmt.Errorf("failed to extract files: %w", err)
    }
    return tempDir, nil
}


func main() {
outputDir, err := extractFiles()
if err != nil {
  log.Fatal("Error extracting files:", err)
}
log.Println("All files successfully extracted to:", outputDir)

//now compile the cogentcore application for web
_, err = script.Exec(`bash -c 'cd `+outputDir+` || exit 0 ; go mod init fiber.skywire.dev/ui ; go get github.com/skycoin/skywire@develop && go mod tidy && go mod vendor && core build web'`).Stdout()
if err != nil {
  fmt.Println("Error:", err)
  return
}

r1 := gin.New()

//Define additional handlers / routes before making handlers for the core application compiled for web.
//r1.GET("/somepath", func(c *gin.Context) {


filepath.Walk(outputDir+"/bin/web", func(path string, info os.FileInfo, err error) error {
	if err != nil {
		return err
	}

	if !info.IsDir() {
		relPath, err := filepath.Rel(outputDir+"/bin/web", path)
		if err != nil {
			return err
		}

		if strings.HasSuffix(relPath, "index.html") {
			r1.GET("/", func(c *gin.Context) {
				c.File(path)
			})
		} else {
			r1.GET("/"+relPath, func(c *gin.Context) {
				c.File(path)
			})
		}
	}
	return nil

})

fmt.Printf("listening on http://127.0.0.1:%d using gin router\n", webPort)
r1.Run(fmt.Sprintf(":%d", webPort)) //nolint

}

it works surprisingly fast.

[kkoreilly]

@0pcom Thank you for asking about this. In our case, we use GitHub Pages to host all of our apps statically; see https://github.com/cogentcore/core/blob/main/.github/workflows/core.yml and https://github.com/cogentcore/cogent/blob/main/.github/workflows/core.yml for examples. That allows us to take advantage of GitHub features while not bloating the repository with any large files, as everything is done in the GitHub Workflow, so the binary doesn't get checked into the repository.

In your case, if you already have a server that you need for other reasons, then you probably want to compile the binary when you first start up the server, as it seems like you have done. That is similar to the GitHub Workflow approach in that you are programmatically compiling the code on the server side, which avoids the need to include it in the Git repo, and you don't need to compile it manually.

Reducing the binary size on web is an ongoing process. The biggest thing will be #974 if possible, but GopherJS is too far behind on versions to use yet. We are changing the font embedding process in #1457, which should reduce web binary size by around 6%. We will also improve icon embedding in #1473, which will further reduce the binary size of some apps.

We experimented with UPX and wasm-opt before and found similar unsuccessful things as you. However, gzip does do a great job with wasm files in general, and all of our apps are served with gzip compression automatically done by GitHub pages. You can also set the -gzip flag in core build web and core run web to have it automatically gzip it for you; core run web automatically sets appropriate HTTP headers, but I am not exactly sure how it would work in different deployment environments. core run web is generally designed for development use, not production deployments, whereas core build web gives you more control.

Overall, it seems like you have found a reasonable solution, and it is not clear what else we can do at this point until GopherJS is ready. If you have any other questions or ideas, I would be glad to hear them.

[0pcom]

Yes, in my case I need a server which is looking at files on the disk (which are updated by another process) that will serve the wasm as well as other endpoints for the wasm running in the browser to communicate with the server and get the updated data.

This is for one user interface among several that the software has, and the others are embedded.
I've taken pains in the last two years to basically get the software down to a single binary with no extra files because it makes packaging all that much simpler. And actually it's a much smaller installation overall.

The size of this is, I think, somewhat critical if the compiled web UI is to be embedded.

In my case, golang + core as a runtime dependency for compiling the UI was an acceptable compromise for the much smaller size of embedding the uncompiled source code - because this particular UI is something only developers run, currently.

But for any truly distributable software, you would not want to have golang as a runtime dependency if you could avoid it, because sometimes only very old versions of golang are available on, for instance, armbian & other .deb based distros, and upgrading to sid to be able to get a recent enough version of golang is time consuming and resource intensive.

So sometimes you just have to bite the bullet and embed the compiled wasm.

I mainly share this for the benefit of others and my own future reference. And that's great to know about the gzip flag.

Core is such an impressive framework that it tempts inclusion of multiple different UIs (desktop & web) in a single application.