Rust extractor

[timofei-iatsenko]

I’m pretty close to building a Rust version of the extractor using SWC.

The recent changes finally closed the gap between the Babel macro and SWC, so they’re now at feature parity. That was the hardest part, since most of the heavy lifting actually happens inside the macro itself.

When I say “extractor”, I don’t mean a full CLI rewrite. I’m talking only about the part that analyzes source files and extracts messages — the bit that’s currently done by Babel.

The idea is to ship the Rust extractor as a separate package at first, alongside the existing Babel one. After some real-world testing, it could become the default, while keeping the Babel extractor around for Vue and other JS-based setups.

To get there, we’d need to port @lingui/babel-plugin-extract-messages to Rust. The plugin is pretty small (~400 LOC), and I think an LLM could handle most of that work without much trouble.

After that, we can build a small Rust program on top of SWC, bake in all the required transforms, compile it into a single binary, and expose it to Node.js using napi-rs. In the end, this would just be a prebuilt binary with one job: extract messages.

[timofei-iatsenko]

@Zxilly @karmeleon @jessicalc @AndrewIngram
I know some of you have run into Lingui performance issues in the past, and I’d really appreciate your thoughts on this proposal. It would also be great if you could share your use cases and any concrete stats you have.

Looking forward to any feedback — thanks!

[timofei-iatsenko]

I have a quick update: I’ve successfully rewrote @lingui/babel-plugin-extract-messages using Rust and SWC, including full source map management. While it’s still in the early stages and needs some polishing, it has already passed the entire original test suite.

I expect this version to be significantly faster than Babel for two main reasons:

Optimized AST Processing:

Unlike a full SWC compilation cycle, I’m only using the swc_parser. We don’t need the extra overhead of removing TypeScript types, downleveling syntax, hygiene passes or printing AST back to code. Our workflow is much leaner:

1. Parse string to AST.
2. Run a resolver pass for scopes and identifiers.
3. Run the LinguiMacro and Extractor visitor passes.

Native Execution (No WASM and serialization overhead):

Standard SWC plugins run in WebAssembly, which can be slower due to memory isolation and the need to serialize/deserialize the AST when moving it between the host and the plugin. By baking the transform directly into the same runtime as swc_core, we use a native pipeline. This removes the serialization penalty entirely.

[karmeleon]

This is awesome! My team currently extracts by building all our entrypoints (>250) in a single Rspack build, then using a custom Rspack plugin and the @lingui/babel-plugin-extract-messages transform to generate a JSON blob of translation keys sorted by the entrypoint that uses them. For legacy reasons, we then process this with a custom toolchain that integrates with Lokalise to generate the final messages JSON blob.

We use this approach over lingui extract-experimental since it allows us to use our existing Rspack build, and we already had the Rspack plugin from our previous i18n infra so it was easy to adapt for Lingui. The plugin only scans files that actually use Lingui and only scans each file once, so we only spend around 11 seconds of the ~10 minute Rspack build extracting all the i18n keys. I don't have numbers handy, but I remember extract-experimental taking several minutes when we tried it. We had to maintain a separate esbuild config to make the builds work, which wasn't ideal since we already had to maintain an Rspack config. It also took a few gigabytes of RAM, and since our Jenkins boxes are already low on RAM from running our Rspack builds, it didn't make sense to pursue it further.

So I wouldn't personally benefit much from an all-Rust extraction toolchain, but I'd be curious to learn what other teams are doing with Lingui at scale!

[timofei-iatsenko]

Hey everyone,

I’ve been investigating the performance of our current extraction command. It felt suspicious that a highly parallel task like extraction -which processes each file separately and should be easily distributed across cores—doesn't really change in speed whether you run it with --workers 2 or --workers 12.

The Benchmark

I benchmarked this on the bsky/social-app project, which has approximately 1,300 files to scan and 2,500 messages.

Result on M3 Max MacBook: (10 physical cores):

• 4-8 workers: The extraction phase alone took about 1.5–1.8 seconds.
• Single Core: It took about 2.5 seconds.

Adding more workers didn't change much, which is very suspicious.

You might say 1.5 seconds is "good enough," but in reality, this is only the extraction phase. The full lingui extract run includes merging, sorting, and writing catalogs to disk, it’s about 3–4 seconds on my MacBook Pro. On a CI server without a powerful CPU, it takes 18 seconds. I see a huge potential for improvement here.

My Experiments

I experimented with the code to find the bottleneck:

1. Moving I/O to the main thread: Node.js already executes I/O operations on a thread pool using libuv. Executing I/O inside worker threads just saturates libuv unnecessarily. Even though metrics showed a clear speed-up in file reading (150ms instead of 550ms), it didn't change the total time. I used transferable buffers to pass files to workers, so serialization overhead isn't the reason.
   I was surprised to find that reading files on the main thread gave inconsistent results (sometimes 150ms, other times 500ms). This behavior became more stable (in a bad way, more often 500ms) when I added more workers (like 12). It seems spawning worker threads happens in the same libuv event loop as file reading, so they interfere with each other.
2. Batching tasks: Instead of passing 1 file as 1 task, I chunked them based on the number of cores. Each worker received a batch of files to process at once to reduce the overhead of task orchestration and communication. This still didn't help; I couldn't break the 1.5-second wall.

Then I tried a SWC-based extractor (it’s almost ready, i'm on my way with build pipelines for native binaries). Since it's native code, we don't need Node.js worker threads because the extractor runs on the libuv thread pool by default.

I replaced the Babel implementation with SWC and left the rest as-is: file reading happens in JS concurrently using Promise.all, and then the extractor runs in a loop.
The result: Execution time dropped to ~600ms - more than 2x faster.

I wanted even more optimization, so I extended the SWC extractor to handle file reading. Instead of reading files in JS and passing them to native code, I passed the file paths and let the Rust code load and process them in parallel using Rayon.

Rust threads and Node.js threads work differently. A Rust thread spawns in microseconds, while Node threads take milliseconds. Also, there is no copy or serialization cost since Rust uses shared memory. Moving both file reading and extraction to Rust avoided Node.js task orchestration completely.
The result: Time dropped to ~400ms, a 1.5x improvement over the previous SWC result.

Conclusion

This experiment confirms what other projects already know: Node.js is not ideal for high-performance, CPU-bound CLI applications. The way worker threads are scheduled and the cost of serialization simply don't allow for peak performance.

The Rust-based extractor is the way to go. I don't believe there are any ways left to squeeze more performance out of the JS-based solution. Now I need to figure out the best way to implement this, as moving file reading into the extractor will significantly change the Lingui CLI architecture.

Rust extractor POC lingui/swc-plugin#200

fyi @mozzius

[Zxilly]

I think perhaps the sample size used for testing should be larger, which could in a sense mitigate the impact of warmup on timing.

[sondr3]

Would this also potentially pave a way to not have to use SWC with e.g. Vite in the future? We are already using the rolldown version of Vite and would like to get rid of SWC too (#2283)

[timofei-iatsenko]

Rolldown still does not allow rust plugins. So answer - no. At least for now. I dug into rust sources of rolldown recently, i investigated will it be possible to use it to power tree-extractor and inject rust version of extractor/macro transform here - and unfortunately i didn't find an easy "public" way of doing so. It seems it's not a first priority for void0 to have native plugins.

Have you becnhmarked how much overhead adds using SWC with a rolldown-vite? Just curious...

[sondr3]

I don't have enough knowledge about this sadly, I just know that LingUI will be blocking us from migrating 100% to using OXC/rolldown at some point. Don't let me sidetrack this issue here with that, you already answered in #2283, I was just curious

[timofei-iatsenko]

Moving back to the extractor topic, i measured how much time take spawning a worker thread:

1. 20-25ms when worker file is empty, no imports - just a performance.now() call.
2. 200-230ms once babel is added.

I also tried to bundle babel as single js file to eliminate IO overhead on reading files by node - but it basically haven't change anything. Still 200+ ms on parsing babel code.

Anyway, even with 250ms of warming up a worker should give a clear performance improvement. Take 2.5s on a single core, than spawning 4 workers (they all spawned in parallel and should be ready right about the same time), so every worker should receive workload for 625ms + 250ms warmup time, total 875ms. That still lower that my measured 1.5-1.8 seconds.

Interesting.

[timofei-iatsenko]

An update here, actually i made a fully working Rust extractor with native bindings https://npmx.dev/package/lingui-swc

It's using an SWC Macro, and passes the test suite of original extractor for 100%. It's not plug and play now, it's just a 2 functions exposed:

extractMessagesFromFiles
extractMessages

But you can create a simple wrapper to connect this to the lingui to test the perf in your case:

import { ExtractorType, ExtractedMessage, ExtractorCtx, defineConfig } from "@lingui/conf"

export const matchRe = new RegExp(
  "\\.(" +
    [".ts", ".mts", ".cts", ".tsx", ".js", ".jsx"]
      .map((ext) => ext.slice(1))
      .join("|") +
    ")$",
  "i",
)

const swcExtractor: ExtractorType = {
  match(filename) {
    return matchRe.test(filename)
  },

  async extract(filename, code, onMessageExtracted, ctx) {
    const { extractMessages } = await import("lingui-extractor-swc")

    const { messages } = await extractMessages(code, filename, {
      parser: { tsx: true, syntax: "typescript" },
    })

    messages.forEach((msg) => {
      onMessageExtracted({
        id: msg.id,
        origin: [msg.origin.filename, msg.origin.line, msg.origin.column],
        placeholders: msg.placeholders,
        context: msg.context,
        comment: msg.comment,
      })
    })
  },
}

// lingui.config.ts
defineConfig({
   extractors: [swcExtractor],
})

Note, that rust extractor created from V5 version branch. it would not be compatible with v6 for a while. (due to change in the id generation)

[timofei-iatsenko]

I updated the package to 0.2.0 https://npmx.dev/package/lingui-swc

• macro bumped to latest v6. (you still can use this extractor with v5, by passing useLinguiV5IdGeneration: true to macro options
• added autodetection of jsx and ecma/typescript support based on the file extension
• added a lingui extractor wrapper into the package, now you can consume this:

import { defineConfig } from '@lingui/cli'
import { createSwcExtractor } from 'lingui-swc'

export default defineConfig({
  /* ... */
  extractors: [createSwcExtractor()],
})

I dogfooded this to our company's application - and it works out of the box as a drop-in replacement. The time dropped from the 2.2 secs to 600ms.

Note: you also need to disable workers in the lingui cli by passing --workers=1. The native extractor utilize multithreading using libuv pool and works much more effectively than threads in the nodejs.

Another note: this extractor produces slightly different column/line mapping, this is due to SWC implementation itself, and it's intended.

Please test it and give a feedback.

[mozzius]

Current:

2.37s

Current w/ workers=1:

3.75s

lingui-swc:

1.91s

lingui-swc w/ workers=1:

1.84s

nice!!