blog/how_to_rewrite_a_cpp_codebase_successfully.html at master · gaultier/blog · GitHub

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
<!--
This file has been auto-generated by main.rs from a markdown file of the same name.
Do not edit it by hand.
-->
<!DOCTYPE html>
<html>
<head>
<title>How to rewrite a C++ codebase successfully</title>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<link type="application/atom+xml" href="/blog/feed.xml" rel="self">
<link rel="shortcut icon" type="image/ico" href="/blog/favicon.ico">
<link rel="stylesheet" type="text/css" href="main.css">
<link rel="stylesheet" href="https://unpkg.com/@highlightjs/cdn-assets@11.8.0/styles/default.min.css">
<script type="module" src="main.js" async></script>
<script type="module" src="search.js" async></script>
</head>
<body>

<div id="banner">
    <div id="name">
        <img id="me" src="me.jpeg">
        <span>Philippe Gaultier</span>
    </div>
    <input id="search" placeholder="🔎 Search" autocomplete=off>
    <ul>
      <li> <button id="dark-light-mode">Dark/Light</button> </li>
      <li> <a href="/blog/body_of_work.html">Body of work</a> </li>
      <li> <a href="/blog/articles-by-tag.html">Tags</a> </li>
      <li> <a href="https://github.com/gaultier/resume/raw/master/Philippe_Gaultier_resume_en.pdf">
          Resume
        </a> </li>

      <li> <a href="/blog/feed.xml">
        <svg viewBox="0 0 24 24" fill="currentColor" xmlns="http://www.w3.org/2000/svg">
        <path fill-rule="evenodd" clip-rule="evenodd" d="M5.5 3.5C4.39543 3.5 3.5 4.39543 3.5 5.5V18.5C3.5 19.6046 4.39543 20.5 5.5 20.5H18.5C19.6046 20.5 20.5 19.6046 20.5 18.5V5.5C20.5 4.39543 19.6046 3.5 18.5 3.5H5.5ZM7 19C8.10457 19 9 18.1046 9 17C9 15.8954 8.10457 15 7 15C5.89543 15 5 15.8954 5 17C5 18.1046 5.89543 19 7 19ZM6.14863 10.5052C6.14863 10.0379 6.52746 9.65906 6.99478 9.65906C7.95949 9.65906 8.91476 9.84908 9.80603 10.2183C10.6973 10.5874 11.5071 11.1285 12.1893 11.8107C12.8715 12.4929 13.4126 13.3027 13.7817 14.194C14.1509 15.0852 14.3409 16.0405 14.3409 17.0052C14.3409 17.4725 13.9621 17.8514 13.4948 17.8514C13.0275 17.8514 12.6486 17.4725 12.6486 17.0052C12.6486 16.2627 12.5024 15.5275 12.2183 14.8416C11.9341 14.1556 11.5177 13.5324 10.9927 13.0073C10.4676 12.4823 9.84437 12.0659 9.15842 11.7817C8.47246 11.4976 7.73726 11.3514 6.99478 11.3514C6.52746 11.3514 6.14863 10.9725 6.14863 10.5052ZM7 5.15385C6.53268 5.15385 6.15385 5.53268 6.15385 6C6.15385 6.46732 6.53268 6.84615 7 6.84615C8.33342 6.84615 9.65379 7.10879 10.8857 7.61907C12.1176 8.12935 13.237 8.87728 14.1799 9.82015C15.1227 10.763 15.8707 11.8824 16.3809 13.1143C16.8912 14.3462 17.1538 15.6666 17.1538 17C17.1538 17.4673 17.5327 17.8462 18 17.8462C18.4673 17.8462 18.8462 17.4673 18.8462 17C18.8462 15.4443 18.5397 13.9039 17.9444 12.4667C17.3491 11.0294 16.4765 9.72352 15.3765 8.6235C14.2765 7.52349 12.9706 6.65091 11.5333 6.05558C10.0961 5.46026 8.55566 5.15385 7 5.15385Z" fill="currentColor"/>
        </svg>
        </a> </li>

      <li> <a href="https://www.linkedin.com/in/philippegaultier/">
        <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" data-supported-dps="24x24" fill="currentColor" width="24" height="24" focusable="false">
              <path d="M20.5 2h-17A1.5 1.5 0 002 3.5v17A1.5 1.5 0 003.5 22h17a1.5 1.5 0 001.5-1.5v-17A1.5 1.5 0 0020.5 2zM8 19H5v-9h3zM6.5 8.25A1.75 1.75 0 118.3 6.5a1.78 1.78 0 01-1.8 1.75zM19 19h-3v-4.74c0-1.42-.6-1.93-1.38-1.93A1.74 1.74 0 0013 14.19a.66.66 0 000 .14V19h-3v-9h2.9v1.3a3.11 3.11 0 012.7-1.4c1.55 0 3.36.86 3.36 3.66z"/>
            </svg>
        </a> </li>
      <li> <a href="https://github.com/gaultier">
        <svg height="32" aria-hidden="true" viewBox="0 0 24 24" version="1.1" width="32" data-view-component="true" fill="currentColor">
          <path d="M12.5.75C6.146.75 1 5.896 1 12.25c0 5.089 3.292 9.387 7.863 10.91.575.101.79-.244.79-.546 0-.273-.014-1.178-.014-2.142-2.889.532-3.636-.704-3.866-1.35-.13-.331-.69-1.352-1.18-1.625-.402-.216-.977-.748-.014-.762.906-.014 1.553.834 1.769 1.179 1.035 1.74 2.688 1.25 3.349.948.1-.747.402-1.25.733-1.538-2.559-.287-5.232-1.279-5.232-5.678 0-1.25.445-2.285 1.178-3.09-.115-.288-.517-1.467.115-3.048 0 0 .963-.302 3.163 1.179.92-.259 1.897-.388 2.875-.388.977 0 1.955.13 2.875.388 2.2-1.495 3.162-1.179 3.162-1.179.633 1.581.23 2.76.115 3.048.733.805 1.179 1.825 1.179 3.09 0 4.413-2.688 5.39-5.247 5.678.417.36.776 1.05.776 2.128 0 1.538-.014 2.774-.014 3.162 0 .302.216.662.79.547C20.709 21.637 24 17.324 24 12.25 24 5.896 18.854.75 12.5.75Z"/>
        </svg>
        </a> </li>
      <li> <a href="https://hachyderm.io/@pg">
        <svg width="75" height="79" viewBox="0 0 75 79" xmlns="http://www.w3.org/2000/svg" fill="currentColor">
        <path d="M73.8393 17.4898C72.6973 9.00165 65.2994 2.31235 56.5296 1.01614C55.05 0.797115 49.4441 0 36.4582 0H36.3612C23.3717 0 20.585 0.797115 19.1054 1.01614C10.5798 2.27644 2.79399 8.28712 0.904997 16.8758C-0.00358524 21.1056 -0.100549 25.7949 0.0682394 30.0965C0.308852 36.2651 0.355538 42.423 0.91577 48.5665C1.30307 52.6474 1.97872 56.6957 2.93763 60.6812C4.73325 68.042 12.0019 74.1676 19.1233 76.6666C26.7478 79.2728 34.9474 79.7055 42.8039 77.9162C43.6682 77.7151 44.5217 77.4817 45.3645 77.216C47.275 76.6092 49.5123 75.9305 51.1571 74.7385C51.1797 74.7217 51.1982 74.7001 51.2112 74.6753C51.2243 74.6504 51.2316 74.6229 51.2325 74.5948V68.6416C51.2321 68.6154 51.2259 68.5896 51.2142 68.5661C51.2025 68.5426 51.1858 68.522 51.1651 68.5058C51.1444 68.4896 51.1204 68.4783 51.0948 68.4726C51.0692 68.4669 51.0426 68.467 51.0171 68.4729C45.9835 69.675 40.8254 70.2777 35.6502 70.2682C26.7439 70.2682 24.3486 66.042 23.6626 64.2826C23.1113 62.762 22.7612 61.1759 22.6212 59.5646C22.6197 59.5375 22.6247 59.5105 22.6357 59.4857C22.6466 59.4609 22.6633 59.4391 22.6843 59.422C22.7053 59.4048 22.73 59.3929 22.7565 59.3871C22.783 59.3813 22.8104 59.3818 22.8367 59.3886C27.7864 60.5826 32.8604 61.1853 37.9522 61.1839C39.1768 61.1839 40.3978 61.1839 41.6224 61.1516C46.7435 61.008 52.1411 60.7459 57.1796 59.7621C57.3053 59.7369 57.431 59.7154 57.5387 59.6831C65.4861 58.157 73.0493 53.3672 73.8178 41.2381C73.8465 40.7606 73.9184 36.2364 73.9184 35.7409C73.9219 34.0569 74.4606 23.7949 73.8393 17.4898Z"/>
        <path d="M61.2484 27.0263V48.114H52.8916V27.6475C52.8916 23.3388 51.096 21.1413 47.4437 21.1413C43.4287 21.1413 41.4177 23.7409 41.4177 28.8755V40.0782H33.1111V28.8755C33.1111 23.7409 31.0965 21.1413 27.0815 21.1413C23.4507 21.1413 21.6371 23.3388 21.6371 27.6475V48.114H13.2839V27.0263C13.2839 22.7176 14.384 19.2946 16.5843 16.7572C18.8539 14.2258 21.8311 12.926 25.5264 12.926C29.8036 12.926 33.0357 14.5705 35.1905 17.8559L37.2698 21.346L39.3527 17.8559C41.5074 14.5705 44.7395 12.926 49.0095 12.926C52.7013 12.926 55.6784 14.2258 57.9553 16.7572C60.1531 19.2922 61.2508 22.7152 61.2484 27.0263Z" fill="#928374" />
        <defs>
        <linearGradient id="paint0_linear_549_34" x1="37.0692" y1="0" x2="37.0692" y2="79" gradientUnits="userSpaceOnUse">
        <stop stop-color="#6364FF"/>
        <stop offset="1" stop-color="#563ACC"/>
        </linearGradient>
        </defs>
        </svg>
        </a> </li>
      <li> <a href="https://bsky.app/profile/pgaultier.bsky.social">
        <svg fill="currentColor" viewBox="0 0 64 57" width="32" style="width: 32px; height: 28.5px;"><path d="M13.873 3.805C21.21 9.332 29.103 20.537 32 26.55v15.882c0-.338-.13.044-.41.867-1.512 4.456-7.418 21.847-20.923 7.944-7.111-7.32-3.819-14.64 9.125-16.85-7.405 1.264-15.73-.825-18.014-9.015C1.12 23.022 0 8.51 0 6.55 0-3.268 8.579-.182 13.873 3.805ZM50.127 3.805C42.79 9.332 34.897 20.537 32 26.55v15.882c0-.338.13.044.41.867 1.512 4.456 7.418 21.847 20.923 7.944 7.111-7.32 3.819-14.64-9.125-16.85 7.405 1.264 15.73-.825 18.014-9.015C62.88 23.022 64 8.51 64 6.55c0-9.818-8.578-6.732-13.873-2.745Z"/></svg>
        </a> </li>
    </ul>
</div>
<div id="search-matches" hidden>
</div>
<div id="pseudo-body">

<div class="article-prelude">
<p><a href="/blog"> ⏴ Back to all articles</a></p>

<p class="publication-date">Published on 2024-05-03. Last modified on 2026-03-04.</p>
</div>
<div class="article-title">
  <h1>How to rewrite a C++ codebase successfully</h1>
<div class="tags"> <a href="/blog/articles-by-tag.html#c" class="tag">C</a> <a href="/blog/articles-by-tag.html#cplusplus" class="tag">C++</a> <a href="/blog/articles-by-tag.html#rust" class="tag">Rust</a> <a href="/blog/articles-by-tag.html#rewrite" class="tag">Rewrite</a> <a href="/blog/articles-by-tag.html#safety" class="tag">Safety</a> </div>
</div>
  <details class="toc"><summary>Table of contents</summary>
<ul>

  <li>
    <a href="#the-project">The project</a>
</li>

  <li>
    <a href="#improve-the-existing-codebase">Improve the existing codebase</a>
</li>

  <li>
    <a href="#get-buy-in">Get buy-in</a>
</li>

  <li>
    <a href="#keeping-buy-in">Keeping buy-in</a>
</li>

  <li>
    <a href="#preparations-to-introduce-the-new-language">Preparations to introduce the new language</a>
</li>

  <li>
    <a href="#incremental-rewrite">Incremental rewrite</a>
</li>

  <li>
    <a href="#fuzzing">Fuzzing</a>
</li>

  <li>
    <a href="#pure-rust-vs-interop-ffi">Pure Rust vs interop (FFI)</a>
<ul>

  <li>
    <a href="#c-ffi-in-rust-is-cumbersome-and-error-prone">C FFI in Rust is cumbersome and error-prone</a>
</li>

  <li>
    <a href="#an-example-of-a-real-bug-at-the-ffi-boundary">An example of a real bug at the FFI boundary</a>
</li>

  <li>
    <a href="#another-example-of-a-real-bug-at-the-ffi-boundary">Another example of a real bug at the FFI boundary</a>
</li>
</ul>
</li>

  <li>
    <a href="#cross-compilation">Cross-compilation</a>
</li>

  <li>
    <a href="#conclusion">Conclusion</a>
</li>
</ul>
</details>

<p><em>Discussions: <a href="https://old.reddit.com/r/programming/comments/1clfhzm/how_to_rewrite_a_c_codebase_successfully/">/r/programming</a>, <a href="https://old.reddit.com/r/rust/comments/1clfhqz/how_to_rewrite_a_c_codebase_successfully/">/r/rust</a>, <a href="https://lobste.rs/s/a6gffu/how_rewrite_c_codebase_successfully">Lobsters</a></em></p>
<p><em>Not your typical 'Rewrite it in Rust' article.</em></p>
<p>I recently wrote about <a href="/blog/you_inherited_a_legacy_cpp_codebase_now_what.html">inheriting a legacy C++ codebase</a>. At some point, although I cannot pinpoint exactly when, a few things became clear to me:</p>
<ul>
<li>No one in the team but me is able - or feels confident enough - to make a change in this codebase</li>
<li>This is a crucial project for the company and will live for years if not decades</li>
<li>The code is pretty bad on all the criteria we care about: correctness, maintainability, security, you name it. I don't blame the original developers, they were understaffed and it was written as a prototype (the famous case of the prototype which becomes the production code).</li>
<li>No hiring of C++ developers is planned or at least in the current budget (also because that's the only C++ project we have and we have many other projects to maintain and extend)</li>
</ul>
<p>So it was apparent to me that sticking with C++ was a dead end. It's simply a conflict of values and priorities: C++ values many things that are not that important in this project, such as performance above all; and it does not give any guarantees about things that are crucial to us, such as memory and temporal safety (special mention to integer under/overflows. Have fun reviewing every single instance of arithmetic operations to check if it can under/overflow).</p>
<p>We bought a race car but what we needed was a family-friendly 5 seater, that's our mistake.</p>
<p>The only solution would be to train everyone in the team on C++ and dedicate a significant amount of time rewriting the most problematic parts of the codebase to perhaps reach a good enough state, and even then, we'd have little confidence our code is robust against nation-state attacks.</p>
<p>It's a judgment call in the end, but that seemed to be more effort than 'simply' introducing a new language and doing a rewrite.</p>
<p>I don't actually like the term 'rewrite'. Folks on the internet will eagerly repeat that rewrites are a bad idea, will undoubtedly fail, and are a sign of hubris and naivety. I have experienced such rewrites, from scratch, and yes that does not end well.</p>
<p>However, I claim, because I've done it, and many others before me, that an <strong>incremental</strong> rewrite can be successful, and is absolutely worth it. It's all about how it is being done, so here's how I proceeded and I hope it can be applied in other cases, and people find it useful.</p>
<p>I think it's a good case study because whilst not a big codebase, it is a complex codebase, and it's used in production on 10+ different operating systems and architectures, including by external customers. This is not a toy.</p>
<p>So join me on this journey, here's the guide to rewrite a C++ codebase successfully. And also what not do!</p>
<h2 id="the-project">
  <a class="title" href="#the-project">The project</a>
  <a class="hash-anchor" href="#the-project" aria-hidden="true" onclick="navigator.clipboard.writeText(this.href);"></a>
</h2>
<p>This project is a library that exposes a C API but the implementation is C++, and it vendors C libraries (e.g. mbedtls) which we build from source. The final artifacts are a <code>libfoo.a</code> static library and a <code>libfoo.h</code> C header. It is used to talk to applets on a <a href="https://en.wikipedia.org/wiki/Smart_card">smart card</a> like your credit card, ID, passport or driving license (yes, smart cards are nowadays everywhere - you probably carry several on you right now), since they use a <del>bizarre</del> interesting communication protocol. The library also implements a home-grown protocol on top of the well-specified smart card protocol, encryption, and business logic.</p>
<p>It is meant to be part of an user-facing application running on smartphones and Point of Sales terminals, as well as in servers running in a datacenter or in the cloud.</p>
<p>This library is used in:</p>
<ul>
<li>Android applications, through JNI</li>
<li>Go back-end services running in Kubernetes, through CGO</li>
<li>iOS applications, through Swift FFI</li>
<li>C and C++ applications running on very small 32 bits ARM boards similar to the first Raspberry Pi</li>
</ul>
<p>Additionally, developers are using macOS (x64 and arm64) and Linux so the library needs to build and run on these platforms.</p>
<p>Since external customers also integrate their applications with our library and we do not control these environments, and because some developer machines and servers use glibc and others musl,  we also need to work with either the glibc and the musl C libraries, as well as clang and gcc, and expose a C89-ish API, to maximize compatibility.</p>
<p>Alright, now that the stage is set, let's go through the steps of rewriting this project.</p>
<h2 id="improve-the-existing-codebase">
  <a class="title" href="#improve-the-existing-codebase">Improve the existing codebase</a>
  <a class="hash-anchor" href="#improve-the-existing-codebase" aria-hidden="true" onclick="navigator.clipboard.writeText(this.href);"></a>
</h2>
<p>That's basically all the steps in <a href="/blog/you_inherited_a_legacy_cpp_codebase_now_what.html">Inheriting a legacy C++ codebase</a>. We need to start the rewrite with a codebase that builds and runs on every platform we support, with tests passing, and a clear README explaining how to setup the project locally. This is a small investment (a few days to a few weeks depending on the scale of the codebase) that will pay massive dividends in the future.</p>
<p>But I think the most important point is to trim all the unused code which is typically the majority of the codebase! No one wants to spend time and effort on rewriting completely unused code.</p>
<p>Additionally, if you fail to convince your team and the stakeholders to do the rewrite, you at least have improved the codebase you are now stuck with. So it's time well spent either way.</p>
<h2 id="get-buy-in">
  <a class="title" href="#get-buy-in">Get buy-in</a>
  <a class="hash-anchor" href="#get-buy-in" aria-hidden="true" onclick="navigator.clipboard.writeText(this.href);"></a>
</h2>
<p>Same as in my previous article: Buy-in from teammates and stakeholders is probably the most important thing to get, and maintain.</p>
<p>It's a big investment in time and thus money we are talking about, it can only work with everyone on board.</p>
<p>Here I think the way to go is showing the naked truth and staying very factual, in terms managers and non-technical people can understand. This is roughly what I presented:</p>
<ul>
<li>The bus factor for this project is 1 (me)</li>
<li>Tool X shows that there are memory leaks at the rate of Y MiB/hour which means the application using our library will be OOM killed after around Z minutes/hours.</li>
<li>Quick and dirty fuzzing manages to make the library crash 133 times in 10 seconds</li>
<li>Linter X detects hundreds of real issues we need to fix</li>
<li>All of these points make it really likely a hacker can exploit our library to gain Remote Code Execution (RCE) or steal secrets</li>
</ul>
<p>Essentially, it's a matter of genuinely presenting the alternative of rewriting being cheaper in terms of time and effort compared to improving the project with pure C++. If your teammates and boss are reality-based, it should be a straightforward decision.</p>
<p>We use at my day job basically a RFC process to introduce a major change. That's great because it forces the person pushing for a change to document the current issues, the possible solutions, and allowing for a rational discussion to take place in the team. And documenting the whole process in a shared document (that allows comments) is very valuable because when people ask about it months later, you can just share the link to it.</p>
<p>After the problematic situation has been presented, I think at least 3 different solutions should be presented and compared (including sticking with pure C++), and seriously consider each option. I find it important here to be as little emotionally invested as possible even if one option is your favorite, and to be ready to work for possibly months on your least favorite option, if it happens to be chosen by the collective.</p>
<p>Ideally, if time permits, a small prototype for the preferred solution should be done, to confirm or infirm early that it can work, and to eliminate doubts. It's a much more compelling argument to say: &quot;Of course it will work, here is prototype I made, let's look at it together!&quot; compared to &quot;I hope it will work, but who knows, oh well I guess we'll see 3 months in...&quot;.</p>
<p>After much debate, we settled on Rust as the new programming language being introduced into the codebase. It's important to note that I am not a Rust die hard fan. I appreciate the language but it's not perfect (see the FFI section later), it has issues, it's just that it solves all the issues we have in this project, especially considering the big focus on security (since we deal with payments),  the relative similarity with the company tech stack (Go), and the willingness of the team to learn it and review code in it.</p>
<p>After all, the goal is also to gain additional developers, and stop being the only person who can even touch this code.</p>
<p>I also seriously considered Go, but after doing a prototype, I was doubtful the many limitations of CGO would allow us to achieve the rewrite. Other teammates also had concerns on how the performance and battery usage would look like on low-end Android and Linux devices, especially 32 bits, having essentially two garbage collectors running concurrently, the JVM one and the Go one.</p>
<h2 id="keeping-buy-in">
  <a class="title" href="#keeping-buy-in">Keeping buy-in</a>
  <a class="hash-anchor" href="#keeping-buy-in" aria-hidden="true" onclick="navigator.clipboard.writeText(this.href);"></a>
</h2>
<p>Keeping buy-in after initially getting it is not a given, since software always takes longer than expected and unexpected hurdles happen all the time. Here, showing the progress through regular demos (weekly or biweekly is a good frequency) is great for stakeholders especially non-technical ones. And it can potentially motivate fellow developers to also learn the new language and help you out.</p>
<p>Additionally, showing how long-standing issues in the old code get automatically solved by the new code, e.g. memory leaks, or fuzzing crashes in one function, are a great sign for stakeholders of the quality improving and the value of the on-going effort.</p>
<p>Be prepared to repeat many many times the decision process that led to the rewrite to your boss, your boss's boss, the odd product manager who's not technical, the salesperson supporting the external customers, etc. It's important to nail the elevator's pitch.</p>
<p>That applies also to teammates, who might be unsure the new programming language 'carries its weight'. It helps to regularly ask them how they feel about the language, the on-going-effort, the roadmap, etc. Also, pairing with them, so that ideally, everyone in the team feels confident working on this project alone.</p>
<h2 id="preparations-to-introduce-the-new-language">
  <a class="title" href="#preparations-to-introduce-the-new-language">Preparations to introduce the new language</a>
  <a class="hash-anchor" href="#preparations-to-introduce-the-new-language" aria-hidden="true" onclick="navigator.clipboard.writeText(this.href);"></a>
</h2>
<p>Before adding the first line of code in the new language, I created a Git tag <code>last-before-rust</code>. The commit right after introduced some code in Rust.</p>
<p>This proved invaluable, because when rewriting the legacy code, I found tens of bugs lying around, and I think that's very typical. Also, this rewriting effort requires time, during which other team members or external customers may report bugs they just found.</p>
<p>Every time such a bug appeared, I switched to this Git tag, and tried to reproduce the bug. Almost every time, the bug was already present before the rewrite. That's a very important information (for me, it was a relief!) for solving the bug, and also for stakeholders. That's the difference in their eye between: We are improving the product by fixing long existing bugs; or: we are introducing new bugs with our risky changes and we should maybe stop the effort completely because it's harming the product.</p>
<p>Furthermore, I think the first commit introducing the new code should add dummy code and focus on making the build system and CI work seamlessly on every supported platform. This is not appealing work but it's necessary. Also, having instructions in the README explaining a bit what each tool does (<code>cargo</code>, <code>rustup</code>, <code>clippy</code>, etc) is very valuable and will ease beginners into contributing in the new language.</p>
<h2 id="incremental-rewrite">
  <a class="title" href="#incremental-rewrite">Incremental rewrite</a>
  <a class="hash-anchor" href="#incremental-rewrite" aria-hidden="true" onclick="navigator.clipboard.writeText(this.href);"></a>
</h2>
<p>Along with stakeholder buy-in, the most important point in the article is that only an <strong>incremental</strong> rewrite can succeed, in my opinion. Rewriting from scratch is bound to fail, I think. At least I have never seen it succeed, and have seen it fail many times.</p>
<p>What does it mean, very pragmatically? Well it's just a few rules of thumb:</p>
<ul>
<li>A small component is picked to be rewritten, the smallest, the better. Ideally it is as small as one function, or one class.</li>
<li>The new implementation is written in the same Git (or whatever CVS you use) repository as the existing code, alongside it. It's a 'bug for bug' implementation which means it does the exact same thing as the old implementation, even if the old seems sometimes non-sensical. In some cases, what the old code tries to do is so broken and insecure, that we have to do something different in the new code, but that should be rare.</li>
<li>Tests for the new implementation are written and pass (so that we know the new implementation is likely correct)</li>
<li>Each site calling the function/class is switched to using the new implementation. After each replacement, the test suite is run and passes (so that we know that nothing broke at the scale of the project; a kind of regression testing). The change is committed. That way is something breaks, we know exactly which change is the culprit.</li>
<li>A small PR is opened, reviewed and merged. Since our changes are anyways incremental, it's up to us to decide that the current diff is of the right size for a PR. We can make the PR as big or small as we want. We can even make a PR with only the new implementation that's not yet used at all.</li>
<li>Once the old function/class is not used anymore by any code, it can be 'garbage-collected' i.e. safely removed. This can even be its own PR depending on the size.</li>
<li>Rinse and repeat until all of the old code has been replaced</li>
</ul>
<p>There are of course thornier cases, but that's the gist of it. What's crucial is that each commit on the main branch builds and runs fine. At not point the codebase is ever broken, does not build, or is in an unknown state.</p>
<p>It's actually not much different from the way I do a refactor in a codebase with just one programming language.</p>
<p>What's very important to avoid are big PRs that are thousands lines long and nobody wants to review them, or long running branches that effectively create a multiverse inside the codebase. It's the same as regular software development, really.</p>
<p>Here are a few additional tips I recommend doing:</p>
<ul>
<li>Starting from the leaves of the call graph is much easier than from the root. For example, if <code>foo</code> calls <code>bar</code> which calls <code>baz</code>, first rewriting <code>baz</code> then <code>bar</code> then <code>foo</code> is straightforward, but the reverse is usually not true.</li>
<li>Thus, mapping out at the start from a high-level what are the existing components and which component calls out to which other component is invaluable in that regard, but also for establishing a rough roadmap for the rewrite, and reporting on the progress (&quot;3 components have been rewritten, 2 left to do!&quot;).</li>
<li>Port the code comments from the old code to the new code if they make sense and add value. In my experience, a few are knowledge gems and should be kept, and most are completely useless noise.</li>
<li>If you can use automated tools (search and replace, or tools operating at the AST level) to change every call site to use the new implementation, it'll make your reviewers very happy, and save you hours and hours of debugging because of a copy-paste mistake</li>
<li>Since Rust and C++ can basically only communicate through a C API (I am aware of experimental projects to make them talk directly but we did not use those - we ultimately want 100% Rust code exposing a C API, just like the old C++ did), it means that each Rust function must be accompanied by a corresponding C function signature, so that C++ can call it as a C function. I recommend automating this process with <a href="https://github.com/mozilla/cbindgen">cbindgen</a>. I have encountered some limitations with it but it's very useful, especially to keep the implementation (in Rust) and the API (in C) in sync, or if your teammates are not comfortable with C.</li>
<li>Automate when you can, for example I added the <code>cbindgen</code> code generation step to CMake so that rebuilding the C++ project would automatically run <code>cbindgen</code> as well as <code>cargo build</code> for the right target in the right mode (debug or release) for the right platforms (<code>--target=...</code>). DevUX matters!</li>
<li>When rewriting a function/class, port the tests for this function/class to the new implementation to avoid reducing the code coverage each time</li>
<li>Make the old and the new test suites fast so that the iteration time is short</li>
<li>When a divergence is detected (a difference in output or side effects between the old and the new implementation), observe with tests or within the debugger the output of the old implementation (that's where the initial Git tag comes handy, and working with small commits) in detail so that you can correct the new implementation. Some people even develop big test suites verifying that the output of the old and the new implementation are exactly the same.</li>
<li>Since it's a bug-for-bug rewrite, <em>what</em> the new implementation does may seem weird or unnecessarily convoluted but shall be kept (at least as a first pass). However, <em>how</em> it does it in the new code should be up to the best software engineering standards, that means tests, fuzzing, documentation, etc.</li>
<li>Thread lightly, what can tank the project is being too bold when rewriting code and by doing so, introducing bugs or subtly changing the behavior which will cause breakage down the line. It's better to be conservative here.</li>
<li>Pick a prefix for all structs and functions in the C API exposed by the Rust code, even if it's just <code>RUST_xxx</code>, so that they are immediately identifiable and greppable. Just like <code>libcurl</code> has the prefix <code>curl_xxx</code>.</li>
</ul>
<p>Finally, there is one hidden advantage of doing an incremental rewrite. A from-scratch rewrite is all or nothing, if it does not fully complete and replace the old implementation, it's useless and wasteful. However, an incremental rewrite is immediately useful, may be paused and continued a number of times, and even if the funding gets cut short and it never fully completes, it's still a clear improvement over the starting point.</p>
<h2 id="fuzzing">
  <a class="title" href="#fuzzing">Fuzzing</a>
  <a class="hash-anchor" href="#fuzzing" aria-hidden="true" onclick="navigator.clipboard.writeText(this.href);"></a>
</h2>
<p>I am a fan a fuzzing, it's great. Almost every time I fuzz some code, I find a corner case I did not think about, especially when doing parsing.</p>
<p>I added fuzzing to the project so that every new Rust function is fuzzed. I initially used <a href="https://rust-fuzz.github.io/book/afl.html">AFL</a> but then turned to <a href="https://rust-fuzz.github.io/book/cargo-fuzz.html">cargo-fuzz</a>, and I'll explain why.</p>
<p>Fuzzing is only useful if code coverage is <a href="https://blog.trailofbits.com/2024/03/01/toward-more-effective-curl-fuzzing/">high</a>. The worst that can happen is to dedicate serious time to setup fuzzing, to only discover at the end that the same few branches are always taken during fuzzing.</p>
<p>Coverage can only be improved if developers can easily see exactly which branches are being executed during fuzzing. And I could not find an easy way with AFL to get a hold on that data.</p>
<p>Using <code>cargo-fuzz</code> and various LLVM tools, I wrote a small shell script to visualize exactly which branches are taken during fuzzing as well as the code coverage in percents for each file and for the project as a whole (right now it's at around 90%).</p>
<p>To get to a high coverage, the quality of the corpus data is paramount, since fuzzing works by doing small mutations of this corpus and observing which branches are taken as a result.</p>
<p>I realized that the existing tests in C++ had lots of useful data in them, e.g.:</p>
<pre>
  <div class="code-header">
    <span>Cpp</span>
    <button class="copy-code" type="button"><svg aria-hidden="true" focusable="false" class="octicon octicon-copy" viewBox="0 0 16 16" width="16" height="16" fill="currentColor" display="inline-block" overflow="visible" style="vertical-align: text-bottom;"><path d="M0 6.75C0 5.784.784 5 1.75 5h1.5a.75.75 0 0 1 0 1.5h-1.5a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-1.5a.75.75 0 0 1 1.5 0v1.5A1.75 1.75 0 0 1 9.25 16h-7.5A1.75 1.75 0 0 1 0 14.25Z"></path><path d="M5 1.75C5 .784 5.784 0 6.75 0h7.5C15.216 0 16 .784 16 1.75v7.5A1.75 1.75 0 0 1 14.25 11h-7.5A1.75 1.75 0 0 1 5 9.25Zm1.75-.25a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-7.5a.25.25 0 0 0-.25-.25Z"></path></svg></button>
  </div>
  <code class="language-cpp"><span class="line-number"></span><span class="code-hl">const std::vector&lt;char&gt; input = {0x32, 0x01, 0x49, ...}; // &lt;= This is the interesting data.</span>
<span class="line-number"></span><span class="code-hl">assert(foo(input) == ...);</span>
</code></pre>
<p>So I had the idea of extracting all the <code>input = ...</code> data from the tests to build a good fuzzing corpus. My first go at it was a hand-written quick and dirty C++ lexer in Rust. It worked but it was clunky. Right after I finished it, I thought: why don't I use <code>tree-sitter</code> to properly parse C++ in Rust?</p>
<p>And so I did, and it turned out great, just 300 lines of Rust walking through each <code>TestXXX.cpp</code> file in the repository and using tree-sitter to extract each pattern. I used the query language of tree-sitter to do so:</p>
<pre>
  <div class="code-header">
    <span>Rust</span>
    <button class="copy-code" type="button"><svg aria-hidden="true" focusable="false" class="octicon octicon-copy" viewBox="0 0 16 16" width="16" height="16" fill="currentColor" display="inline-block" overflow="visible" style="vertical-align: text-bottom;"><path d="M0 6.75C0 5.784.784 5 1.75 5h1.5a.75.75 0 0 1 0 1.5h-1.5a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-1.5a.75.75 0 0 1 1.5 0v1.5A1.75 1.75 0 0 1 9.25 16h-7.5A1.75 1.75 0 0 1 0 14.25Z"></path><path d="M5 1.75C5 .784 5.784 0 6.75 0h7.5C15.216 0 16 .784 16 1.75v7.5A1.75 1.75 0 0 1 14.25 11h-7.5A1.75 1.75 0 0 1 5 9.25Zm1.75-.25a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-7.5a.25.25 0 0 0-.25-.25Z"></path></svg></button>
  </div>
  <code class="language-rust"><span class="line-number"></span><span class="code-hl">let query = tree_sitter::Query::new(</span>
<span class="line-number"></span><span class="code-hl">    tree_sitter_cpp::language(),</span>
<span class="line-number"></span><span class="code-hl">    &quot;(initializer_list (number_literal)+) @capture&quot;,</span>
<span class="line-number"></span><span class="code-hl">)</span>
</code></pre>
<p>The tree-sitter website thankfully has a playground where I could experiment and tweak the query and see the results live.</p>
<p>As time went on and more and more C++ tests were migrated to Rust tests, it was very easy to extend this small Rust program that builds the corpus data, to also scan the Rust tests!</p>
<p>A typical Rust test would look like this:</p>
<pre>
  <div class="code-header">
    <span>Rust</span>
    <button class="copy-code" type="button"><svg aria-hidden="true" focusable="false" class="octicon octicon-copy" viewBox="0 0 16 16" width="16" height="16" fill="currentColor" display="inline-block" overflow="visible" style="vertical-align: text-bottom;"><path d="M0 6.75C0 5.784.784 5 1.75 5h1.5a.75.75 0 0 1 0 1.5h-1.5a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-1.5a.75.75 0 0 1 1.5 0v1.5A1.75 1.75 0 0 1 9.25 16h-7.5A1.75 1.75 0 0 1 0 14.25Z"></path><path d="M5 1.75C5 .784 5.784 0 6.75 0h7.5C15.216 0 16 .784 16 1.75v7.5A1.75 1.75 0 0 1 14.25 11h-7.5A1.75 1.75 0 0 1 5 9.25Zm1.75-.25a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-7.5a.25.25 0 0 0-.25-.25Z"></path></svg></button>
  </div>
  <code class="language-rust"><span class="line-number"></span><span class="code-hl">const INPUT: [u8; 4] = [0x01, 0x02, 0x03, 0x04]; // &lt;= This is the interesting data.</span>
<span class="line-number"></span><span class="code-hl">assert_eq!(foo(&amp;INPUT), ...);</span>
</code></pre>
<p>And the query to extract the interesting data would be:</p>
<pre>
  <div class="code-header">
    <span>Rust</span>
    <button class="copy-code" type="button"><svg aria-hidden="true" focusable="false" class="octicon octicon-copy" viewBox="0 0 16 16" width="16" height="16" fill="currentColor" display="inline-block" overflow="visible" style="vertical-align: text-bottom;"><path d="M0 6.75C0 5.784.784 5 1.75 5h1.5a.75.75 0 0 1 0 1.5h-1.5a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-1.5a.75.75 0 0 1 1.5 0v1.5A1.75 1.75 0 0 1 9.25 16h-7.5A1.75 1.75 0 0 1 0 14.25Z"></path><path d="M5 1.75C5 .784 5.784 0 6.75 0h7.5C15.216 0 16 .784 16 1.75v7.5A1.75 1.75 0 0 1 14.25 11h-7.5A1.75 1.75 0 0 1 5 9.25Zm1.75-.25a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-7.5a.25.25 0 0 0-.25-.25Z"></path></svg></button>
  </div>
  <code class="language-rust"><span class="line-number"></span><span class="code-hl">let query = tree_sitter::Query::new(</span>
<span class="line-number"></span><span class="code-hl">    tree_sitter_rust::language(),</span>
<span class="line-number"></span><span class="code-hl">    // TODO: Maybe make this query more specific with:</span>
<span class="line-number"></span><span class="code-hl">    // `(let_declaration value: (array_expression (integer_literal)+)) @capture`.</span>
<span class="line-number"></span><span class="code-hl">    // But in a few cases, the byte array is defined with `const`, not `let`.</span>
<span class="line-number"></span><span class="code-hl">    &quot;(array_expression (integer_literal)+) @capture&quot;,</span>
<span class="line-number"></span><span class="code-hl">)</span>
</code></pre>
<p>However I discovered that not all data was successfully extracted. What about this code:</p>
<pre>
  <div class="code-header">
    <span>Rust</span>
    <button class="copy-code" type="button"><svg aria-hidden="true" focusable="false" class="octicon octicon-copy" viewBox="0 0 16 16" width="16" height="16" fill="currentColor" display="inline-block" overflow="visible" style="vertical-align: text-bottom;"><path d="M0 6.75C0 5.784.784 5 1.75 5h1.5a.75.75 0 0 1 0 1.5h-1.5a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-1.5a.75.75 0 0 1 1.5 0v1.5A1.75 1.75 0 0 1 9.25 16h-7.5A1.75 1.75 0 0 1 0 14.25Z"></path><path d="M5 1.75C5 .784 5.784 0 6.75 0h7.5C15.216 0 16 .784 16 1.75v7.5A1.75 1.75 0 0 1 14.25 11h-7.5A1.75 1.75 0 0 1 5 9.25Zm1.75-.25a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-7.5a.25.25 0 0 0-.25-.25Z"></path></svg></button>
  </div>
  <code class="language-rust"><span class="line-number"></span><span class="code-hl">const BAR : u8 = 0x42;</span>
<span class="line-number"></span><span class="code-hl">const INPUT: [u8; 4] = [BAR, 0x02, 0x03, 0x04]; // &lt;= This is the interesting data.</span>
<span class="line-number"></span><span class="code-hl">assert_eq!(foo(&amp;INPUT), ...);</span>
</code></pre>
<p>We have a constant <code>BAR</code> which trips up tree-sitter, because it only sees a literal (i.e. 3 letters: 'B', 'A' and 'R') and does not know its value.</p>
<p>The way I solved this issue was to do two passes: once to collect all constants along with their values in a map, and then a second pass to find all arrays in tests:</p>
<pre>
  <div class="code-header">
    <span>Rust</span>
    <button class="copy-code" type="button"><svg aria-hidden="true" focusable="false" class="octicon octicon-copy" viewBox="0 0 16 16" width="16" height="16" fill="currentColor" display="inline-block" overflow="visible" style="vertical-align: text-bottom;"><path d="M0 6.75C0 5.784.784 5 1.75 5h1.5a.75.75 0 0 1 0 1.5h-1.5a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-1.5a.75.75 0 0 1 1.5 0v1.5A1.75 1.75 0 0 1 9.25 16h-7.5A1.75 1.75 0 0 1 0 14.25Z"></path><path d="M5 1.75C5 .784 5.784 0 6.75 0h7.5C15.216 0 16 .784 16 1.75v7.5A1.75 1.75 0 0 1 14.25 11h-7.5A1.75 1.75 0 0 1 5 9.25Zm1.75-.25a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-7.5a.25.25 0 0 0-.25-.25Z"></path></svg></button>
  </div>
  <code class="language-rust"><span class="line-number"></span><span class="code-hl">let query = tree_sitter::Query::new(</span>
<span class="line-number"></span><span class="code-hl">    tree_sitter_rust::language(),</span>
<span class="line-number"></span><span class="code-hl">    &quot;(const_item value: (integer_literal)) @capture &quot;,</span>
<span class="line-number"></span><span class="code-hl">)</span>
</code></pre>
<p>So that we can then resolve the literals to their numeric value.</p>
<p>That's how I implemented a compiler for the Kotlin programming language in the past and it worked great. Maybe there are more advanced approaches but this one is dead-simple and fast so it's good enough for us.</p>
<p>I am pretty happy with how this turned out, scanning all C++ and Rust files to find interesting test data in them to build the corpus. I think this was key to move from the initial 20% code coverage with fuzzing (using a few hard-coded corpus files) to 90%. It's fast too.</p>
<p>Also, it means the corpus gets better each time we had a test (be it in C++ or Rust), for free.</p>
<p>Does it mean that the corpus will grow to an extreme size? Well, worry not, because LLVM comes with a fuzzing corpus minimizer:</p>
<pre>
  <div class="code-header">
    <span>Shell</span>
    <button class="copy-code" type="button"><svg aria-hidden="true" focusable="false" class="octicon octicon-copy" viewBox="0 0 16 16" width="16" height="16" fill="currentColor" display="inline-block" overflow="visible" style="vertical-align: text-bottom;"><path d="M0 6.75C0 5.784.784 5 1.75 5h1.5a.75.75 0 0 1 0 1.5h-1.5a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-1.5a.75.75 0 0 1 1.5 0v1.5A1.75 1.75 0 0 1 9.25 16h-7.5A1.75 1.75 0 0 1 0 14.25Z"></path><path d="M5 1.75C5 .784 5.784 0 6.75 0h7.5C15.216 0 16 .784 16 1.75v7.5A1.75 1.75 0 0 1 14.25 11h-7.5A1.75 1.75 0 0 1 5 9.25Zm1.75-.25a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-7.5a.25.25 0 0 0-.25-.25Z"></path></svg></button>
  </div>
  <code class="language-shell"><span class="line-number"></span><span class="code-hl"># Minimize the fuzzing corpus (in place).</span>
<span class="line-number"></span><span class="code-hl">$ cargo +nightly fuzz cmin [...]</span>
</code></pre>
<p>For each file in the corpus, it feeds it as input to our code, observes which branches are taken, and if a new set of branches is taken, this file remains (or perhaps gets minimized even more, not sure how smart this tool is). Otherwise it is deemed a duplicate and is trimmed.</p>
<p>So:</p>
<ol>
<li>We generate the corpus with our program</li>
<li>Minimize it</li>
<li>Run the fuzzing for however long we wish. It runs in CI for every commit and developers can also run it locally.</li>
<li>When fuzzing is complete, we print the code coverage statistics</li>
</ol>
<p>Finally, we still have the option to add manually crafted files to this corpus if we wish. For example after hitting a bug in the wild, and fixing it, we can add a reproducer file to the corpus as a kind of regression test.</p>
<h2 id="pure-rust-vs-interop-ffi">
  <a class="title" href="#pure-rust-vs-interop-ffi">Pure Rust vs interop (FFI)</a>
  <a class="hash-anchor" href="#pure-rust-vs-interop-ffi" aria-hidden="true" onclick="navigator.clipboard.writeText(this.href);"></a>
</h2>
<p>Writing Rust has been a joy, even for more junior developers in the team. Pure Rust code was pretty much 100% correct on the first try.</p>
<p>However we had to use <code>unsafe {}</code> blocks in the FFI layer. We segregated all the FFI code to one file, and converted the C FFI structs to Rust idiomatic structs as soon as possible, so that the bulk of the Rust code can be idiomatic and safe.</p>
<p>But that means this FFI code is the most likely part of the Rust code to have bugs. To get some confidence in its correctness, we write Rust tests using the C FFI functions (as if we were a C consumer of the library) running under <a href="https://github.com/rust-lang/miri">Miri</a> which acts as valgrind essentially, simulating a CPU and checking that our code is memory safe. Tests run perhaps 5 to 10 times as slow as without Miri but this has proven invaluable since it detected many bugs ranging from alignment issues to memory leaks and use-after-free issues.</p>
<p>We run tests under Miri in CI to make sure each commit is reasonably safe.</p>
<p>So beware: introducing Rust to a C or C++ codebase may actually introduce new memory safety issues, usually all located in the FFI code.</p>
<p>Thankfully that's a better situation to be in than to have to inspect all of the codebase when a memory issue is detected.</p>
<h3 id="c-ffi-in-rust-is-cumbersome-and-error-prone">
  <a class="title" href="#c-ffi-in-rust-is-cumbersome-and-error-prone">C FFI in Rust is cumbersome and error-prone</a>
  <a class="hash-anchor" href="#c-ffi-in-rust-is-cumbersome-and-error-prone" aria-hidden="true" onclick="navigator.clipboard.writeText(this.href);"></a>
</h3>
<p>The root cause for all these issues is that the C API that C++ and Rust use to call each other is very limited in its expressiveness w.r.t ownership, as well as many Rust types not being marked <code>#[repr(C)]</code>, even types you would expect to, such as <code>Option</code>, <code>Vec</code> or <code>&amp;[u8]</code>. That means that you have to define your own equivalent types:</p>
<pre>
  <div class="code-header">
    <span>Rust</span>
    <button class="copy-code" type="button"><svg aria-hidden="true" focusable="false" class="octicon octicon-copy" viewBox="0 0 16 16" width="16" height="16" fill="currentColor" display="inline-block" overflow="visible" style="vertical-align: text-bottom;"><path d="M0 6.75C0 5.784.784 5 1.75 5h1.5a.75.75 0 0 1 0 1.5h-1.5a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-1.5a.75.75 0 0 1 1.5 0v1.5A1.75 1.75 0 0 1 9.25 16h-7.5A1.75 1.75 0 0 1 0 14.25Z"></path><path d="M5 1.75C5 .784 5.784 0 6.75 0h7.5C15.216 0 16 .784 16 1.75v7.5A1.75 1.75 0 0 1 14.25 11h-7.5A1.75 1.75 0 0 1 5 9.25Zm1.75-.25a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-7.5a.25.25 0 0 0-.25-.25Z"></path></svg></button>
  </div>
  <code class="language-rust"><span class="line-number"></span><span class="code-hl">#[repr(C)]</span>
<span class="line-number"></span><span class="code-hl">// An option type that can be used from C</span>
<span class="line-number"></span><span class="code-hl">pub struct OptionC&lt;T&gt; {</span>
<span class="line-number"></span><span class="code-hl">    pub has_value: bool,</span>
<span class="line-number"></span><span class="code-hl">    pub value: T,</span>
<span class="line-number"></span><span class="code-hl">}</span>
<span class="line-number"></span><span class="code-hl"></span>
<span class="line-number"></span><span class="code-hl"></span>
<span class="line-number"></span><span class="code-hl">#[repr(C)]</span>
<span class="line-number"></span><span class="code-hl">// Akin to `&amp;[u8]`, for C.</span>
<span class="line-number"></span><span class="code-hl">pub struct ByteSliceView {</span>
<span class="line-number"></span><span class="code-hl">    pub ptr: *const u8,</span>
<span class="line-number"></span><span class="code-hl">    pub len: usize,</span>
<span class="line-number"></span><span class="code-hl">}</span>
<span class="line-number"></span><span class="code-hl"></span>
<span class="line-number"></span><span class="code-hl">/// Owning Array i.e. `Vec&lt;T&gt;` in Rust or `std::vector&lt;T&gt;` in C++.</span>
<span class="line-number"></span><span class="code-hl">#[repr(C)]</span>
<span class="line-number"></span><span class="code-hl">pub struct OwningArrayC&lt;T&gt; {</span>
<span class="line-number"></span><span class="code-hl">    pub data: *mut T,</span>
<span class="line-number"></span><span class="code-hl">    pub len: usize,</span>
<span class="line-number"></span><span class="code-hl">    pub cap: usize,</span>
<span class="line-number"></span><span class="code-hl">}</span>
<span class="line-number"></span><span class="code-hl"></span>
<span class="line-number"></span><span class="code-hl">/// # Safety</span>
<span class="line-number"></span><span class="code-hl">/// Only call from C.</span>
<span class="line-number"></span><span class="code-hl">#[no_mangle]</span>
<span class="line-number"></span><span class="code-hl">pub extern &quot;C&quot; fn make_owning_array_u8(len: usize) -&gt; OwningArrayC&lt;u8&gt; {</span>
<span class="line-number"></span><span class="code-hl">    vec![0; len].into()</span>
<span class="line-number"></span><span class="code-hl">}</span>
</code></pre>
<p>Apparently, Rust developers do not want to commit to a particular ABI for these types, to avoid missing out on some future optimizations. So it means that every Rust struct now needs the equivalent &quot;FFI friendly&quot; struct along with conversion functions (usually implemented as <code>.into()</code> for convenience):</p>
<pre>
  <div class="code-header">
    <span>Rust</span>
    <button class="copy-code" type="button"><svg aria-hidden="true" focusable="false" class="octicon octicon-copy" viewBox="0 0 16 16" width="16" height="16" fill="currentColor" display="inline-block" overflow="visible" style="vertical-align: text-bottom;"><path d="M0 6.75C0 5.784.784 5 1.75 5h1.5a.75.75 0 0 1 0 1.5h-1.5a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-1.5a.75.75 0 0 1 1.5 0v1.5A1.75 1.75 0 0 1 9.25 16h-7.5A1.75 1.75 0 0 1 0 14.25Z"></path><path d="M5 1.75C5 .784 5.784 0 6.75 0h7.5C15.216 0 16 .784 16 1.75v7.5A1.75 1.75 0 0 1 14.25 11h-7.5A1.75 1.75 0 0 1 5 9.25Zm1.75-.25a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-7.5a.25.25 0 0 0-.25-.25Z"></path></svg></button>
  </div>
  <code class="language-rust"><span class="line-number"></span><span class="code-hl">struct Foo&lt;'a&gt; {</span>
<span class="line-number"></span><span class="code-hl">    x: Option&lt;usize&gt;,</span>
<span class="line-number"></span><span class="code-hl">    y: &amp;'a [u8],</span>
<span class="line-number"></span><span class="code-hl">    z: Vec&lt;u8&gt;,</span>
<span class="line-number"></span><span class="code-hl">}</span>
<span class="line-number"></span><span class="code-hl"></span>
<span class="line-number"></span><span class="code-hl"></span>
<span class="line-number"></span><span class="code-hl">#[repr(C)]</span>
<span class="line-number"></span><span class="code-hl">struct FooC {</span>
<span class="line-number"></span><span class="code-hl">    x: OptionC&lt;usize&gt;,</span>
<span class="line-number"></span><span class="code-hl">    y: ByteSliceView,</span>
<span class="line-number"></span><span class="code-hl">    z: OwningArrayC&lt;u8&gt;,</span>
<span class="line-number"></span><span class="code-hl">}</span>
</code></pre>
<p>Which is cumbersome but still fine, especially since Rust has powerful macros (which I investigated using but did not eventually). However, since Rust also does not have great idiomatic support for custom allocators, we stuck with the standard memory allocator, which meant that each struct with heap-allocated fields has to have a deallocation function:</p>
<pre>
  <div class="code-header">
    <span>Rust</span>
    <button class="copy-code" type="button"><svg aria-hidden="true" focusable="false" class="octicon octicon-copy" viewBox="0 0 16 16" width="16" height="16" fill="currentColor" display="inline-block" overflow="visible" style="vertical-align: text-bottom;"><path d="M0 6.75C0 5.784.784 5 1.75 5h1.5a.75.75 0 0 1 0 1.5h-1.5a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-1.5a.75.75 0 0 1 1.5 0v1.5A1.75 1.75 0 0 1 9.25 16h-7.5A1.75 1.75 0 0 1 0 14.25Z"></path><path d="M5 1.75C5 .784 5.784 0 6.75 0h7.5C15.216 0 16 .784 16 1.75v7.5A1.75 1.75 0 0 1 14.25 11h-7.5A1.75 1.75 0 0 1 5 9.25Zm1.75-.25a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-7.5a.25.25 0 0 0-.25-.25Z"></path></svg></button>
  </div>
  <code class="language-rust"><span class="line-number"></span><span class="code-hl">#[no_mangle]</span>
<span class="line-number"></span><span class="code-hl">pub extern &quot;C&quot; fn foo_free(foo: &amp;FooC) {</span>
<span class="line-number"></span><span class="code-hl">    ...</span>
<span class="line-number"></span><span class="code-hl">}</span>
</code></pre>
<p>And the C or C++ calling code would have to do:</p>
<pre>
  <div class="code-header">
    <span>Cpp</span>
    <button class="copy-code" type="button"><svg aria-hidden="true" focusable="false" class="octicon octicon-copy" viewBox="0 0 16 16" width="16" height="16" fill="currentColor" display="inline-block" overflow="visible" style="vertical-align: text-bottom;"><path d="M0 6.75C0 5.784.784 5 1.75 5h1.5a.75.75 0 0 1 0 1.5h-1.5a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-1.5a.75.75 0 0 1 1.5 0v1.5A1.75 1.75 0 0 1 9.25 16h-7.5A1.75 1.75 0 0 1 0 14.25Z"></path><path d="M5 1.75C5 .784 5.784 0 6.75 0h7.5C15.216 0 16 .784 16 1.75v7.5A1.75 1.75 0 0 1 14.25 11h-7.5A1.75 1.75 0 0 1 5 9.25Zm1.75-.25a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-7.5a.25.25 0 0 0-.25-.25Z"></path></svg></button>
  </div>
  <code class="language-cpp"><span class="line-number"></span><span class="code-hl">FooC foo{};</span>
<span class="line-number"></span><span class="code-hl">if (foo_parse(&amp;foo, bytes) == SUCCESS) {</span>
<span class="line-number"></span><span class="code-hl">    // do something with foo...</span>
<span class="line-number"></span><span class="code-hl">    ...</span>
<span class="line-number"></span><span class="code-hl"></span>
<span class="line-number"></span><span class="code-hl">    foo_free(foo);</span>
<span class="line-number"></span><span class="code-hl">}</span>
</code></pre>
<p>To simplify this, I introduced a <code>defer</code> <a href="https://www.gingerbill.org/article/2015/08/19/defer-in-cpp/">construct</a> to C++ (thanks Gingerbill!):</p>
<pre>
  <div class="code-header">
    <span>Cpp</span>
    <button class="copy-code" type="button"><svg aria-hidden="true" focusable="false" class="octicon octicon-copy" viewBox="0 0 16 16" width="16" height="16" fill="currentColor" display="inline-block" overflow="visible" style="vertical-align: text-bottom;"><path d="M0 6.75C0 5.784.784 5 1.75 5h1.5a.75.75 0 0 1 0 1.5h-1.5a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-1.5a.75.75 0 0 1 1.5 0v1.5A1.75 1.75 0 0 1 9.25 16h-7.5A1.75 1.75 0 0 1 0 14.25Z"></path><path d="M5 1.75C5 .784 5.784 0 6.75 0h7.5C15.216 0 16 .784 16 1.75v7.5A1.75 1.75 0 0 1 14.25 11h-7.5A1.75 1.75 0 0 1 5 9.25Zm1.75-.25a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-7.5a.25.25 0 0 0-.25-.25Z"></path></svg></button>
  </div>
  <code class="language-cpp"><span class="line-number"></span><span class="code-hl">FooC foo{};</span>
<span class="line-number"></span><span class="code-hl">defer({foo_free(foo);});</span>
<span class="line-number"></span><span class="code-hl"></span>
<span class="line-number"></span><span class="code-hl">if (foo_parse(&amp;foo, bytes) == SUCCESS) {</span>
<span class="line-number"></span><span class="code-hl">    // do something with foo...</span>
<span class="line-number"></span><span class="code-hl">    ...</span>
<span class="line-number"></span><span class="code-hl">}</span>
</code></pre>
<p>Which feels right at home for Go developers, and is an improvement over the style in use in the old C++ code where it was fully manual calls to new/delete.</p>
<p>Still, it's more work than what you'd have to do in pure idiomatic Rust or C++ code (or even C code with arenas for that matter).</p>
<p>In Zig or Odin, I would probably have used arenas to avoid that, or a general allocator with <code>defer</code>.</p>
<h3 id="an-example-of-a-real-bug-at-the-ffi-boundary">
  <a class="title" href="#an-example-of-a-real-bug-at-the-ffi-boundary">An example of a real bug at the FFI boundary</a>
  <a class="hash-anchor" href="#an-example-of-a-real-bug-at-the-ffi-boundary" aria-hidden="true" onclick="navigator.clipboard.writeText(this.href);"></a>
</h3>
<p>More perniciously, it's easy to introduce memory unsafety at the FFI boundary. Here is a real bug I introduced, can you spot it? I elided all the error handling to make it easier to spot:</p>
<pre>
  <div class="code-header">
    <span>Rust</span>
    <button class="copy-code" type="button"><svg aria-hidden="true" focusable="false" class="octicon octicon-copy" viewBox="0 0 16 16" width="16" height="16" fill="currentColor" display="inline-block" overflow="visible" style="vertical-align: text-bottom;"><path d="M0 6.75C0 5.784.784 5 1.75 5h1.5a.75.75 0 0 1 0 1.5h-1.5a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-1.5a.75.75 0 0 1 1.5 0v1.5A1.75 1.75 0 0 1 9.25 16h-7.5A1.75 1.75 0 0 1 0 14.25Z"></path><path d="M5 1.75C5 .784 5.784 0 6.75 0h7.5C15.216 0 16 .784 16 1.75v7.5A1.75 1.75 0 0 1 14.25 11h-7.5A1.75 1.75 0 0 1 5 9.25Zm1.75-.25a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-7.5a.25.25 0 0 0-.25-.25Z"></path></svg></button>
  </div>
  <code class="language-rust"><span class="line-number"></span><span class="code-hl">#[repr(C)]</span>
<span class="line-number"></span><span class="code-hl">struct BarC {</span>
<span class="line-number"></span><span class="code-hl">    x: ByteSliceView,</span>
<span class="line-number"></span><span class="code-hl">}</span>
<span class="line-number"></span><span class="code-hl"></span>
<span class="line-number"></span><span class="code-hl">#[no_mangle]</span>
<span class="line-number"></span><span class="code-hl">unsafe extern &quot;C&quot; fn bar_parse(input: *const u8, input_len: usize, bar_c: &amp;mut BarC) {</span>
<span class="line-number"></span><span class="code-hl">    let input: &amp;[u8] = unsafe { std::slice::from_raw_parts(input, input_len) };</span>
<span class="line-number"></span><span class="code-hl"></span>
<span class="line-number"></span><span class="code-hl">    let bar: Bar = Bar {</span>
<span class="line-number"></span><span class="code-hl">        x: [input[0], input[1]],</span>
<span class="line-number"></span><span class="code-hl">    };</span>
<span class="line-number"></span><span class="code-hl"></span>
<span class="line-number"></span><span class="code-hl">    *bar_c = BarC {</span>
<span class="line-number"></span><span class="code-hl">        x: ByteSliceView {</span>
<span class="line-number"></span><span class="code-hl">            ptr: bar.x.as_ptr(),</span>
<span class="line-number"></span><span class="code-hl">            len: bar.x.len(),</span>
<span class="line-number"></span><span class="code-hl">        },</span>
<span class="line-number"></span><span class="code-hl">    };</span>
<span class="line-number"></span><span class="code-hl">}</span>
</code></pre>
<p><code>clippy</code> did not notice anything. <code>address-sanitizer</code> did not notice anything. However, both <code>miri</code> and <code>valgrind</code> did, and fuzzing crashed (which was not easy to troubleshoot but at least pinpointed to a problem).</p>
<p>So...found it? Still nothing? Well, let's be good developers and add a test for it:</p>
<pre>
  <div class="code-header">
    <span>Rust</span>
    <button class="copy-code" type="button"><svg aria-hidden="true" focusable="false" class="octicon octicon-copy" viewBox="0 0 16 16" width="16" height="16" fill="currentColor" display="inline-block" overflow="visible" style="vertical-align: text-bottom;"><path d="M0 6.75C0 5.784.784 5 1.75 5h1.5a.75.75 0 0 1 0 1.5h-1.5a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-1.5a.75.75 0 0 1 1.5 0v1.5A1.75 1.75 0 0 1 9.25 16h-7.5A1.75 1.75 0 0 1 0 14.25Z"></path><path d="M5 1.75C5 .784 5.784 0 6.75 0h7.5C15.216 0 16 .784 16 1.75v7.5A1.75 1.75 0 0 1 14.25 11h-7.5A1.75 1.75 0 0 1 5 9.25Zm1.75-.25a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-7.5a.25.25 0 0 0-.25-.25Z"></path></svg></button>
  </div>
  <code class="language-rust"><span class="line-number"></span><span class="code-hl">#[test]</span>
<span class="line-number"></span><span class="code-hl">fn bar() {</span>
<span class="line-number"></span><span class="code-hl">    // This mimicks how C/C++ code would call our function.</span>
<span class="line-number"></span><span class="code-hl">    let mut bar_c = MaybeUninit::&lt;BarC&gt;::uninit();</span>
<span class="line-number"></span><span class="code-hl">    let input = [0, 1, 2];</span>
<span class="line-number"></span><span class="code-hl">    unsafe {</span>
<span class="line-number"></span><span class="code-hl">        bar_parse(</span>
<span class="line-number"></span><span class="code-hl">            input.as_ptr(),</span>
<span class="line-number"></span><span class="code-hl">            input.len(),</span>
<span class="line-number"></span><span class="code-hl">            bar_c.as_mut_ptr().as_mut().unwrap(),</span>
<span class="line-number"></span><span class="code-hl">        );</span>
<span class="line-number"></span><span class="code-hl">    }</span>
<span class="line-number"></span><span class="code-hl"></span>
<span class="line-number"></span><span class="code-hl">    let bar_c = unsafe { bar_c.assume_init_ref() };</span>
<span class="line-number"></span><span class="code-hl">    let x: &amp;[u8] = (&amp;bar_c.x).into();</span>
<span class="line-number"></span><span class="code-hl">    assert_eq!(x, [0, 1].as_slice());</span>
<span class="line-number"></span><span class="code-hl">}</span>
</code></pre>
<p>If you're lucky, <code>cargo test</code> would fail at the last assertion saying that the value is not what we expected, but in my case it passed every time, and so the bug stayed undetected for a while. That's because we unknowingly introduced undefined behavior, and as such, how or if it manifests is impossible to tell.</p>
<p>Let's run the test with Miri:</p>
<pre>
  <div class="code-header">
    <span>Plaintext</span>
    <button class="copy-code" type="button"><svg aria-hidden="true" focusable="false" class="octicon octicon-copy" viewBox="0 0 16 16" width="16" height="16" fill="currentColor" display="inline-block" overflow="visible" style="vertical-align: text-bottom;"><path d="M0 6.75C0 5.784.784 5 1.75 5h1.5a.75.75 0 0 1 0 1.5h-1.5a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-1.5a.75.75 0 0 1 1.5 0v1.5A1.75 1.75 0 0 1 9.25 16h-7.5A1.75 1.75 0 0 1 0 14.25Z"></path><path d="M5 1.75C5 .784 5.784 0 6.75 0h7.5C15.216 0 16 .784 16 1.75v7.5A1.75 1.75 0 0 1 14.25 11h-7.5A1.75 1.75 0 0 1 5 9.25Zm1.75-.25a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-7.5a.25.25 0 0 0-.25-.25Z"></path></svg></button>
  </div>
  <code class="language-plaintext"><span class="line-number"></span><span class="code-hl">running 1 test</span>
<span class="line-number"></span><span class="code-hl">test api::tests::bar ... error: Undefined Behavior: out-of-bounds pointer use: alloc195648 has been freed, so this pointer is dangling</span>
<span class="line-number"></span><span class="code-hl">    --&gt; src/tlv.rs:321:18</span>
<span class="line-number"></span><span class="code-hl">     |</span>
<span class="line-number"></span><span class="code-hl">321  |         unsafe { &amp;*core::ptr::slice_from_raw_parts(item.ptr, item.len) }</span>
<span class="line-number"></span><span class="code-hl">     |                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ out-of-bounds pointer use: alloc195648 has been freed, so this pointer is dangling</span>
<span class="line-number"></span><span class="code-hl">     |</span>
<span class="line-number"></span><span class="code-hl">     = help: this indicates a bug in the program: it performed an invalid operation, and caused Undefined Behavior</span>
<span class="line-number"></span><span class="code-hl">     = help: see https://doc.rust-lang.org/nightly/reference/behavior-considered-undefined.html for further information</span>
<span class="line-number"></span><span class="code-hl">help: alloc195648 was allocated here:</span>
<span class="line-number"></span><span class="code-hl">    --&gt; src/api.rs:1396:9</span>
<span class="line-number"></span><span class="code-hl">     |</span>
<span class="line-number"></span><span class="code-hl">1396 |     let bar: Bar = Bar {</span>
<span class="line-number"></span><span class="code-hl">     |         ^^^</span>
<span class="line-number"></span><span class="code-hl">help: alloc195648 was deallocated here:</span>
<span class="line-number"></span><span class="code-hl">    --&gt; src/api.rs:1406:1</span>
<span class="line-number"></span><span class="code-hl">     |</span>
<span class="line-number"></span><span class="code-hl">1406 | }</span>
</code></pre>
<p>Miri is great, I tell you.</p>
<p>The issue here is that we essentially return a pointer to local variable (<code>x</code>) from inside the function, so the pointer is dangling.</p>
<p>Alternatively we can call our function from C/C++ and run that under valgrind:</p>
<pre>
  <div class="code-header">
    <span>C</span>
    <button class="copy-code" type="button"><svg aria-hidden="true" focusable="false" class="octicon octicon-copy" viewBox="0 0 16 16" width="16" height="16" fill="currentColor" display="inline-block" overflow="visible" style="vertical-align: text-bottom;"><path d="M0 6.75C0 5.784.784 5 1.75 5h1.5a.75.75 0 0 1 0 1.5h-1.5a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-1.5a.75.75 0 0 1 1.5 0v1.5A1.75 1.75 0 0 1 9.25 16h-7.5A1.75 1.75 0 0 1 0 14.25Z"></path><path d="M5 1.75C5 .784 5.784 0 6.75 0h7.5C15.216 0 16 .784 16 1.75v7.5A1.75 1.75 0 0 1 14.25 11h-7.5A1.75 1.75 0 0 1 5 9.25Zm1.75-.25a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-7.5a.25.25 0 0 0-.25-.25Z"></path></svg></button>
  </div>
  <code class="language-c"><span class="line-number"></span><span class="code-hl">int main() {</span>
<span class="line-number"></span><span class="code-hl">  BarC bar{};</span>
<span class="line-number"></span><span class="code-hl">  const uint8_t input[] = {0, 1, 2, 3};</span>
<span class="line-number"></span><span class="code-hl">  bar_parse(input, sizeof(input), &amp;bar);</span>
<span class="line-number"></span><span class="code-hl">  assert(bar.x.ptr[0] == 0);</span>
<span class="line-number"></span><span class="code-hl">  assert(bar.x.ptr[1] == 1);</span>
<span class="line-number"></span><span class="code-hl">}</span>
</code></pre>
<p>And I get:</p>
<pre>
  <div class="code-header">
    <span>Plaintext</span>
    <button class="copy-code" type="button"><svg aria-hidden="true" focusable="false" class="octicon octicon-copy" viewBox="0 0 16 16" width="16" height="16" fill="currentColor" display="inline-block" overflow="visible" style="vertical-align: text-bottom;"><path d="M0 6.75C0 5.784.784 5 1.75 5h1.5a.75.75 0 0 1 0 1.5h-1.5a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-1.5a.75.75 0 0 1 1.5 0v1.5A1.75 1.75 0 0 1 9.25 16h-7.5A1.75 1.75 0 0 1 0 14.25Z"></path><path d="M5 1.75C5 .784 5.784 0 6.75 0h7.5C15.216 0 16 .784 16 1.75v7.5A1.75 1.75 0 0 1 14.25 11h-7.5A1.75 1.75 0 0 1 5 9.25Zm1.75-.25a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-7.5a.25.25 0 0 0-.25-.25Z"></path></svg></button>
  </div>
  <code class="language-plaintext"><span class="line-number"></span><span class="code-hl">==805913== Conditional jump or move depends on uninitialised value(s)</span>
<span class="line-number"></span><span class="code-hl">==805913==    at 0x127C34: main (src/example.cpp:13)</span>
<span class="line-number"></span><span class="code-hl">==805913== </span>
<span class="line-number"></span><span class="code-hl">==805913== Conditional jump or move depends on uninitialised value(s)</span>
<span class="line-number"></span><span class="code-hl">==805913==    at 0x127C69: main (src/example.cpp:14)</span>
</code></pre>
<p>Which is not very informative, but better than nothing. <code>Miri</code>'s output is much more actionable.</p>
<p>So in conclusion, Rust's FFI capabilities work but are tedious are error-prone in my opinion, and so require extra care and testing with Miri/fuzzing, with high code coverage of the FFI functions. It's not enough to only test the pure (non FFI) Rust code.</p>
<h3 id="another-example-of-a-real-bug-at-the-ffi-boundary">
  <a class="title" href="#another-example-of-a-real-bug-at-the-ffi-boundary">Another example of a real bug at the FFI boundary</a>
  <a class="hash-anchor" href="#another-example-of-a-real-bug-at-the-ffi-boundary" aria-hidden="true" onclick="navigator.clipboard.writeText(this.href);"></a>
</h3>
<p>When I started this rewrite, I was under the impression that the Rust standard library uses the C memory allocator (basically, <code>malloc</code>) under the covers when it needs to allocate some memory.</p>
<p>However, I quickly discovered that it is not (anymore?) the case, Rust uses its own allocator - at least on Linux where there is no C library shipping with the kernel.
Miri again is the MVP here since it detected the issue of mixing the C and Rust allocations which prompted this section.</p>
<p>As Bryan Cantrill once said: &quot;glibc on Linux, it's just, like, your opinion dude&quot;. Meaning, glibc is just one option, among many, since Linux is just the kernel and does not ship with a libC. So the Rust standard library cannot expect a given C library on every Linux system, like it would be on macOS or the BSDs or Illumos. All of that to say: Rust implements its own memory allocator.</p>
<p>The consequence of this, is that allocating memory on the C/C++ side, and freeing it on the Rust side, is undefined behavior: it amounts to freeing a pointer that was never allocated by this allocator. And vice-versa, allocating a pointer from Rust and freeing it from C.</p>
<p>That has dire consequences since most memory allocators do not detect this in release mode. You might free completely unrelated memory leading to use-after-free later, or corrupt the memory allocator structures. It's bad.</p>
<p>Here's a simplified example of code that triggered this issue:</p>
<pre>
  <div class="code-header">
    <span>Rust</span>
    <button class="copy-code" type="button"><svg aria-hidden="true" focusable="false" class="octicon octicon-copy" viewBox="0 0 16 16" width="16" height="16" fill="currentColor" display="inline-block" overflow="visible" style="vertical-align: text-bottom;"><path d="M0 6.75C0 5.784.784 5 1.75 5h1.5a.75.75 0 0 1 0 1.5h-1.5a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-1.5a.75.75 0 0 1 1.5 0v1.5A1.75 1.75 0 0 1 9.25 16h-7.5A1.75 1.75 0 0 1 0 14.25Z"></path><path d="M5 1.75C5 .784 5.784 0 6.75 0h7.5C15.216 0 16 .784 16 1.75v7.5A1.75 1.75 0 0 1 14.25 11h-7.5A1.75 1.75 0 0 1 5 9.25Zm1.75-.25a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-7.5a.25.25 0 0 0-.25-.25Z"></path></svg></button>
  </div>
  <code class="language-rust"><span class="line-number"></span><span class="code-hl">#[repr(C)]</span>
<span class="line-number"></span><span class="code-hl">pub struct FooC {</span>
<span class="line-number"></span><span class="code-hl">    foo: u8,</span>
<span class="line-number"></span><span class="code-hl">    bar: *mut usize,</span>
<span class="line-number"></span><span class="code-hl">}</span>
<span class="line-number"></span><span class="code-hl"></span>
<span class="line-number"></span><span class="code-hl">#[no_mangle]</span>
<span class="line-number"></span><span class="code-hl">pub extern &quot;C&quot; fn parse_foo(in_bytes: *const u8, in_bytes_len: usize, foo: &amp;mut FooC) {</span>
<span class="line-number"></span><span class="code-hl">    let in_bytes: &amp;[u8] = unsafe { &amp;*core::ptr::slice_from_raw_parts(in_bytes, in_bytes_len) };</span>
<span class="line-number"></span><span class="code-hl"></span>
<span class="line-number"></span><span class="code-hl">    // Parse `foo` from `in_bytes` but `bar` is sometimes not present in the payload.</span>
<span class="line-number"></span><span class="code-hl">    // In that case it is set manually by the calling code.</span>
<span class="line-number"></span><span class="code-hl">    *foo = FooC {</span>
<span class="line-number"></span><span class="code-hl">        foo: in_bytes[0],</span>
<span class="line-number"></span><span class="code-hl">        bar: if in_bytes_len == 1 {</span>
<span class="line-number"></span><span class="code-hl">            core::ptr::null_mut()</span>
<span class="line-number"></span><span class="code-hl">        } else {</span>
<span class="line-number"></span><span class="code-hl">            let x = Box::new(in_bytes[1] as usize);</span>
<span class="line-number"></span><span class="code-hl">            Box::into_raw(x)</span>
<span class="line-number"></span><span class="code-hl">        },</span>
<span class="line-number"></span><span class="code-hl">    }</span>
<span class="line-number"></span><span class="code-hl">}</span>
<span class="line-number"></span><span class="code-hl"></span>
<span class="line-number"></span><span class="code-hl">#[no_mangle]</span>
<span class="line-number"></span><span class="code-hl">pub extern &quot;C&quot; fn free_foo(foo: &amp;mut FooC) {</span>
<span class="line-number"></span><span class="code-hl">    if !foo.bar.is_null() {</span>
<span class="line-number"></span><span class="code-hl">        unsafe {</span>
<span class="line-number"></span><span class="code-hl">            let _ = Box::from_raw(foo.bar);</span>
<span class="line-number"></span><span class="code-hl">        }</span>
<span class="line-number"></span><span class="code-hl">    }</span>
<span class="line-number"></span><span class="code-hl">}</span>
</code></pre>
<p>And the calling code:</p>
<pre>
  <div class="code-header">
    <span>Cpp</span>
    <button class="copy-code" type="button"><svg aria-hidden="true" focusable="false" class="octicon octicon-copy" viewBox="0 0 16 16" width="16" height="16" fill="currentColor" display="inline-block" overflow="visible" style="vertical-align: text-bottom;"><path d="M0 6.75C0 5.784.784 5 1.75 5h1.5a.75.75 0 0 1 0 1.5h-1.5a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-1.5a.75.75 0 0 1 1.5 0v1.5A1.75 1.75 0 0 1 9.25 16h-7.5A1.75 1.75 0 0 1 0 14.25Z"></path><path d="M5 1.75C5 .784 5.784 0 6.75 0h7.5C15.216 0 16 .784 16 1.75v7.5A1.75 1.75 0 0 1 14.25 11h-7.5A1.75 1.75 0 0 1 5 9.25Zm1.75-.25a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-7.5a.25.25 0 0 0-.25-.25Z"></path></svg></button>
  </div>
  <code class="language-cpp"><span class="line-number"></span><span class="code-hl">FooC foo{};</span>
<span class="line-number"></span><span class="code-hl"></span>
<span class="line-number"></span><span class="code-hl">const uint8_t data[] = { 1 };</span>
<span class="line-number"></span><span class="code-hl">parse_foo(data, sizeof(data), &amp;foo);</span>
<span class="line-number"></span><span class="code-hl">if (foo.bar == nullptr) {</span>
<span class="line-number"></span><span class="code-hl">  foo.bar = new size_t{99999};</span>
<span class="line-number"></span><span class="code-hl">}</span>
<span class="line-number"></span><span class="code-hl"></span>
<span class="line-number"></span><span class="code-hl">free_foo(&amp;foo);</span>
</code></pre>
<p>This is undefined behavior if the array is of size 1, since in that case the Rust allocator will free a pointer allocated by the C allocator, and address sanitizer catches it:</p>
<pre>
  <div class="code-header">
    <span>Plaintext</span>
    <button class="copy-code" type="button"><svg aria-hidden="true" focusable="false" class="octicon octicon-copy" viewBox="0 0 16 16" width="16" height="16" fill="currentColor" display="inline-block" overflow="visible" style="vertical-align: text-bottom;"><path d="M0 6.75C0 5.784.784 5 1.75 5h1.5a.75.75 0 0 1 0 1.5h-1.5a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-1.5a.75.75 0 0 1 1.5 0v1.5A1.75 1.75 0 0 1 9.25 16h-7.5A1.75 1.75 0 0 1 0 14.25Z"></path><path d="M5 1.75C5 .784 5.784 0 6.75 0h7.5C15.216 0 16 .784 16 1.75v7.5A1.75 1.75 0 0 1 14.25 11h-7.5A1.75 1.75 0 0 1 5 9.25Zm1.75-.25a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-7.5a.25.25 0 0 0-.25-.25Z"></path></svg></button>
  </div>
  <code class="language-plaintext"><span class="line-number"></span><span class="code-hl">SUMMARY: AddressSanitizer: alloc-dealloc-mismatch /home/runner/work/llvm-project/llvm-project/final/llvm-project/compiler-rt/lib/asan/asan_malloc_linux.cpp:52:3 in free</span>
</code></pre>
<p>However, it is only detected with sanitizers on and if a test (or fuzzing) triggers this case. Or by Miri if a Rust test covers this function.</p>
<hr />
<p>So I recommend sticking to one 'side', be it C/C++ or Rust, of the FFI boundary, to allocate and free all the memory used in FFI structures. Rust has an edge here since the long-term goal is to have 100% of Rust so it will have to allocate all the memory anyway in the end.</p>
<p>Depending on the existing code style, it might be hard to ensure that the C/C++ allocator is not used at all for structures used in FFI, due to abstractions and hidden memory allocations.</p>
<p>One possible solution (which I did not implement but considered) is making FFI structures a simple opaque pointer (or 'handle') so that the caller has to use FFI functions to allocate and free this structure. That also means implementing getter/setters for certain fields since the structures are now opaque. It maximizes the ABI compatibility, since the caller cannot rely on a given struct size, alignment, or fields.</p>
<p>However that entails more work and more functions in the API.</p>
<p><code>libcurl</code> is an example of such an approach, <code>libuv</code> is an example of a library which did not do this initially, but plans to move to this approach in future versions, which would be a breaking change for clients.</p>
<p>So to summarize, Miri is so essential that I don't know whether it's viable to write Rust code with lots of FFI (and thus lots of unsafe blocks) without it. If Miri did not exist, I would seriously consider using only arenas or reconsider the use of Rust.</p>
<h2 id="cross-compilation">
  <a class="title" href="#cross-compilation">Cross-compilation</a>
  <a class="hash-anchor" href="#cross-compilation" aria-hidden="true" onclick="navigator.clipboard.writeText(this.href);"></a>
</h2>
<p>Rust has great cross-compilation support; C++ not so much. Nonetheless I managed to coerced CMake into cross-compiling to every platform we support from my Linux laptop. After using Docker for more than 10 years I am firmly against using Docker for that, it's just clunky and slow and not a good fit. Also we already have to cross-compile to the mobile platforms anyway so why not make that work for all platforms?</p>
<p>That way, I can even cross-compile tests and example programs in C or C++ using the library and run them inside <code>qemu</code> to make sure all platforms work as expected.</p>
<p>I took inspiration from the CMake code in the Android project, which has to cross-compile for many architectures. Did you know that Android supports x86 (which is 32 bits), x86_64, arm (which is 32 bits), aarch64 (sometimes called arm64), and more?</p>
<p>In short, you instruct CMake to cross-compile by supplying on the command-line the variables <code>CMAKE_SYSTEM_PROCESSOR</code> and <code>CMAKE_SYSTEM_NAME</code>, which are the equivalent of <code>GOARCH</code> and <code>GOOS</code> if you are familiar with Go. E.g.:</p>
<pre>
  <div class="code-header">
    <span>Shell</span>
    <button class="copy-code" type="button"><svg aria-hidden="true" focusable="false" class="octicon octicon-copy" viewBox="0 0 16 16" width="16" height="16" fill="currentColor" display="inline-block" overflow="visible" style="vertical-align: text-bottom;"><path d="M0 6.75C0 5.784.784 5 1.75 5h1.5a.75.75 0 0 1 0 1.5h-1.5a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-1.5a.75.75 0 0 1 1.5 0v1.5A1.75 1.75 0 0 1 9.25 16h-7.5A1.75 1.75 0 0 1 0 14.25Z"></path><path d="M5 1.75C5 .784 5.784 0 6.75 0h7.5C15.216 0 16 .784 16 1.75v7.5A1.75 1.75 0 0 1 14.25 11h-7.5A1.75 1.75 0 0 1 5 9.25Zm1.75-.25a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-7.5a.25.25 0 0 0-.25-.25Z"></path></svg></button>
  </div>
  <code class="language-shell"><span class="line-number"></span><span class="code-hl">$ cmake -B .build -S src -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ -DCMAKE_SYSTEM_NAME=Linux -DCMAKE_SYSTEM_PROCESSOR=arm</span>
</code></pre>
<p>On the Rust side, you tell <code>cargo</code> to cross-compile by supplying the <code>--target</code> command-line argument, e.g.: <code>--target=x86_64-unknown-linux-musl</code>. This works by virtue of installing the pre-compiled toolchain for this platform with <code>rustup</code> first:</p>
<pre>
  <div class="code-header">
    <span>Shell</span>
    <button class="copy-code" type="button"><svg aria-hidden="true" focusable="false" class="octicon octicon-copy" viewBox="0 0 16 16" width="16" height="16" fill="currentColor" display="inline-block" overflow="visible" style="vertical-align: text-bottom;"><path d="M0 6.75C0 5.784.784 5 1.75 5h1.5a.75.75 0 0 1 0 1.5h-1.5a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-1.5a.75.75 0 0 1 1.5 0v1.5A1.75 1.75 0 0 1 9.25 16h-7.5A1.75 1.75 0 0 1 0 14.25Z"></path><path d="M5 1.75C5 .784 5.784 0 6.75 0h7.5C15.216 0 16 .784 16 1.75v7.5A1.75 1.75 0 0 1 14.25 11h-7.5A1.75 1.75 0 0 1 5 9.25Zm1.75-.25a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-7.5a.25.25 0 0 0-.25-.25Z"></path></svg></button>
  </div>
  <code class="language-shell"><span class="line-number"></span><span class="code-hl">$ rustup target add x86_64-unknown-linux-musl</span>
</code></pre>
<p>So now we have to convert in CMake <code>CMAKE_SYSTEM_ARCHITECTURE</code> and <code>CMAKE_SYSTEM_NAME</code> into a target triple that clang and cargo can understand. Of course you have to do all the hard work yourself. This is complicated by lots of factors like Apple using the architecture name <code>arm64</code> instead of <code>aarch64</code>, iOS peculiarities, soft vs hard float, arm having multiple variants (v6, v7, v8, etc), and so on. Your mileage may vary. We opt-in into using musl with a CMake command line option, on Linux.</p>
<p>Here it is in all its glory:</p>
<pre>
  <div class="code-header">
    <span>Cmake</span>
    <button class="copy-code" type="button"><svg aria-hidden="true" focusable="false" class="octicon octicon-copy" viewBox="0 0 16 16" width="16" height="16" fill="currentColor" display="inline-block" overflow="visible" style="vertical-align: text-bottom;"><path d="M0 6.75C0 5.784.784 5 1.75 5h1.5a.75.75 0 0 1 0 1.5h-1.5a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-1.5a.75.75 0 0 1 1.5 0v1.5A1.75 1.75 0 0 1 9.25 16h-7.5A1.75 1.75 0 0 1 0 14.25Z"></path><path d="M5 1.75C5 .784 5.784 0 6.75 0h7.5C15.216 0 16 .784 16 1.75v7.5A1.75 1.75 0 0 1 14.25 11h-7.5A1.75 1.75 0 0 1 5 9.25Zm1.75-.25a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-7.5a.25.25 0 0 0-.25-.25Z"></path></svg></button>
  </div>
  <code class="language-cmake"><span class="line-number"></span><span class="code-hl"># We need to craft the target triple to make it work when cross-compiling.</span>
<span class="line-number"></span><span class="code-hl"># NOTE: If an architecture supports both soft-float and hard-float, we pick hard-float (`hf`).</span>
<span class="line-number"></span><span class="code-hl"># since we do not target any real hardware with soft-float.</span>
<span class="line-number"></span><span class="code-hl"># Linux has two main libcs, glibc (the default) and musl (opt-in with `FMW_LIBC_MUSL=1`), useful for Alpine.</span>
<span class="line-number"></span><span class="code-hl">if (CMAKE_SYSTEM_NAME STREQUAL &quot;Linux&quot; AND CMAKE_SYSTEM_PROCESSOR STREQUAL &quot;x86_64&quot; AND NOT DEFINED FMW_LIBC_MUSL)</span>
<span class="line-number"></span><span class="code-hl">    set(TARGET_TRIPLE &quot;x86_64-unknown-linux-gnu&quot;)</span>
<span class="line-number"></span><span class="code-hl">elseif (CMAKE_SYSTEM_NAME STREQUAL &quot;Linux&quot; AND CMAKE_SYSTEM_PROCESSOR STREQUAL &quot;x86_64&quot; AND &quot;${FMW_LIBC_MUSL}&quot; EQUAL 1)</span>
<span class="line-number"></span><span class="code-hl">    set(TARGET_TRIPLE &quot;x86_64-unknown-linux-musl&quot;)</span>
<span class="line-number"></span><span class="code-hl">elseif (CMAKE_SYSTEM_NAME STREQUAL &quot;Linux&quot; AND CMAKE_SYSTEM_PROCESSOR STREQUAL &quot;arm&quot; AND NOT DEFINED FMW_LIBC_MUSL)</span>
<span class="line-number"></span><span class="code-hl">    set(TARGET_TRIPLE &quot;arm-unknown-linux-gnueabihf&quot;)</span>
<span class="line-number"></span><span class="code-hl">elseif (CMAKE_SYSTEM_NAME STREQUAL &quot;Linux&quot; AND CMAKE_SYSTEM_PROCESSOR STREQUAL &quot;arm&quot; AND &quot;${FMW_LIBC_MUSL}&quot; EQUAL 1)</span>
<span class="line-number"></span><span class="code-hl">    set(TARGET_TRIPLE &quot;arm-unknown-linux-musleabihf&quot;)</span>
<span class="line-number"></span><span class="code-hl">elseif (CMAKE_SYSTEM_NAME STREQUAL &quot;Linux&quot; AND CMAKE_SYSTEM_PROCESSOR STREQUAL &quot;aarch64&quot; AND NOT DEFINED FMW_LIBC_MUSL)</span>
<span class="line-number"></span><span class="code-hl">    set(TARGET_TRIPLE &quot;aarch64-unknown-linux-gnu&quot;)</span>
<span class="line-number"></span><span class="code-hl">elseif (CMAKE_SYSTEM_NAME STREQUAL &quot;Linux&quot; AND CMAKE_SYSTEM_PROCESSOR STREQUAL &quot;aarch64&quot; AND &quot;${FMW_LIBC_MUSL}&quot; EQUAL 1)</span>
<span class="line-number"></span><span class="code-hl">    set(TARGET_TRIPLE &quot;aarch64-unknown-linux-musl&quot;)</span>
<span class="line-number"></span><span class="code-hl">elseif (CMAKE_SYSTEM_NAME STREQUAL &quot;Linux&quot; AND CMAKE_SYSTEM_PROCESSOR STREQUAL &quot;armv7&quot;)</span>
<span class="line-number"></span><span class="code-hl">    set(TARGET_TRIPLE &quot;armv7-unknown-linux-gnueabihf&quot;)</span>
<span class="line-number"></span><span class="code-hl">elseif (CMAKE_SYSTEM_NAME STREQUAL &quot;Darwin&quot; AND CMAKE_SYSTEM_PROCESSOR STREQUAL &quot;aarch64&quot;)</span>
<span class="line-number"></span><span class="code-hl">    set(TARGET_TRIPLE &quot;aarch64-apple-darwin&quot;)</span>
<span class="line-number"></span><span class="code-hl">elseif (CMAKE_SYSTEM_NAME STREQUAL &quot;Darwin&quot; AND CMAKE_SYSTEM_PROCESSOR STREQUAL &quot;arm64&quot;)</span>
<span class="line-number"></span><span class="code-hl">    set(TARGET_TRIPLE &quot;aarch64-apple-darwin&quot;)</span>
<span class="line-number"></span><span class="code-hl">elseif (CMAKE_SYSTEM_NAME STREQUAL &quot;Darwin&quot; AND CMAKE_SYSTEM_PROCESSOR STREQUAL &quot;x86_64&quot;)</span>
<span class="line-number"></span><span class="code-hl">    set(TARGET_TRIPLE &quot;x86_64-apple-darwin&quot;)</span>
<span class="line-number"></span><span class="code-hl">elseif (CMAKE_SYSTEM_NAME STREQUAL &quot;iOS&quot; AND CMAKE_SYSTEM_PROCESSOR STREQUAL &quot;x86_64&quot;)</span>
<span class="line-number"></span><span class="code-hl">    set(TARGET_TRIPLE &quot;x86_64-apple-ios&quot;)</span>
<span class="line-number"></span><span class="code-hl">    execute_process(COMMAND xcrun --sdk iphonesimulator --show-sdk-path OUTPUT_VARIABLE CMAKE_OSX_SYSROOT)</span>
<span class="line-number"></span><span class="code-hl">    string(REPLACE &quot;\n&quot; &quot;&quot; CMAKE_OSX_SYSROOT ${CMAKE_OSX_SYSROOT})</span>
<span class="line-number"></span><span class="code-hl">elseif (CMAKE_SYSTEM_NAME STREQUAL &quot;iOS&quot; AND CMAKE_SYSTEM_PROCESSOR STREQUAL &quot;aarch64&quot;)</span>
<span class="line-number"></span><span class="code-hl">    set(TARGET_TRIPLE &quot;aarch64-apple-ios&quot;)</span>
<span class="line-number"></span><span class="code-hl">    execute_process(COMMAND xcrun --sdk iphoneos --show-sdk-path OUTPUT_VARIABLE CMAKE_OSX_SYSROOT)</span>
<span class="line-number"></span><span class="code-hl">    string(REPLACE &quot;\n&quot; &quot;&quot; CMAKE_OSX_SYSROOT ${CMAKE_OSX_SYSROOT})</span>
<span class="line-number"></span><span class="code-hl">elseif (CMAKE_SYSTEM_NAME STREQUAL &quot;iOS&quot; AND CMAKE_SYSTEM_PROCESSOR STREQUAL &quot;arm64&quot;)</span>
<span class="line-number"></span><span class="code-hl">    set(TARGET_TRIPLE &quot;aarch64-apple-ios&quot;)</span>
<span class="line-number"></span><span class="code-hl">    execute_process(COMMAND xcrun --sdk iphoneos --show-sdk-path OUTPUT_VARIABLE CMAKE_OSX_SYSROOT)</span>
<span class="line-number"></span><span class="code-hl">    string(REPLACE &quot;\n&quot; &quot;&quot; CMAKE_OSX_SYSROOT ${CMAKE_OSX_SYSROOT})</span>
<span class="line-number"></span><span class="code-hl">elseif (CMAKE_SYSTEM_NAME STREQUAL &quot;Android&quot; AND CMAKE_SYSTEM_PROCESSOR STREQUAL &quot;arm&quot;)</span>
<span class="line-number"></span><span class="code-hl">    set(TARGET_TRIPLE &quot;arm-linux-androideabi&quot;)</span>
<span class="line-number"></span><span class="code-hl">elseif (CMAKE_SYSTEM_NAME STREQUAL &quot;Android&quot; AND CMAKE_SYSTEM_PROCESSOR STREQUAL &quot;armv7&quot;)</span>
<span class="line-number"></span><span class="code-hl">    set(TARGET_TRIPLE &quot;armv7-linux-androideabi&quot;)</span>
<span class="line-number"></span><span class="code-hl">elseif (CMAKE_SYSTEM_NAME STREQUAL &quot;Android&quot; AND CMAKE_SYSTEM_PROCESSOR STREQUAL &quot;armv7-a&quot;)</span>
<span class="line-number"></span><span class="code-hl">    set(TARGET_TRIPLE &quot;armv7-linux-androideabi&quot;)</span>
<span class="line-number"></span><span class="code-hl">elseif (CMAKE_SYSTEM_NAME STREQUAL &quot;Android&quot; AND CMAKE_SYSTEM_PROCESSOR STREQUAL &quot;aarch64&quot;)</span>
<span class="line-number"></span><span class="code-hl">    set(TARGET_TRIPLE &quot;aarch64-linux-android&quot;)</span>
<span class="line-number"></span><span class="code-hl">elseif (CMAKE_SYSTEM_NAME STREQUAL &quot;Android&quot; AND CMAKE_SYSTEM_PROCESSOR STREQUAL &quot;i686&quot;)</span>
<span class="line-number"></span><span class="code-hl">    set(TARGET_TRIPLE &quot;i686-linux-android&quot;)</span>
<span class="line-number"></span><span class="code-hl">elseif (CMAKE_SYSTEM_NAME STREQUAL &quot;Android&quot; AND CMAKE_SYSTEM_PROCESSOR STREQUAL &quot;x86_64&quot;)</span>
<span class="line-number"></span><span class="code-hl">    set(TARGET_TRIPLE &quot;x86_64-linux-android&quot;)</span>
<span class="line-number"></span><span class="code-hl">else()</span>
<span class="line-number"></span><span class="code-hl">    message(FATAL_ERROR &quot;Invalid OS/Architecture, not supported: CMAKE_SYSTEM_NAME=${CMAKE_SYSTEM_NAME} CMAKE_SYSTEM_PROCESSOR=${CMAKE_SYSTEM_PROCESSOR}&quot;)</span>
<span class="line-number"></span><span class="code-hl">endif()</span>
<span class="line-number"></span><span class="code-hl"></span>
<span class="line-number"></span><span class="code-hl">message(STATUS &quot;Target triple: ${TARGET_TRIPLE}&quot;)</span>
<span class="line-number"></span><span class="code-hl"></span>
<span class="line-number"></span><span class="code-hl"># If we are cross compiling manually (e.g to Linux arm), `CMAKE_C_COMPILER_TARGET` and `CMAKE_CXX_COMPILER_TARGET` are unset and we need to set them manually.</span>
<span class="line-number"></span><span class="code-hl"># But if we are cross compiling through a separate build system e.g. to Android or iOS, they will set these variables and we should not override them.</span>
<span class="line-number"></span><span class="code-hl">if ( NOT DEFINED CMAKE_C_COMPILER_TARGET )</span>
<span class="line-number"></span><span class="code-hl">   set(CMAKE_C_COMPILER_TARGET ${TARGET_TRIPLE})</span>
<span class="line-number"></span><span class="code-hl">endif()</span>
<span class="line-number"></span><span class="code-hl">if ( NOT DEFINED CMAKE_CXX_COMPILER_TARGET )</span>
<span class="line-number"></span><span class="code-hl">   set(CMAKE_CXX_COMPILER_TARGET ${TARGET_TRIPLE})</span>
<span class="line-number"></span><span class="code-hl">endif()</span>
</code></pre>
<p>There was a lot of trial and error as you can guess.</p>
<p>Also, gcc is not directly supported for cross-compilation in this approach because gcc does not support a <code>--target</code> option like clang does, since it's not a cross-compiler. You have to download the variant you need e.g. <code>gcc-9-i686-linux-gnu</code> to compile for x86, and set <code>CMAKE_C_COMPILER</code> and <code>CMAKE_CXX_COMPILER</code> to <code>gcc-9-i686-linux-gnu</code>. However, in that case you are not setting <code>CMAKE_SYSTEM_NAME</code> and <code>CMAKE_SYSTEM_PROCESSOR</code> since it's in theory not cross-compiling, so <code>cargo</code> will not have its <code>--target</code> option filled, so it won't work for the Rust code. I advise sticking with clang in this setup. Still, when not cross-compiling, gcc works fine.</p>
<p>Finally, I wrote a Lua script to cross-compile for every platform we support to make sure I did not break anything. I resorted to using the Zig toolchain (not the language) to be able to statically link with musl or cross-compile from Linux to iOS which I could not achieve with pure clang. However this is only my local setup, we do not use the Zig toolchain when building the production artifacts (e.g. the iOS build is done in a macOS virtual machine, not from a Linux machine).</p>
<p>This is very useful also if you have several compile-time feature flags and want to build in different configurations for all platforms, e.g. enable/disable logs at compile time:</p>
<pre>
  <div class="code-header">
    <span>Lua</span>
    <button class="copy-code" type="button"><svg aria-hidden="true" focusable="false" class="octicon octicon-copy" viewBox="0 0 16 16" width="16" height="16" fill="currentColor" display="inline-block" overflow="visible" style="vertical-align: text-bottom;"><path d="M0 6.75C0 5.784.784 5 1.75 5h1.5a.75.75 0 0 1 0 1.5h-1.5a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-1.5a.75.75 0 0 1 1.5 0v1.5A1.75 1.75 0 0 1 9.25 16h-7.5A1.75 1.75 0 0 1 0 14.25Z"></path><path d="M5 1.75C5 .784 5.784 0 6.75 0h7.5C15.216 0 16 .784 16 1.75v7.5A1.75 1.75 0 0 1 14.25 11h-7.5A1.75 1.75 0 0 1 5 9.25Zm1.75-.25a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-7.5a.25.25 0 0 0-.25-.25Z"></path></svg></button>
  </div>
  <code class="language-lua"><span class="line-number"></span><span class="code-hl">local android_sdk = arg[1]</span>
<span class="line-number"></span><span class="code-hl">if android_sdk == nil or android_sdk == &quot;&quot; then</span>
<span class="line-number"></span><span class="code-hl">  print(&quot;Missing Android SDK as argv[1] e.g. '~/Android/Sdk/ndk/21.4.7075529'.&quot;)</span>
<span class="line-number"></span><span class="code-hl">  os.exit(1)</span>
<span class="line-number"></span><span class="code-hl">end</span>
<span class="line-number"></span><span class="code-hl"></span>
<span class="line-number"></span><span class="code-hl">local build_root = arg[2]</span>
<span class="line-number"></span><span class="code-hl">if build_root == nil then</span>
<span class="line-number"></span><span class="code-hl">  build_root = &quot;/tmp/&quot;</span>
<span class="line-number"></span><span class="code-hl">end</span>
<span class="line-number"></span><span class="code-hl"></span>
<span class="line-number"></span><span class="code-hl">local rustup_targets = {</span>
<span class="line-number"></span><span class="code-hl">  &quot;aarch64-apple-darwin&quot;,</span>
<span class="line-number"></span><span class="code-hl">  &quot;aarch64-linux-android&quot;,</span>
<span class="line-number"></span><span class="code-hl">  &quot;aarch64-unknown-linux-gnu&quot;,</span>
<span class="line-number"></span><span class="code-hl">  &quot;aarch64-unknown-linux-musl&quot;,</span>
<span class="line-number"></span><span class="code-hl">  &quot;arm-linux-androideabi&quot;,</span>
<span class="line-number"></span><span class="code-hl">  &quot;arm-unknown-linux-gnueabihf&quot;,</span>
<span class="line-number"></span><span class="code-hl">  &quot;arm-unknown-linux-musleabihf&quot;,</span>
<span class="line-number"></span><span class="code-hl">  &quot;armv7-linux-androideabi&quot;,</span>
<span class="line-number"></span><span class="code-hl">  &quot;armv7-unknown-linux-gnueabi&quot;,</span>
<span class="line-number"></span><span class="code-hl">  &quot;armv7-unknown-linux-gnueabihf&quot;,</span>
<span class="line-number"></span><span class="code-hl">  &quot;armv7-unknown-linux-musleabi&quot;,</span>
<span class="line-number"></span><span class="code-hl">  &quot;armv7-unknown-linux-musleabihf&quot;,</span>
<span class="line-number"></span><span class="code-hl">  &quot;i686-linux-android&quot;,</span>
<span class="line-number"></span><span class="code-hl">  &quot;x86_64-apple-darwin&quot;,</span>
<span class="line-number"></span><span class="code-hl">  &quot;x86_64-linux-android&quot;,</span>
<span class="line-number"></span><span class="code-hl">  &quot;x86_64-unknown-linux-gnu&quot;,</span>
<span class="line-number"></span><span class="code-hl">  &quot;x86_64-unknown-linux-musl&quot;,</span>
<span class="line-number"></span><span class="code-hl">}</span>
<span class="line-number"></span><span class="code-hl"></span>
<span class="line-number"></span><span class="code-hl">for i = 1,#rustup_targets do</span>
<span class="line-number"></span><span class="code-hl">  local target = rustup_targets[i]</span>
<span class="line-number"></span><span class="code-hl">  os.execute(&quot;rustup target install &quot; .. target)</span>
<span class="line-number"></span><span class="code-hl">end</span>
<span class="line-number"></span><span class="code-hl"></span>
<span class="line-number"></span><span class="code-hl"></span>
<span class="line-number"></span><span class="code-hl">local targets = {</span>
<span class="line-number"></span><span class="code-hl">  {os=&quot;Linux&quot;, arch=&quot;x86_64&quot;, cc=&quot;clang&quot;, cxx=&quot;clang++&quot;, cmakeArgs=&quot;&quot;},</span>
<span class="line-number"></span><span class="code-hl">  {os=&quot;Linux&quot;, arch=&quot;aarch64&quot;, cc=&quot;clang&quot;, cxx=&quot;clang++&quot;, cmakeArgs=&quot;&quot;},</span>
<span class="line-number"></span><span class="code-hl">  {os=&quot;Linux&quot;, arch=&quot;arm&quot;, cc=&quot;clang&quot;, cxx=&quot;clang++&quot;, cmakeArgs=&quot;&quot;},</span>
<span class="line-number"></span><span class="code-hl">  {os=&quot;Linux&quot;, arch=&quot;armv7&quot;, cc=&quot;clang&quot;, cxx=&quot;clang++&quot;, cmakeArgs=&quot;&quot;},</span>
<span class="line-number"></span><span class="code-hl">  {os=&quot;Linux&quot;, arch=&quot;arm&quot;, cc=&quot;zig&quot;, cxx=&quot;zig&quot;, cmakeArgs=&quot;-DCMAKE_C_COMPILER_ARG1=cc -DCMAKE_CXX_COMPILER_ARG1=c++ -DFMW_LIBC_MUSL=1 -DCMAKE_C_COMPILER_TARGET=arm-linux-musleabihf -DCMAKE_CXX_COMPILER_TARGET=arm-linux-musleabihf&quot;},</span>
<span class="line-number"></span><span class="code-hl">  {os=&quot;Linux&quot;, arch=&quot;aarch64&quot;, cc=&quot;zig&quot;, cxx=&quot;zig&quot;, cmakeArgs=&quot;-DCMAKE_C_COMPILER_ARG1=cc -DCMAKE_CXX_COMPILER_ARG1=c++ -DFMW_LIBC_MUSL=1 -DCMAKE_C_COMPILER_TARGET=aarch64-linux-musl -DCMAKE_CXX_COMPILER_TARGET=aarch64-linux-musl&quot;},</span>
<span class="line-number"></span><span class="code-hl">  {os=&quot;Linux&quot;, arch=&quot;x86_64&quot;, cc=&quot;zig&quot;, cxx=&quot;zig&quot;, cmakeArgs=&quot;-DCMAKE_C_COMPILER_ARG1=cc -DCMAKE_CXX_COMPILER_ARG1=c++ -DFMW_LIBC_MUSL=1 -DCMAKE_C_COMPILER_TARGET=x86_64-linux-musl -DCMAKE_CXX_COMPILER_TARGET=x86_64-linux-musl&quot;},</span>
<span class="line-number"></span><span class="code-hl">  {os=&quot;Darwin&quot;, arch=&quot;x86_64&quot;, cc=&quot;zig&quot;, cxx=&quot;zig&quot;, cmakeArgs=&quot;-DCMAKE_C_COMPILER_ARG1=cc -DCMAKE_CXX_COMPILER_ARG1=c++ -DFMW_LIBC_MUSL=1 -DCMAKE_C_COMPILER_TARGET=x86_64-macos-none -DCMAKE_CXX_COMPILER_TARGET=x86_64-macos-none&quot;},</span>
<span class="line-number"></span><span class="code-hl">  {os=&quot;Darwin&quot;, arch=&quot;arm64&quot;, cc=&quot;zig&quot;, cxx=&quot;zig&quot;, cmakeArgs=&quot;-DCMAKE_C_COMPILER_ARG1=cc -DCMAKE_CXX_COMPILER_ARG1=c++ -DFMW_LIBC_MUSL=1 -DCMAKE_C_COMPILER_TARGET=aarch64-macos-none -DCMAKE_CXX_COMPILER_TARGET=aarch64-macos-none&quot;},</span>
<span class="line-number"></span><span class="code-hl">  {os=&quot;Android&quot;, arch=&quot;armv7-a&quot;, cc=&quot;clang&quot;, cxx=&quot;clang++&quot;, cmakeArgs=&quot;-DCMAKE_ANDROID_NDK='&quot; .. android_sdk .. &quot;'&quot;},</span>
<span class="line-number"></span><span class="code-hl">  {os=&quot;Android&quot;, arch=&quot;aarch64&quot;, cc=&quot;clang&quot;, cxx=&quot;clang++&quot;, cmakeArgs=&quot;-DCMAKE_ANDROID_NDK='&quot; .. android_sdk .. &quot;'&quot;},</span>
<span class="line-number"></span><span class="code-hl">  {os=&quot;Android&quot;, arch=&quot;i686&quot;, cc=&quot;clang&quot;, cxx=&quot;clang++&quot;, cmakeArgs=&quot;-DCMAKE_ANDROID_NDK='&quot; .. android_sdk .. &quot;'&quot;},</span>
<span class="line-number"></span><span class="code-hl">  {os=&quot;Android&quot;, arch=&quot;x86_64&quot;, cc=&quot;clang&quot;, cxx=&quot;clang++&quot;, cmakeArgs=&quot;-DCMAKE_ANDROID_NDK='&quot; .. android_sdk .. &quot;'&quot;},</span>
<span class="line-number"></span><span class="code-hl">}</span>
<span class="line-number"></span><span class="code-hl"></span>
<span class="line-number"></span><span class="code-hl">for i = 1,#targets do</span>
<span class="line-number"></span><span class="code-hl">  local target = targets[i]</span>
<span class="line-number"></span><span class="code-hl">  local build_dir = &quot;.build-&quot; .. target.os .. &quot;-&quot; .. target.arch .. &quot;-&quot; .. target.cc .. &quot;-&quot; .. target.cxx .. &quot;-&quot; .. target.cmakeArgs</span>
<span class="line-number"></span><span class="code-hl">  build_dir = string.gsub(build_dir, &quot;%s+&quot;, &quot;_&quot;)</span>
<span class="line-number"></span><span class="code-hl">  build_dir = string.gsub(build_dir, &quot;^./+&quot;, &quot;_&quot;)</span>
<span class="line-number"></span><span class="code-hl">  build_dir = build_root .. &quot;/&quot; .. build_dir</span>
<span class="line-number"></span><span class="code-hl">  print(build_dir)</span>
<span class="line-number"></span><span class="code-hl"></span>
<span class="line-number"></span><span class="code-hl">  local cmd_handle = io.popen(&quot;command -v llvm-ar&quot;)</span>
<span class="line-number"></span><span class="code-hl">  local llvm_ar = cmd_handle:read('*a')</span>
<span class="line-number"></span><span class="code-hl">  cmd_handle:close()</span>
<span class="line-number"></span><span class="code-hl">  llvm_ar = string.gsub(llvm_ar, &quot;%s+$&quot;, &quot;&quot;)</span>
<span class="line-number"></span><span class="code-hl"></span>
<span class="line-number"></span><span class="code-hl">  local cmd_handle = io.popen(&quot;command -v llvm-ranlib&quot;)</span>
<span class="line-number"></span><span class="code-hl">  local llvm_ranlib = cmd_handle:read('*a')</span>
<span class="line-number"></span><span class="code-hl">  cmd_handle:close()</span>
<span class="line-number"></span><span class="code-hl">  llvm_ranlib = string.gsub(llvm_ranlib, &quot;%s+$&quot;, &quot;&quot;)</span>
<span class="line-number"></span><span class="code-hl"></span>
<span class="line-number"></span><span class="code-hl">  local build_cmd = &quot;cmake -DCMAKE_BUILD_TYPE=RelWithDebInfo -B '&quot; .. build_dir .. &quot;' -DCMAKE_AR=&quot; .. llvm_ar .. &quot; -DCMAKE_RANLIB=&quot; .. llvm_ranlib .. &quot; -DCMAKE_SYSTEM_NAME=&quot; .. target.os .. &quot; -DCMAKE_SYSTEM_PROCESSOR=&quot; .. target.arch .. &quot; -DCMAKE_C_COMPILER=&quot; .. target.cc .. &quot; -DCMAKE_CXX_COMPILER=&quot; .. target.cxx .. &quot; &quot; ..  target.cmakeArgs .. &quot; -S src/. -G Ninja&quot;</span>
<span class="line-number"></span><span class="code-hl">  print(build_cmd)</span>
<span class="line-number"></span><span class="code-hl">  os.execute(build_cmd)</span>
<span class="line-number"></span><span class="code-hl"></span>
<span class="line-number"></span><span class="code-hl">  -- Work-around for getting rid of mbedtls linker flags specific to Apple's LLVM fork that are actually not needed.</span>
<span class="line-number"></span><span class="code-hl">  if target.os == &quot;Darwin&quot; then</span>
<span class="line-number"></span><span class="code-hl">    os.execute(&quot;sed -i '&quot; .. build_dir .. &quot;/CMakeFiles/rules.ninja' -e 's/ -no_warning_for_no_symbols -c//g'&quot;)</span>
<span class="line-number"></span><span class="code-hl">  end</span>
<span class="line-number"></span><span class="code-hl"></span>
<span class="line-number"></span><span class="code-hl">  os.execute(&quot;ninja -C '&quot; .. build_dir .. &quot;'&quot;)</span>
<span class="line-number"></span><span class="code-hl">end</span>
</code></pre>
<p>I look forward to only having Rust code and deleting all of this convoluted stuff.</p>
<p>That's something that people do not mention often when saying that modern C++ is good enough and secure enough. Well, first I disagree with this statement, but more broadly, the C++ toolchain to cross-compile sucks. You only have clang that can cross-compile in theory but in practice you have to resort to the Zig toolchain to automate cross-compiling the standard library etc.</p>
<p>Also, developers not deeply familiar with either C or C++ do not want to touch all this CMake/Autotools with a ten-foot pole. And I understand them. Stockholm syndrome notwithstanding, these are pretty slow, convoluted, niche programming languages and no one wants to actively learn and use them unless they have to.</p>
<p>Once you are used to simply typing <code>go build</code> or <code>cargo build</code>, you really start to ask yourself if those weird things are worth anyone's time.</p>
<h2 id="conclusion">
  <a class="title" href="#conclusion">Conclusion</a>
  <a class="hash-anchor" href="#conclusion" aria-hidden="true" onclick="navigator.clipboard.writeText(this.href);"></a>
</h2>
<p>The rewrite is not yet fully done, but we have already more Rust code than C++ code, and it's moving along nicely, at our own pace (it's not by far the only project we have on our lap). Once all C++ code is removed, we will do a final pass to remove the CMake stuff and build directly via <code>cargo</code>. We'll see if that works when integrating with other build systems e.g. Bazel for Android or Xcode for iOS.</p>
<p>Developers who learned Rust are overall very happy with it and did not have too many fights with the borrow checker, with one notable exception of trying to migrate a C struct that used an intrusive linked list (ah, the dreaded linked list v. borrow checker!). My suggestion was to simply use a <code>Vec</code> in Rust since the linked list was not really justified here, and the problem was solved.</p>
<p>Adding unit tests was trivial in Rust compared to C++ and as a result people would write a lot more of them. Built-in support for tests is expected in 2024 by developers. I don't think one single C++ test was written during this rewrite, now that I think of it.</p>
<p>Everyone was really satisfied with the tooling, even though having to first do <code>rustup target add ...</code> before cross-compiling tripped up a few people, since in Go that's done automatically behind the scenes (I think one difference is that Go compiles everything from source and so does not need to download pre-compiled blobs?).</p>
<p>Everyone also had an easy time with their text editor/IDE, Rust is ubiquitous enough now that every editor will have support for it.</p>
<p>All the tooling we needed to scan dependencies for vulnerabilities, linting, etc was present and polished. Shootout to <code>osv-scanner</code> from Google, which allowed us to scan both the Rust and C++ dependencies in the same project (and it evens supports Go).</p>
<p>As expected, developers migrating C++ code to Rust code had a breeze with the Rust code and almost every time asked for assistance when dealing with the C++ code. C++ is just too complex a language for most developers, especially compared to its alternatives.</p>
<p>CMake/Make/Ninja proved surprisingly difficult for developers not accustomed to them, but I mentioned that already. I think half of my time during this rewrite was actually spent coercing all the various build systems (Bazel/Xcode/CMake/cargo/Go) on the various platforms into working well together. If there is no one in the team who's really familiar with build systems, I think this is going to be a real challenge.</p>
<p>So, I hope this article alleviated your concerns about rewriting your C++ codebase. It can absolutely be done, just pick the right programming language for you and your context, do it incrementally, don't overpromise, establish a rough roadmap with milestones, regularly show progress to stakeholders (even if it's just you, it helps staying motivated!), and make sure the team is on-board and enjoying the process.</p>
<p>You know, like any other software project, really!</p>
<p><a href="/blog"> ⏴ Back to all articles</a></p>

<div id="donate">
  <em>
  <p>If you enjoy what you're reading, you want to support me, and can afford it: <a href="https://paypal.me/philigaultier?country.x=DE&locale.x=en_US">Support me</a>. That allows me to write more cool articles!</p>

  <p>
    This blog is <a href="https://github.com/gaultier/blog">open-source</a>!
    If you find a problem, please open a Github issue.
    The content of this blog as well as the code snippets are under the <a href="https://en.wikipedia.org/wiki/BSD_licenses#3-clause_license_(%22BSD_License_2.0%22,_%22Revised_BSD_License%22,_%22New_BSD_License%22,_or_%22Modified_BSD_License%22)">BSD-3 License</a> which I also usually use for all my personal projects. It's basically free for every use but you have to mention me as the original author.
  </p>
  </em>
</div>

</div>
</body>
</html>