Any suggest to couple two functions?

[HShan886]

Hello,
My binary has two functions without return instruction, that's the functions depend on each-other. After bolt, the two functions were splited into separate location. Therefore, how to emit the functions together?
Thank you firstly.

[yavtuk]

Hi @Haishan312, try to add -lite=0 to the list of bolt option

[HShan886]

Hi @Haishan312, try to add -lite=0 to the list of bolt option

Hi, @yavtuk This option doesn't work

[yavtuk]

Can you add simple example and some details what you are doing and what you are waiting ?

[rafaelauler]

As a mitigation, try --lite=1 --skip-functions=funcA,funcB

where funcA and funcB are these two special functions.

Ideally BOLT heuristics would detect that funcB is not another function, but instead a secondary entry point of funcA, and move these two together. Somehow this is failing for your input, so I agree with @yavtuk in the sense that it would be nice to have a test case.

[HShan886]

@yavtuk @rafaelauler thank you for replying. assembly code of the two functions shown as follow:

0000000004c684c4 <ec_encode_data_mbinit>:
 4c684c4:       a9b27bfd        stp     x29, x30, [sp, #-224]!
 4c684c8:       a90127e8        stp     x8, x9, [sp, #16]
 4c684cc:       a90207e0        stp     x0, x1, [sp, #32]
 4c684d0:       a9030fe2        stp     x2, x3, [sp, #48]
 4c684d4:       a90417e4        stp     x4, x5, [sp, #64]
 4c684d8:       a9051fe6        stp     x6, x7, [sp, #80]
 4c684dc:       ad0307e0        stp     q0, q1, [sp, #96]
 4c684e0:       ad040fe2        stp     q2, q3, [sp, #128]
 4c684e4:       ad0517e4        stp     q4, q5, [sp, #160]
 4c684e8:       ad061fe6        stp     q6, q7, [sp, #192]
 4c684ec:       94000091        bl      4c68730 <ec_encode_data_dispatcher>
 4c684f0:       a94127e8        ldp     x8, x9, [sp, #16]
 4c684f4:       f9000120        str     x0, [x9]
 4c684f8:       a94207e0        ldp     x0, x1, [sp, #32]
 4c684fc:       a9430fe2        ldp     x2, x3, [sp, #48]
 4c68500:       a94417e4        ldp     x4, x5, [sp, #64]
 4c68504:       a9451fe6        ldp     x6, x7, [sp, #80]
 4c68508:       ad4307e0        ldp     q0, q1, [sp, #96]
 4c6850c:       ad440fe2        ldp     q2, q3, [sp, #128]
 4c68510:       ad4517e4        ldp     q4, q5, [sp, #160]
 4c68514:       ad461fe6        ldp     q6, q7, [sp, #192]
 4c68518:       a8ce7bfd        ldp     x29, x30, [sp], #224

0000000004c6851c <ec_encode_data>:
 4c6851c:       9000d689        adrp    x9, 6738000 <_ZTVN5river12IoStatPartPBE@@Base+0x104120>
 4c68520:       f9450d29        ldr     x9, [x9, #2584]
 4c68524:       f940012a        ldr     x10, [x9]
 4c68528:       d61f0140        br      x10   --> call ec_encode_data_mbinit

BOLT will put ec_encode_data_mbinit into other location. Then binary result in segment fault.

[rafaelauler]

For the case that you posted, it mostly depends on what the ELF symbol table is telling us. The compiler will probably not generate this code (let me know if that's not the case), so this is likely code written directly in assembly language. For these cases, the programmer needs to correctly classify the symbol types. If "ec_encode_data" is registered as being type "function" (ELF STT_FUNC symbol type), BOLT will think that it is a separate function and will try to move it independently.

If it is registered as NOTYPE, though, BOLT will consider that the previous function (ec_encode_data_mbinit) contains this code as well, and will correctly move these two pieces together.

In a high level language, you cannot rely on function layout. You can't write code that somehow falls back to the next function in the hope that the next function is always going to be there. The linker is free to move functions around as long as you build with -ffunction-sections. BOLT leverages this assumption to freely shuffle functions around without breaking the ABI or the language contracts. If this code was built with -ffunction-sections, it would like break in the linker too, not just BOLT.

Whenever writing code in assembly language, it is unlikely that you will adhere to the same rigorous standards that a compiler has in terms of correctly emitting all necessary metadata. These cases are challenging for BOLT, but we do support this if ec_encode_data is not explictly registered as a function in the ELF sym tab. Unfortunately, if the ELF symbol table is saying it is a function, we have to take this statement at face value and assume this code is layout independent.

As a workaround, if you can't fix the source code, the only way to make BOLT ingest this is by adding these symbols to --skip-functions.

Let me know what does "readelf -Ws your_binary_name | grep ec_encode_data" says for this binary, as well as "readelf -Ws your_binary_name | grep ec_encode_data_mbinit", to confirm that I interpreted correctly your case.

[HShan886]

@rafaelauler you are right. These code is written by assembly language. Moveover, these functions are open source and update long ago.
172679: 0000000004c684c4 0 NOTYPE LOCAL DEFAULT 13 ec_encode_data_mbinit
417248: 0000000004c6851c 16 FUNC GLOBAL DEFAULT 13 ec_encode_data

[HShan886]

@rafaelauler I try --lite=1 --skip-functions=ec_encode_data_mbinit,ec_encode_data option, but get a UNREACHABLE error. And stack shows:

#0  0x0000fffff7cb4a98 in raise () from /lib64/libc.so.6
#1  0x0000fffff7ca150c in abort () from /lib64/libc.so.6
#2  0x0000000000b22e2c in llvm::llvm_unreachable_internal (msg=0x39c83a8 "not implemented",
    file=0x39c8360 "/tmp//work/llvm-swf/bolt/include/bolt/Core/MCPlusBuilder.h", line=1578)
    at /tmp//work/llvm-swf/llvm/lib/Support/ErrorHandling.cpp:212
#3  0x00000000021687c4 in llvm::bolt::MCPlusBuilder::createRelocation (this=0x4dc66b0, Fixup=..., MAB=...)
    at /tmp//work/llvm-swf/bolt/include/bolt/Core/MCPlusBuilder.h:1578
#4  0x00000000021bd6d8 in llvm::bolt::BinaryFunction::scanExternalRefs (this=0x267faf08)
    at /tmp//work/llvm-swf/bolt/lib/Core/BinaryFunction.cpp:1562
#5  0x0000000000c6deb0 in llvm::bolt::RewriteInstance::disassembleFunctions (this=0x4db2e60)
    at /tmp//work/llvm-swf/bolt/lib/Rewrite/RewriteInstance.cpp:2878
#6  0x0000000000c62eac in llvm::bolt::RewriteInstance::run (this=0x4db2e60)
    at /tmp//work/llvm-swf/bolt/lib/Rewrite/RewriteInstance.cpp:748
#7  0x0000000000407cd8 in main (argc=14, argv=0xfffffffff008) at /tmp//work/llvm-swf/bolt/tools/driver/llvm-bolt.cpp:244

[yota9]

The .size directives seems to be skipped for ec_encode_data_mbinit. Could you please try to add it manually in the source code and check if bolt would handle it?

[yavtuk]

also it make sense to add the type for ec_encode_data_mbinit

#ifndef APPLE
.type \name()_mbinit,%function
#endif

[HShan886]

@yota9 add tail call instructions "bl ec_encode_data" in the source code, then program runs ok after bolt optimization.