For binary built for esp32 target xtensa-esp32-elf-objdump produces incorrect disassembly, while xtensa-esp32s3-elf-objdump is correct (GCC-247)

[michael-sayapin]

Environment

• Module or chip used: ESP32-WROOM-32
• IDF version (run git describe --tags to find it): v4.4.1
• Build System: idf.py (ninja)
• Compiler version (run xtensa-esp32-elf-gcc --version to find it): xtensa-esp32-elf-gcc (crosstool-NG esp-2021r1) 8.4.0
• Operating System: Linux
• Power Supply: external 5V

Problem Description

I'm building the .elf file with idf.py and following sdkconfig:

CONFIG_IDF_CMAKE=y
CONFIG_IDF_TARGET_ARCH_XTENSA=y
CONFIG_IDF_TARGET="esp32"
CONFIG_IDF_TARGET_ESP32=y
CONFIG_IDF_FIRMWARE_CHIP_ID=0x0000
CONFIG_ESP32_REV_MIN_0=y
# CONFIG_ESP32_REV_MIN_1 is not set
# CONFIG_ESP32_REV_MIN_2 is not set
# CONFIG_ESP32_REV_MIN_3 is not set
CONFIG_ESP32_REV_MIN=0

Here is the real code of one of the tasks:

#define WITH_MUTEX(what, mutex, timeout_tick)                             \
  ({                                                                      \
    __label__ error_macros_with_mutex_exit;                               \
    errors_Error __error = {0};                                           \
    if (pdTRUE != xSemaphoreTake(mutex, (timeout_tick))) {                \
      __error = ERROR(errors_Error_system_tag,                            \
                      errors_System_SYSTEM_TAKE_MUTEX_FAILED, "%s:%u",    \
                      __func__, __LINE__);                                \
      goto error_macros_with_mutex_exit;                                  \
    }                                                                     \
    MDF_LOGD("Mutex %s taken in %s, before %s", #mutex, __func__, #what); \
    __error = (what);                                                     \
    if (pdTRUE != xSemaphoreGive(mutex)) {                                \
      /* highly unlikely, since mutex is successfully taken */            \
      protocol_utils_destroy_error(__error);                              \
      __error = ERROR(errors_Error_system_tag,                            \
                      errors_System_SYSTEM_TAKE_MUTEX_FAILED,             \
                      "xSemaphoreGive %s:%u", __func__, __LINE__);        \
      goto error_macros_with_mutex_exit;                                  \
    }                                                                     \
    MDF_LOGD("Mutex %s given in %s, after %s", #mutex, __func__, #what);  \
  error_macros_with_mutex_exit:                                           \
    __error;                                                              \
  })

...

static void xxx_task(void *pvParameters) {
  char *task_name = pcTaskGetTaskName(NULL);
  MDF_LOGD("%s is running (priority %d)", task_name, uxTaskPriorityGet(NULL));
  errors_Error error = {0};
  while (true) {
    error = WITH_MUTEX(xxx_do_something(), g_mutex, portMAX_DELAY);
    if (0 != error.which_code) {
      protocol_utils_destroy_error(error);
      vTaskDelay(pdMS_TO_TICKS(XXX_TASK_SLEEP_ERROR_MS));
      continue;
    }
    MDF_LOGD("%s free stack memory: %d", task_name,
             uxTaskGetStackHighWaterMark(NULL));
    vTaskDelay(pdMS_TO_TICKS(XXX_TASK_SLEEP_MS));
  }
  MDF_LOGW("%s exited", task_name);
  vTaskDelete(NULL);
}

Expected Behavior

xtensa-esp32s3-elf-objdump -d produces correct disassembly:

400eb64c <xxx_task>:
400eb64c:	00c136        	entry	a1, 96
400eb64f:	00a0a2        	movi	a10, 0
400eb652:	a9ffa5        	call8	4009564c <pcTaskGetName>
400eb655:	ffafb2        	movi	a11, -1
400eb658:	9a8b81        	l32r	a8, 400d2084 <_stext+0x2064>
400eb65b:	08a8      	l32i.n	a10, a8, 0
400eb65d:	a86225        	call8	40093c80 <xQueueSemaphoreTake>
400eb660:	201a26        	beqi	a10, 1, 400eb684 <xxx_task+0x38>
400eb663:	9a8ba1        	l32r	a10, 400d2090 <_stext+0x2070>
400eb666:	55a2b2        	movi	a11, 0x255
400eb669:	11b9      	s32i.n	a11, a1, 4
400eb66b:	01a9      	s32i.n	a10, a1, 0
400eb66d:	9a86f1        	l32r	a15, 400d2088 <_stext+0x2068>
400eb670:	9a87e1        	l32r	a14, 400d208c <_stext+0x206c>
400eb673:	cd0c      	movi.n	a13, 12
400eb675:	dc0c      	movi.n	a12, 13
400eb677:	4ab3e5        	call8	401361b4 <__error_create>
400eb67a:	a1a9      	s32i.n	a10, a1, 40
400eb67c:	b1b9      	s32i.n	a11, a1, 44
400eb67e:	c1c9      	s32i.n	a12, a1, 48
400eb680:	001006        	j	400eb6c4 <xxx_task+0x78>
400eb683:	00          	.byte 00
400eb684:	ffc8a5        	call8	400eb310 <xxx_do_something>
400eb687:	a1a9      	s32i.n	a10, a1, 40
400eb689:	b1b9      	s32i.n	a11, a1, 44
400eb68b:	c1c9      	s32i.n	a12, a1, 48
400eb68d:	0d0c      	movi.n	a13, 0
400eb68f:	0dcd      	mov.n	a12, a13
400eb691:	0dbd      	mov.n	a11, a13
400eb693:	9a7c81        	l32r	a8, 400d2084 <_stext+0x2064>
400eb696:	08a8      	l32i.n	a10, a8, 0
400eb698:	a80425        	call8	400936dc <xQueueGenericSend>
400eb69b:	251a26        	beqi	a10, 1, 400eb6c4 <xxx_task+0x78>
400eb69e:	a1a8      	l32i.n	a10, a1, 40
400eb6a0:	b1b8      	l32i.n	a11, a1, 44
400eb6a2:	c1c8      	l32i.n	a12, a1, 48
400eb6a4:	4cdf65        	call8	4013849c <protocol_utils_destroy_error>
400eb6a7:	9a7aa1        	l32r	a10, 400d2090 <_stext+0x2070>
400eb6aa:	55a2b2        	movi	a11, 0x255
400eb6ad:	11b9      	s32i.n	a11, a1, 4
400eb6af:	01a9      	s32i.n	a10, a1, 0
400eb6b1:	9a78f1        	l32r	a15, 400d2094 <_stext+0x2074>
400eb6b4:	9a76e1        	l32r	a14, 400d208c <_stext+0x206c>
400eb6b7:	cd0c      	movi.n	a13, 12
400eb6b9:	dc0c      	movi.n	a12, 13
400eb6bb:	4aafa5        	call8	401361b4 <__error_create>
400eb6be:	a1a9      	s32i.n	a10, a1, 40
400eb6c0:	b1b9      	s32i.n	a11, a1, 44
400eb6c2:	c1c9      	s32i.n	a12, a1, 48
400eb6c4:	0a2182        	l32i	a8, a1, 40
400eb6c7:	0b21a2        	l32i	a10, a1, 44
400eb6ca:	c198      	l32i.n	a9, a1, 48
400eb6cc:	7189      	s32i.n	a8, a1, 28
400eb6ce:	81a9      	s32i.n	a10, a1, 32
400eb6d0:	9199      	s32i.n	a9, a1, 36
400eb6d2:	f48080        	extui	a8, a8, 0, 16
400eb6d5:	089c      	beqz.n	a8, 400eb6e9 <xxx_task+0x9d>
400eb6d7:	71a8      	l32i.n	a10, a1, 28
400eb6d9:	81b8      	l32i.n	a11, a1, 32
400eb6db:	09cd      	mov.n	a12, a9
400eb6dd:	4cdbe5        	call8	4013849c <protocol_utils_destroy_error>
400eb6e0:	d0a7a2        	movi	a10, 0x7d0
400eb6e3:	a8f2a5        	call8	4009460c <vTaskDelay>
400eb6e6:	ffdac6        	j	400eb655 <xxx_task+0x9>
400eb6e9:	95cfa1        	l32r	a10, 400d0e28 <_stext+0xe08>
400eb6ec:	a8f1e5        	call8	4009460c <vTaskDelay>
400eb6ef:	ffd886        	j	400eb655 <xxx_task+0x9>

Actual Behavior

xtensa-esp32-elf-objdump -d produces this disassembly:

400eb64c <xxx_task>:
400eb64c:       00c136          entry   a1, 96
400eb64f:       00a0a2          movi    a10, 0
400eb652:       a9ffa5          call8   4009564c <pcTaskGetName>
400eb655:       ffafb2          movi    a11, -1
400eb658:       9a8b81          l32r    a8, 400d2084 <_stext+0x2064>
400eb65b:       08a8            l32i.n  a10, a8, 0
400eb65d:       a86225          call8   40093c80 <xQueueSemaphoreTake>
400eb660:       201a26          beqi    a10, 1, 400eb684 <xxx_task+0x38>
400eb663:       9a8ba1          l32r    a10, 400d2090 <_stext+0x2070>
400eb666:       55a2b2          movi    a11, 0x255
400eb669:       11b9            s32i.n  a11, a1, 4
400eb66b:       01a9            s32i.n  a10, a1, 0
400eb66d:       9a86f1          l32r    a15, 400d2088 <_stext+0x2068>
400eb670:       9a87e1          l32r    a14, 400d208c <_stext+0x206c>
400eb673:       cd0c            movi.n  a13, 12
400eb675:       dc0c            movi.n  a12, 13
400eb677:       4ab3e5          call8   401361b4 <__error_create>
400eb67a:       a1a9            s32i.n  a10, a1, 40
400eb67c:       b1b9            s32i.n  a11, a1, 44
400eb67e:       c1c9            s32i.n  a12, a1, 48
400eb680:       001006          j       400eb6c4 <xxx_task+0x78>
400eb683:       c8a500          lsi     f0, a5, 0x320
400eb686:       ff              .byte 0xff
400eb687:       a1a9            s32i.n  a10, a1, 40
400eb689:       b1b9            s32i.n  a11, a1, 44
400eb68b:       c1c9            s32i.n  a12, a1, 48
400eb68d:       0d0c            movi.n  a13, 0
400eb68f:       0dcd            mov.n   a12, a13
400eb691:       0dbd            mov.n   a11, a13
400eb693:       9a7c81          l32r    a8, 400d2084 <_stext+0x2064>
400eb696:       08a8            l32i.n  a10, a8, 0
400eb698:       a80425          call8   400936dc <xQueueGenericSend>
400eb69b:       251a26          beqi    a10, 1, 400eb6c4 <xxx_task+0x78>
400eb69e:       a1a8            l32i.n  a10, a1, 40
400eb6a0:       b1b8            l32i.n  a11, a1, 44
400eb6a2:       c1c8            l32i.n  a12, a1, 48
400eb6a4:       4cdf65          call8   4013849c <protocol_utils_destroy_error>
400eb6a7:       9a7aa1          l32r    a10, 400d2090 <_stext+0x2070>
400eb6aa:       55a2b2          movi    a11, 0x255
400eb6ad:       11b9            s32i.n  a11, a1, 4
400eb6af:       01a9            s32i.n  a10, a1, 0
400eb6b1:       9a78f1          l32r    a15, 400d2094 <_stext+0x2074>
400eb6b4:       9a76e1          l32r    a14, 400d208c <_stext+0x206c>
400eb6b7:       cd0c            movi.n  a13, 12
400eb6b9:       dc0c            movi.n  a12, 13
400eb6bb:       4aafa5          call8   401361b4 <__error_create>
400eb6be:       a1a9            s32i.n  a10, a1, 40
400eb6c0:       b1b9            s32i.n  a11, a1, 44
400eb6c2:       c1c9            s32i.n  a12, a1, 48
400eb6c4:       0a2182          l32i    a8, a1, 40
400eb6c7:       0b21a2          l32i    a10, a1, 44
400eb6ca:       c198            l32i.n  a9, a1, 48
400eb6cc:       7189            s32i.n  a8, a1, 28
400eb6ce:       81a9            s32i.n  a10, a1, 32
400eb6d0:       9199            s32i.n  a9, a1, 36
400eb6d2:       f48080          extui   a8, a8, 0, 16
400eb6d5:       089c            beqz.n  a8, 400eb6e9 <xxx_task+0x9d>
400eb6d7:       71a8            l32i.n  a10, a1, 28
400eb6d9:       81b8            l32i.n  a11, a1, 32
400eb6db:       09cd            mov.n   a12, a9
400eb6dd:       4cdbe5          call8   4013849c <protocol_utils_destroy_error>
400eb6e0:       d0a7a2          movi    a10, 0x7d0
400eb6e3:       a8f2a5          call8   4009460c <vTaskDelay>
400eb6e6:       ffdac6          j       400eb655 <xxx_task+0x9>
400eb6e9:       95cfa1          l32r    a10, 400d0e28 <_stext+0xe08>
400eb6ec:       a8f1e5          call8   4009460c <vTaskDelay>
400eb6ef:       ffd886          j       400eb655 <xxx_task+0x9>

Notice that the call to xxx_do_something is missing.

[igrr]

@michael-sayapin could you please attach the ELF file which has this issue?

[michael-sayapin]

@michael-sayapin could you please attach the ELF file which has this issue?

Let me check with the team, as it is a commercial product and I'm not sure I can open the ELF file. I will try to reproduce with a minimal example.

[igrr]

Another option for you to consider is sending it to me (ivan at espressif).

[igrr]

Thank you for sharing the ELF file @michael-sayapin.

Upon a brief look, it seems that there are two issues here:

1. ESP32 and ESP32-S3 builds of objdump interpret the padding "0x00" byte after the unconditional jump differently. ESP32 version of objdump interprets it along with two other bytes as an "lsi" instruction. The ESP32-S3 version of objdump interprets it as a padding byte and correctly decodes the next 3 bytes as "call8". In theory, ESP32-S3 version should behave the same way ESP32 version does (i.e. incorrectly).

   We will investigate why this happens.

2. Currently, ESP-IDF linker scripts don't add special ELF sections which contain information about such padding bytes, and which can used by the disassembler to avoid issues like this. This fact has been noticed recently by @o-marshmallow and there is a patch available internally for this issue. We haven't merged the patch yet, still investigating one issue which has surfaced when we ran our test suite.

   I'm attaching the patch here for you to try. Please try applying this patch to ESP-IDF, link the application again and see if the ESP32 disassembler starts interpreting your ELF file correctly. 17268.patch.txt

[michael-sayapin]

Finally had a moment to try the patch. Turns out, the builder we use is based on IDF 4.2, so I had to modify the patch quite a bit to apply. In the end it applied fully.

However, the disassembly result is the same, and seems like it made it a bit worse.

PS: Both non-patched and patched versions contain correct calls if disassembled with -r flag (relocation entries).