Skip to content

Dynamic Linker

Jump to bottom Edit New page
WALDEMAR KOZACZUK edited this page Feb 17, 2025 · 37 revisions

Introduction

OSv comprises many components but the dynamic linker is probably one of the most essential ones as it interacts with and ties all other components together and is responsible for bootstrapping an application. In essence, it involves locating an ELF file on the filesystem, loading it into memory using mmap(), processing its headers and segments to relocate symbols, configuring TLS (Thread Local Storage), executing its DT_INIT/DT_INIT_ARRAY functions, loading any dependant ELF objects, and finally starting the app. Please note that unlike Linux, the dynamic linker is an integral part of the OSv kernel. Most of the dynamic linker code is located in core/app.cc, core/elf.cc, arch/x64/arch-elf.cc and arch/aarch64/arch-elf.cc.

Entities

elf::program

Represents the dynamic linker's view of the running program. Typically there is only one instance of it created by elf::create_main_program() called from loader.cc. The program constructor sets up the program base in memory, initializes _core - an instance of elf::memory_image to represent the ELF of OSv kernel, and finally sets up a default set of "supplied" modules like libc.so.6, libpthread.so.0, etc in _modules_rcu. The default main program is stored in the s_program global variable, so effectively the elf::program is a singleton, but it is possible to create multiple program instances for new ELF namespaces.

The key methods:

  • std::shared_ptr<object>get_library(std::string name, ..) - the main method called by osv::application constructor and libc/dlfcn.cc:dlopen() to instantiate an elf::object representing newly loaded ELF. The method delegates the key part of the ELF loading logic to program::load_object(..) (see next below), then builds a static TLS template if present by calling init_static_tls() on the new object and finally invokes the DT_INIT/DT_INIT_ARRAY functions by delegating to program::init_library().
  • std::shared_ptr<elf::object>load_object(std::string name, ..) - locates an ELF file on the filesystem, creates an instance of elf::object to represent it, and finally orchestrates new ELF initialization logic by invoking a number of key object methods on it - load_segments(), process_headers(), load_needed(), relocate() and fix_permissions(). For example, the load_segments() memory-maps all PT_LOAD segments into memory, the load_needed() finds and loads all dependant child objects per DT_NEEDED, and the relocate() processes all relocations. Please note that even though the new object is a member of _modules_rcu, it is NOT visible yet from symbol-lookup perspective at this point.
  • voidinit_library(int argc, char** argv) - invokes DT_INIT/DT_INIT_ARRAY functions on the ELF object and its dependant children in the correct order and eventually makes the new object visible for symbol lookup (please see object::lookup_symbol(const char* name,..)). The init_library() is typically invoked later (delayed) to run on the new application thread to make sure that the init functions have access to the app thread-local variables if needed.
  • symbol_modulelookup(const char* name) - iterates over all objects returned by program::modules_list program::modules_get() and calls object::lookup_symbol(name) for each to finally return a symbol_module tuple for the first found occurrence. The modules_list holds a list of elf::objects maintained in search-priority order and managed as an RCU (Read-Copy-Update)[] structure.

elf::object

Represents an ELF object and implements logic to load an ELF file into memory and process its headers and relocations.

The key methods used during loading an app and its libraries:

  • voidload_segments() - memory-maps all PT_LOAD segments of the ELF file into memory
  • voidprocess_headers() - processes all headers after loading the ELF file (its PT_LOAD segments) into memory - see above
    • the important _ehdr of type Elf64_Ehdr is located at the base (_base) of the ELF in memory
    • iterates over the entries of type Elf64_Phdr (program header) of the array _phdrs and,
    • depending on the type of the header (p_type) - PT_DYNAMIC, PT_INTERP, PT_TLS, PT_GNU_EH_FRAME among others - it identifies addresses of some important data structures describing ELF:
      • _dynamic_table - pointer to the dynamic table (see the dynamic_*() and _dynamic_*() functions
      • TLS segment location and size in file and location in memory
  • voidrelocate() - the top method that processes all relocations - references to those variables or functions that may be defined in the same object (self) or a different one - by calling:
    • relocate_pltgot(),
    • relocate_rela(),
    • relocate_relr()
  • voidrelocate_pltgot() - processes mostly the function relocations found in the PLT (Procedure Linkage Table); iterates over entries in DT_JMPREL and either:
    • calls object::symbol(sym, true) to find the symbol and object::arch_relocate_jump_slot() if bind_now
    • sets the jump slots to resolve lazily later (PLT_GOT) or
    • calls object::arch_relocate_tls_desc() otherwise
  • boolarch_relocate_jump_slot(u32 sym, void *addr, Elf64_Sxword addend, bool ignore_missing) writes symbol.relocated_addr() to the relocation jump slot address (the addr argument)
  • voidarch_relocate_tls_desc(u32 sym, void *addr, Elf64_Sxword addend) - processes so called TLS descriptor relocations:
    • in essence, it setups function calls - __tlsdesc_static or __tlsdesc_dynamic - to access thread-local variables in an arch-specific way
  • voidrelocate_rela() - processes mostly variable relocations: iterates over the table of relocation entries per DT_RELA and calls arch_relocate_rela() for each Elf64_Rela* and passes:
    • its relocation type (p->r_info & 0xffffffff),
    • index in the symbol table of the object being relocated (p->r_info >> 32),
    • address of the relocation (_base + p->r_offset: where to write the relocation value to) and
    • addend (p->r_addend)
  • boolarch_relocate_rela(u32 type, u32 sym, void *addr, Elf64_Sxword addend) - based on the relocation type (type argument) determines the relocation value (symbol relocated address or object module index or st_value for TLS) and writes it to the relocation address (addr argument)
    • the exact implementation and types of the relocation processed are very arch-specific. In many cases, it calls object::symbol(unsigned idx, bool ignore_missing = false) or object::symbol_other(unsigned idx)
    • the x64 relocations include:
      • regular variables: R_X86_64_NONE, R_X86_64_COPY, R_X86_64_64, R_X86_64_RELATIVE, R_X86_64_IRELATIVE, R_X86_64_JUMP_SLOT, R_X86_64_GLOB_DAT, and
      • TLS relocations - R_X86_64_DTPMOD64, R_X86_64_DTPOFF64, R_X86_64_TPOFF64
    • the aarch64 relocations include:
      • regular variables: R_AARCH64_NONE, R_AARCH64_NONE2, R_AARCH64_ABS64, R_AARCH64_COPY, R_AARCH64_GLOB_DAT, R_AARCH64_JUMP_SLOT, R_AARCH64_RELATIVE, R_AARCH64_IRELATIVE, and
      • TLS relocations - R_AARCH64_TLS_TPREL64
  • voidload_needed() - loads dependant ELF objects specified in DT_NEEDED and delegates to program::load_object()

Other methods used:

  • boolarch_init_reloc_dyn(struct init_table *t, u32 type, u32 sym, void *addr, void *base, Elf64_Sxword addend) - used solely to relocate symbols in OSv kernel ELF and is indirectly called by loader premain() function
  • T* object::dynamic_ptr(unsigned tag) - returns locations of various entries in the ELF dynamic table; the _dynamic_table address is identified by process_headers() and the table holds various entries of type Elf64_Dyn with fields:
    • depending on d_tag the d_val specifies the value OR d_ptr specifies the pointer
    • d_tag specifies the type:
      • DT_SYMTAB - the dynamic symbol table
      • DT_STRTAB - the dynamic string tabl
      • DT_HASH - the symbol hash table
      • DT_GNU_HASH
      • DT_RELA - relocation table with Elf64_Rela entries
      • DT_RELR
      • DT_JMPREL
      • DT_PLTGOT - the linkage table
      • DT_INIT - the initialization function
      • DT_INIT_ARRAY
      • DT_FINI - the termination function
      • DT_FINI_ARRAY
      • DT_VERSYM
  • void*resolve_pltgot(unsigned index)
    • finds relocation info under dynamic_ptr<Elf64_Rela>(DT_JMPREL) and symbol index and,
    • finds the symbol by calling object::symbol() and,
    • calls object::arch_relocate_jump_slot() to write the symbol`s relocated address
  • symbol_modulesymbol(unsigned idx, bool ignore_missing) - entry point to the symbol lookup logic:
    • finds its name in the object symbols table (DT_SYMTAB) using the passed-in index (idx) and,
    • searches for a symbol by name in all objects the programs knows about by calling program::lookup(name); if symbol not found it aborts if ignore_missing is false otherwise just warns,
    • returns the symbol_module which is a tuple of the object the symbol resides in and the symbol definition (Elf64_Sym *);
    • called by following methods during the relocation phase:
      • arch_relocate_rela()
      • arch_relocate_jump_slot()
      • resolve_pltgot()
  • Elf64_Sym*lookup_symbol(const char* name) - looks up symbol by name by delegating to either lookup_symbol_old or lookup_symbol_gnu; bails out if object not visible (during construction)
    • Elf64_Sym* object::lookup_symbol_old(const char* name) - ???
    • Elf64_Sym* object::lookup_symbol_gnu(const char* name) - uses GNU hashmap

elf::file

  • void load_elf_header()
  • void load_program_headers()

elf::memory_image

elf::symbol_module

osv::app_registry

osv::application

Represents a running program and its _program member points to the program the application was created for.

Flow

The program::get_library() is the critical point where the dynamic linker gets involved in instantiating new applications.

The main program (kernel?) gets instantiated by elf::create_main_program() called from loader.cc

application::new_program() instantiates new program for new ELF namespace with new base address.

TLS (Thread Local Storage)

Thread local storage (TLS) is a mechanism that allows applications and shared libraries to use variables stored in a memory area specific to a given thread. These include variables marked with __thread and C++ thread_local modifiers. For TLS variables to work correctly, the OSv dynamic linker needs to recognize TLS segments in an ELF file, construct static TLS blocks in memory, process relevant relocations (aka references to those variables), and provide certain functions like __tls_get_addr among other things. There are many aspects of making TLS work - where the variable is defined, who accesses it, and when. For example, a TLS variable may be:

  • defined in OSv kernel, an app executable or shared library,
  • referenced by the same ELF object code (self) or another one,
  • referenced by a relocation or dynamically via dlsym()
  • located in an ELF loaded because of DT_NEEDED or by dlopen()

Before we delve into what OSv dynamic linker does to support TLS, it is important to understand two different formats of static TLS block layout - so-called Variant I and Variant II - and 4 different models of accessing TLS variables: local-exec, initial-exec, general-dynamic and local-dynamic.

Static TLS block is an area of memory allocated for each thread independently, intended to store thread-local variables and built from a template derived when loading the main application and its dependant libraries. The template, in essence, specifies the total size of the TLS block, the offsets in it for each ELF object including the OSv kernel, and any initial values for the TLS variables in those objects. The static TLS does not change once the thread is created and running; that is why it is called "static" after all. Bear in mind that the kernel threads' static TLS block will only be large enough just for kernel TLS variables, but the application threads' TLS block will be larger to accommodate both the kernel and the application and its libraries variables. The static also means that the offsets to the variables in the static block from so-called thread register (FS on x64 and TPIDR_EL0/1 on aarch64) are constant.

In Variant I (used in AArch64 port) the data is laid out from left to right (local-exec for PIEs, kernel followed with other objects). In Variant II (used in X86_64 port) the data is laid out from right to left which is exactly opposite to Variant I.

  • Variant I
    // (1) - TLS memory area layout with app as shared library
    // |------|--------------|-----|-----|-----|
    // |<NONE>|KERNEL        |SO_1 |SO_2 |SO_3 |
    // |------|--------------|-----|-----|-----|

    // (2) - TLS memory area layout with PIE or position dependant executable
    // |------|--------------|-----|-----|
    // | EXE  |KERNEL        |SO_2 |SO_3 |
    // |------|--------------|-----|-----|
  • Variant II
    // (1) - TLS memory area layout with app shared library
    // |-----|-----|-----|--------------|------|
    // |SO_3 |SO_2 |SO_1 |KERNEL        |<NONE>|
    // |-----|-----|-----|--------------|------|

    // (2) - TLS memory area layout with PIE or position dependant executable
    //       |-----|-----|---------------------|
    //       |SO_3 |SO_2 |KERNEL        | EXE  |
    //       |-----|-----|--------------|------|

The dynamic TLS means that the memory where TLS variables live (typically in dlopen-ed library) is allocated dynamically after the application starts and is NOT part of the static TLS block. It also means that offsets to the thread register will NOT be constant. Therefore the _tls_get_addr() or dynamic TLS descriptors are used to accommodate access to those variables. The dynamic TLS access is also therefore slower than the static one.

Flow

The role of the dynamic linker with respect to TLS handling is to connect the "dots" which at a high-level can be divided into 4 phases:

  • processing TLS program header to detect the size and other specifics of TLS data,
  • processing TLS-related relocations,
  • building memory blueprint for TLS - so-called template,
  • and finally allocating and initializing TLS blocks for each thread before it is started.

In the 1st phase, as OSv dynamic linker loads an ELF object in core/elf.cc:std::shared_ptr<elf::object> program::load_object(..), it first mmaps all PT_LOAD segments and then processes all headers (see core/elf.cc:void object::process_headers()) to detect any TLS segment and capture its size, alignment, and its address in memory.

In the 2nd phase, the dynamic linker processes all relocations including those in both GOT (Global Offset Table, see core/elf.cc:void object::relocate_rela()) and PLT (Procedure Linkage Table, see core/elf.cc:void object::relocate_pltgot()). Some of those relocations are TLS specific and are processed in a specific way:

  • R_X86_64_DTPMOD64: to identify the module ID (unique id assigned to each ELF object) where the given TLS variable referenced by this relocation is located; please note that the R_X86_64_DTPMOD64 relocation may be in one ELF object and its definition and location may be in another one or the same; in addition, if index == STN_UNDEF, the relocation applies to hidden (static) TLS variables in given module and is used to determine its module index passed later to __tls_get_addr() - offset in TLS block is known in advance,
  • R_X86_64_DTPOFF64: to calculate the offset in the TLS block of the object where the variable is going to "live"; the values determined for both R_X86_64_DTPMOD64 and R_X86_64_DTPOFF64 are stored in the relocation placeholders in the GOT and then referenced as input to the __tls_get_addr() function provided by OSv dynamic linker; please note that even though both relocations are intended for the dynamic model based on __tls_get_addr(), the variables can live in either static TLS or lazily allocated block if the object was dlopen()-ed,
  • R_X86_64_TPOFF64/R_AARCH64_TLS_TPREL64: to calculate offset of the TLS variable in static TLS - this means that variable will "live" in the statically allocated memory area; in addition, each of these relocations triggers a call to void object::alloc_static_tls() to determine the offset of the TLS block of the object owning the variable and eventually fed in order to properly build application TLS template,
  • R_AARCH64_TLSDESC: to place address of static TLS "resolver" function and its only argument - offset of the variable in static TLS.

In the 3rd phase, after std::shared_ptr<elf::object> program::load_object(..) completes loading ELF in memory, OSv dynamic linker calls void object::init_static_tls() (see std::shared_ptr<object> program::get_library(..)) to build the TLS template for the main app object based on its own TLS segment and any of its dependencies. More specifically it calculates total TLS template size (stored in object::_initial_tls_size member variable) and iterates over all its dependant objects to copy their TLS data to object::_initial_tls buffer by delegating to the void object::prepare_initial_tls(..) or void object::prepare_local_tls(..) methods that are arch (AArch64 or X86_64) specific. At the same time, it also stores the offsets of TLS data for each object in object::_initial_tls_offsets vector. So at the end the init_static_tls() sets 3 key variables - object::_initial_tls, object::_initial_tls_size and object::_initial_tls_offsets.

Finally, for each thread created, its constructor (see core/sched.cc::thread::thread()) allocates so called TCB (Thread Control Block) which includes static TLS block by delegating to arch-specific setup_tcb() (see arch/*/arch-switch.hh for details). The static TLS block (thread::_tcb->tls_base) is in essence an instance of the TLS template built in previous phases. Finally, the index (_tls) vector is populated with offsets for each module based on object::_initial_tls_offsets.

TLS descriptors

__tls_get_addr()

ELF Namespaces

References

Add a custom footer
Add a custom sidebar
Clone this wiki locally