Psy-Q SDK research

[emoose]

E: repo now links in converted Psy-Q obj files for 99% of the SDK used in SH, there's still a single .obj file from SDK that couldn't be used though.

TODO:

• snmain.obj: seems this was built from a SNMAIN.S assembly file, there's older code available for that which matches very closely to what SH uses (LIBSN.ZIP). Sadly we can't use snmain.obj from the SDK since it uses strange relocation types which ELF doesn't support - wonder if it'd be possible to update the asm and built that into an ELF .o instead, or if we could just copy snmain.s.o from our build into the lib folder (as long as we can be sure it doesn't include any of our decomp symbols, which could change at any time)
• .a library archives: atm we have to use a bunch of loose .o files for each SDK object, which also needs the ld script splat creates to be edited to use them - would be nicer if we could pack them into .a library file instead, but that then causes linker to start checking function references to decide which .o files from the .a should be included (and since SH uses multiple overlays, those references won't be found & a bunch of .o will be skipped), found a workaround for it but was pretty hacky, maybe there's a better way we could handle them.

---

There was some talk about Psy-Q on discord a while ago but it's a bit scattered, figured it's worth an issue here.

The Psy-Q they use seems a weird mix of 4.4 and 4.3 patch libs:

>printver SLUS_007.07
SLUS_007.07:    libc (1):       4.4.0.0
SLUS_007.07:    libapi (0):     4.4.0.0
SLUS_007.07:    libspu (17):    4.4.0.0
SLUS_007.07:    libetc (6):     4.4.0.0
SLUS_007.07:    libcd (4):      4.3.1.0
SLUS_007.07:    libgpu (7):     4.3.0.0

>printver BODYPROG.bin
BODYPROG.bin:   libcd (4):      0.4.8.0 (likely misidentified)
BODYPROG.bin:   libgte (9):     4.3.0.0
BODYPROG.bin:   libcard (3):    4.3.0.0
BODYPROG.bin:   libgs (8):      4.3.0.0
BODYPROG.bin:   libpad (12):    4.3.0.0

>printver STREAM.bin
STREAM.bin:     libpress (14):  4.3.0.0

libcd 4.3.1.0 was included with DTL-S2002 v2.2, inside PSX\LIB\PATCHES, but not sure why that was used instead of the 4.4 version.

Possible they might have been using some kind of incomplete-4.4-patch version that was only released online, eg:

• Base: 4.3 SDK
• Patch: 4.3.1 libcd
• Patch: 4.4 libc/libapi/libspu/libetc

Hopefully the 4.4 SDK that's available will match up with those 4.4 libs, but guess we'll see.

Also possible they might have used ECOFF version of the libs too, some SDK discs included those as well as the normal PsyQ obj format, ECOFF is usually used on SGI machines which TS had been pictured using to dev the game.
(sadly the 4.3.1 patch version hasn't been found as ECOFF yet)

[emoose]

After cleaning up some of the label names from our sym files objdiff shows pretty close match with the bios.s.o from our build vs PsyQ bios.obj converted with psyq-obj-parser (compared to previous result):

Hopefully can try checking the other objs soon.

[emoose]

Managed to get it linking & matching with psyq libcd 4.3.1 obj files, took some massaging though.

• Used PSYLIB to extract libcd.lib file into seperate obj files
• After converting to ELF with psyq-obj-parser, bundled them into a libcd.a with ar rcs libcd.a *.o
• Added lib lines to main.yaml splat config to try pulling in .rodata/.text/.data (like sssv splat conf does)

      - [0x930, lib, libcd, event, .rdata]
      - [0x950, lib, libcd, sys, .rdata]
      - [0x960, lib, libcd, bios, .rdata] # Has jtbl, bss pointers.
      - [0xBC0, lib, libcd, cdread, .rdata]
...
      - [0x466C, lib, libcd, event] # CdInit
      - [0x47BC, lib, libcd, sys] # CD related functions.
      - [0x4F4C, lib, libcd, bios] # More CD functions, has bss StMode 
      - [0x66AC, lib, libcd, cdread] # Still CD.
...
      - [0x12E20, lib, libcd, event, .data]
      - [0x12E30, lib, libcd, sys, .data]
      - [0x12EB0, lib, libcd, bios, .data]
      - [0x131D0, lib, libcd, event, .data]

• built main, didn't match
• looked in EXE, none of the libcd stuff was actually pulled in
• in main.ld linker script, there were lines added like lib/libcd.a:event.o(.rdata)
• changed those to pull the raw .o files instead of using .a, lib/libcd/event.o(.rdata), same for .text/.data
• added .bss sections there too, under build/asm/main/psyq/libetc/intr_dma.s.o(.bss);:

        lib/libcd/event.o(.bss);
        lib/libcd/sys.o(.bss);
        lib/libcd/bios.o(.bss);
        lib/libcd/cdread.o(.bss);

• re-ran linker and objcopy commands, now it built matching EXE fine

So the issue is with using .a archives, but afaik splat lib only works with those...

Apparently with .a archives the linker might be trying to check for references to the .o symbols before it pulls them in, and fails to see them for some reason, maybe because the code that uses them comes after this? looks like in SSSV the code came before any libs .text/.rodata/etc.

(didn't add .bss to splat config since we have all of psyq bss under a single line there, wasn't sure how to split that, but even with the bss lines above changed to libcd.a it still got ignored ;_;)

[emoose]

Could be wrong about the .a symbol referencing stuff, looks like main .text does come before the libcd .text code, so not sure why ld refuses to pull it in...

Unless it's something to do with .rodata for libcd coming before it all, being unreferenced prior to where its added & skipped, then when libcd .text is tried to link in, because it referenced .rodata that been skipped already it's now deciding to skip the .text too? 😕

Really it'd be easier to just link the loose .o files for it, hadn't seen any splat option for that though, unless there's something we can do with makefile for it (or could just use lib and post-process the .ld files after to use .o instead of .a, kinda hacky though)

[emoose]

Was able to get main built & matching with all psyq libs 🥳

Only one that didn't work was libsn snmain entrypoint, psyq-obj-parser didn't want to convert to ELF because of some relocation issue, guess we could leave that since it was probably asm anyway.

There's still the issue with .a archives, used this line to generate + change .a to .o + build:

make regenerate && sed -i 's/\.a:/\//g' linkers/main.ld && make check

Here's main.yaml setup to use lib files, just 8 bytes of .bss which aren't accounted for with it:

main.yaml

name: Main Executable (SLUS_007.07)
sha1: f38344565fb731befc52fd968eb5a667be37acf3
options:
  basename: main
  target_path: rom/SLUS_007.07
  base_path: ..
  platform: psx
  compiler: GCC

  asm_path: asm/main
  src_path: src/main
  build_path: build/
  # create_asm_dependencies: True

  ld_script_path: linkers/main.ld

  find_file_boundaries: False
  gp_value: 0x80022BB0
  global_vram_start: 0x80010000
  global_vram_end: 0x80200000

  o_as_suffix: False
  use_legacy_include_asm: False

  asm_function_macro: glabel
  asm_jtbl_label_macro: jlabel
  asm_data_macro: dlabel

  section_order: [".rodata", ".text", ".data", ".sdata", ".sbss", ".bss"]
  symbol_addrs_path:
    - configs/sym.bodyprog.txt
    - configs/sym.main.txt
  reloc_addrs_path:
    - configs/rel.main.txt

  undefined_funcs_auto_path: linkers/undefined_funcs_auto.main.txt
  undefined_syms_auto_path: linkers/undefined_syms_auto.main.txt

  extensions_path: tools/splat_ext

  subalign: 4

  string_encoding: ASCII
  data_string_encoding: ASCII
  rodata_string_guesser_level: 2
  data_string_guesser_level: 2

  ld_bss_is_noload: False
segments:
  - name: header
    type: header
    start: 0x0

  - name: main
    type: code
    start: 0x800
    vram: 0x80010000
    align: 4
    subalign: 4
    bss_size: 0x1EC8
    subsegments:
      - [0x800, .rodata, main]
      - [0x808, .rodata, fsqueue_3]
      - [0x814, pad, fsqueue_3] # Extra padding for some reason.
      - [0x818, .rodata, fsqueue_2] # jtbl

      - [0x830, lib, libspu, spu, .rdata]
      - [0x880, lib, libetc, vsync, .rdata]
      - [0x890, lib, libetc, intr, .rdata]
      - [0x900, lib, libetc, intr_dma, .rdata]
      - [0x930, lib, libcd, event, .rdata]
      - [0x950, lib, libcd, sys, .rdata]
      - [0x960, lib, libcd, bios, .rdata] # Has jtbl, bss pointers.
      - [0xBC0, lib, libcd, cdread, .rdata]
      - [0xC10, lib, libgpu, tmd, .rdata]
      - [0xD50, lib, libgpu, sys, .rdata]

      - [0xF70, c, main] # Main function.
      - [0x12D0, c, fileinfo] # File entry functions.
      - [0x1658, c, fsqueue_1] # fs queue functions part 1.
      - [0x1CC4, c, fsqueue_2] # fs queue functions part 2, split due to usage of jtbl in rodata.
      - [0x1E30, c, fsqueue_3] # fs queue functions part 3.
      - [0x2470, c, fsmem] # fs heap allocator.
      - [0x2778, c, memcpy] # Custom memcpy implementation.
      - [0x28B8, c, rng] # RNG functions.
      - [0x293C, c, memmove32] # memmove32

      - [0x29BC, lib, libapi, a36] #EnterCriticalSection
      - [0x29CC, lib, libapi, a37] #ExitCriticalSection
      - [0x29DC, lib, libapi, a50] #open
      - [0x29EC, lib, libapi, a52] #read
      - [0x29FC, lib, libapi, a54] #close
      - [0x2A0C, lib, libapi, a08] #OpenEvent
      - [0x2A1C, lib, libapi, a12] #EnableEvent
      - [0x2A2C, lib, libc, c21] #strcat
      - [0x2A3C, lib, libc, c23] #strcmp
      - [0x2A4C, lib, libc, c40] #bzero
      - [0x2A5C, lib, libc, c43] #memset
      - [0x2A6C, lib, libspu, s_i] # SpuInit
      - [0x2A8C, lib, libspu, s_ini]
      - [0x2BEC, lib, libspu, spu] # Has bss section: https://github.com/lab313ru/psx_psyq_signatures/blob/e9e46e7e133ef275a79bfce650924f98edb086bc/470/LIBSPU.LIB.json#L279
      - [0x372C, lib, libapi, a07] #DeliverEvent
      - [0x373C, lib, libc, c63] #printf
      - [0x374C, lib, libspu, s_dcb]
      - [0x377C, asm, psyq/libsn/snmain] # entry point (psyq-obj-parser doesn't support it...)
      - [0x38FC, lib, libapi, c57] #InitHeap
      - [0x390C, lib, libetc, vsync] #VSync
      - [0x3B1C, lib, libapi, a91] #ChangeClearPAD
      - [0x3B2C, lib, libapi, l10] #ChangeClearRCnt
      - [0x3B3C, lib, libc, a63] # puts from psyq 4.4?
      - [0x3B5C, lib, libetc, intr] # Callback functions
      - [0x421C, lib, libapi, c114] # _96_remove
      - [0x423C, lib, libapi, a23] # ReturnFromException
      - [0x424C, lib, libapi, a24] # ResetEntryInt
      - [0x425C, lib, libapi, a25] # HookEntryInt
      - [0x426C, lib, libc, c19] # setjmp
      - [0x427C, lib, libetc, intr_vb] # startIntrVSync
      - [0x439C, lib, libetc, intr_dma] # startIntrDMA
      - [0x463C, lib, libetc, vmode] # Set/GetVideoMode
      - [0x466C, lib, libcd, event] # CdInit
      - [0x47BC, lib, libcd, sys] # CD related functions.
      - [0x4F4C, lib, libcd, bios] # More CD functions, has bss StMode 
      - [0x66AC, lib, libcd, cdread] # Still CD.
      - [0x707C, lib, libgpu, tmd]
      - [0x88FC, lib, libgpu, sys]
      - [0xB90C, lib, libapi, c73] # GPU_cw

      - [0xB91C, .data, fileinfo]
      - [0x11ADC, .data, main]
      - [0x11B4C, lib, libspu, s_ini, .data]
      - [0x11BCC, lib, libspu, spu, .data]
      - [0x11C2C, lib, libspu, s_rmp0, .data]
      - [0x11C2C, lib, libspu, s_m, .data]
      - [0x11C6C, data, psyq/libsn/snmain]
      - [0x11C90, lib, libetc, vsync, .data]
      - [0x11CB0, lib, libetc, intr, .data]
      - [0x11CB0, lib, libetc, intr_vb, .data]
      - [0x12DE0, lib, libetc, intr_dma, .data]
      - [0x12E10, lib, libetc, vmode, .data]
      - [0x12E20, lib, libcd, event, .data]
      - [0x12E30, lib, libcd, sys, .data]
      - [0x12EB0, lib, libcd, bios, .data]
      - [0x131D0, lib, libcd, cdread, .data]
      - [0x131D0, lib, libgpu, sys, .data]

      - [0x133B0, .sdata, fileinfo]
      - [0x13468, pad, fileinfo]
      - [0x1346C, sdata, start]
      - [0x13474, .sdata, main]
      - [0x13488, pad, main]
      - [0x13490, .sdata, rng]

      - [0x13494, sbss, psyq/libsn/snmain]

      - [0x13498, .bss, fsqueue_1]
      - [0x138C0, .bss, fsmem]
      - [0x139A0, lib, libspu, spu, .bss]
      - [0x139B4, bss, psyq_unk]
      - [0x139B8, lib, libcd, event, .bss]
      - [0x139B8, lib, libcd, sys, .bss]
      - [0x139B8, lib, libcd, bios, .bss]
      - [0x139B8, lib, libcd, cdread, .bss]
      - [0x139EC, bss, psyq_unk2]
      - [0x139F0, lib, libgpu, tmd, .bss]
      - [0x13A10, lib, libgpu, sys, .bss] # Adds a lot of extra data past end of file, recommend editing .bss size in sys.o to 0x5F0.
  - [0x14000]

Pack of converted libs, just extract to root of decomp: sh1-lib-elfs.zip

[emoose]

Ah finally figured out issue with .a files, just had to add path to each one to end of the linker command 🥇

emoose@677541a should let main link from psyq .a files.

With that they mostly work, but were a couple issues with some SDK funcs that get included in main but are only ever referenced by other overlays, had to add a lib_externs.ld file that forces linker to reference those.

There was also issue with splat not marking some SDK funcs as defined for some reason, so they ended up in undefined_funcs_auto.main.txt, which then caused linker not to bother linking from SDK libs.

Not really sure why those aren't defined, only a few SDK funcs are affected, maybe those ones make bios syscalls or something which makes splat fail to recognize them properly?

Added a workaround by forcing splat to mark as defined only inside main, needs an extra sym.txt file for that though, since the other overlays need them to stay undefined.

Now that .a can be linked maybe can get bodyprog to use PsyQ too, got it pretty close before but there was issue with some global vars not being ordered the same in the SDK libs, apparently linker might reorder variables when .a is linked, but haven't tried that yet.

[emoose]

Looked into linking libs into bodyprog, doesn't seem linker is handling reordering vars sadly, could be because we're specifying each .o in the linker script instead of just letting linker pull in each .o that's needed (which we wouldn't be able to do, since psylink used a different ordering to normal ld afaik)

Right now I got all the psyq stuff linked into bodyprog except for libcd & libgs, both of those still have different .bss variable ordering...

Tried seeing if I could edit the order in the .o, luckily these vars are all relocatable so only relocations + symbol table would need to be changed, wrote up a lief script to handle placing them in same order as game, and with this libcd actually links in fine 🐱

import lief

# Load the object file
obj = lief.parse("cdrom.o")

# Each .bss symbol in cdrom.o, in the order they appear in SH
sym_table = ["Stframe_no", "Stsector_offset", "StRgb24", "StCdIntrFlag", "StCHANNEL", "StStartFrame", "CChannel", "StFinalSector", "StEmu_Idx", "StRingIdx1", "StRingIdx2", "StRingIdx3", "StEmu_Addr", "StEndFrame", "StSTART_FLAG", "StRingBase", "StRingAddr", "StRingSize"]

cur_addr = 0

for sym_name in sym_table:
    sym = obj.get_symbol(sym_name)
    if sym:
        print(f"{sym_name}: 0x{sym.value:x}")
    else:
        print(f"{sym_name}: not found")
    # update symbol to point to new addr
    sym.value = cur_addr
    cur_addr += 4

obj.write("cdrom_new.o")

Note for anyone working on similar issue: this script only really changes where the symbols in the .o point to (and maybe fixing relocations?), this can fix other objects that reference those symbols, but doesn't actually update any code in the object being modified, so there may still be code that needs to be edited manually after this. Luckily in SH1 there was only a single .o that needed that.

Need to look into libgs next, but that might be harder since there's a bunch of .o files that have reordered bss IIRC...