implement Serialization serialize/deserialize

[stevengj]

This package should implement custom methods for Serialization.serialize and deserialize of ArbFloat etcetera.

Right now, it uses the fallback methods, which just read and write the raw struct contents. This fails when the precision becomes sufficiently large, in which case the struct contents include C pointers, leading to errors in packages that rely on Julia serialization/deserialization such as Distributed.jl — see #74 and this discourse thread.

See also this discussion on Google Groups, in which the lead FLINT developer (Fredrik Johansson) recommends arb_dump_str and arb_load_str for serialization/deserialization.

For example, here is a rough draft of a serialize method for ArbFloat:

using Serialization

function Serialization.serialize(s::AbstractSerializer, x::ArbFloat)
    serialize_type(s, typeof(x))
    unsafestr = ccall(@libarb(arb_dump_str), Cstring, (Ref{ArbReal},), x)
    len = Int64(ccall(:strlen, Csize_t, (Cstring,), unsafestr))
    write(s.io, len)
    unsafe_write(s.io, pointer(unsafestr), len)
    ccall(@libflint(flint_free), Cvoid, (Cstring,), unsafestr)
    return nothing
end

It uses unsafe_write, which allows it to directly write the arb_dump_str buffer contents to the serialization stream; I'm being a little low-level here because serialization is often performance-critical and so I want to avoid unnecessary copies of the string data. The inverse deserialize function should look something like:

function Serialization.deserialize(s::AbstractSerializer, ::Type{ArbFloat{P}}) where {P}
    len = read(s.io, Int64)::Int64
    str = Serialization.deserialize_string(s, Int(len))

    # TODO: call arb_load_str to constuct arb_t from str

    return nothing
end

This seems like it will require a new low-level constructor to use arb_load_str instead of arb_init.

[JeffreySarnoff]

I have spent the better part of a week on this getting nowhere. :)
Details to follow.

[JeffreySarnoff]

I sent this message to the flintlib/flint mailing list

[arf|arb]_dump_str seems to work for me
[arf|arb]_load_str are confounding
I'm sure I have overlooked something, still after trying any number of ways without success -- how should this work?

in Julia

using ArbNumerics
using ArbNumerics: @libarb, @libflint # the same with FLINT3 

function arf_dump_str(x::ArbFloat{P}) where {P}
    str = Ref{Ptr{Cchar}}(C_NULL)
    ccall(@libarb(arf_dump_str), Ptr{Cchar}, 
             (Ref{ArbFloat{P}}, Ref{Ptr{Cchar}}), x, str)
end

arf_dump_string(x::ArbFloat{P}) where {P} =
   unsafe_string(arf_dump_str(x))

function arb_dump_str(x::ArbReal{P}) where {P}
   str = Ref{Ptr{Cchar}}(C_NULL)
   ccall(@libarb(arb_dump_str), Ptr{Cchar},
            (Ref{ArbReal{P}}, Ref{Ptr{Cchar}}), x, str)
end

arb_dump_string(x::ArbReal{P}) where {P} =
   unsafe_string(arb_dump_str(x))

setworkingprecision(ArbFloat, 60)
arf = ArbFloat(4.0)
arb = ArbReal(4.0)

arf_str = arf_dump_str(arf)
# Ptr{Int8}  @0x000001901e7361d0
arf_string = arf_dump_string(arf)
# "1 2"

arb_str = arb_dump_str(arb)
# Ptr{Int8} @0x000001901e7360a0
arb_cstr = Cstring(arb_str)
# Cstring(0x000001901e7360a0)
arb_string = arb_dump_string(arb)
# "1 2 0 0"

# [arf|arb]_load_str should take the output of [arf|arb]_dump_str
# and set an ArbFloat|ArbReal variable to match the str's value
# int arb_load_str(arb_t x, const char * str)

# One of these types must be correct
const StrT = Union{String, Cstring, Ptr{Int8}, Ptr{UInt8}}

function arb_load_str(x::ArbReal{P}, str::T) where {P, T<:StrT}
    ok = ccall(@libarb(arb_load_str), Cint, 
                      (Ref{ArbReal}, Ref{T}), x, str)
    if !iszero(ok)
       throw(ErrorException("arb_load_str failed"))
    end
    x
end

z = ArbReal(7)
arb_load_str(z, arb_str)      # ERROR: arb_load_str failed
arb_load_str(z, arb_string)   # ERROR: arb_load_str failed
arb_load_str(z, arb_cstr)     # ERROR: arb_load_str failed

[JeffreySarnoff]

This was an issue for me

     arf_load_str is documented to accept the output from arf_dump_str
in Julia, it does not: arf_load_str accepts unsafe_string(arf_dump_str(arf))

     arb_load_str is documented to accept the output from arb_dump_str
in Julia, it does not: arb_load_str accepts unsafe_string(arb_dump_str(arb))

shown with Arblib.jl for Arf

arf_dump_cstring(x::Arf)::Cstring = Arblib.dump(x)  # returns Cstring
arf_dump_string(x::Arf)::String = unsafe_string(arf_dump_cstring(x))

arf_load_string(target::Arf, source::String)::Arf = Arblib.load!(target, source)
arf_load_cstring(target::Arf, source::Cstring)::Arf = arf_load_string(target, unsafe_string(source))

arf = Arf(16)
arf_to_set = Arf(0)

arf_cstr = arf_dump_cstring(arf)        # Cstring(0x0000018d404e0660) e.g.
arf_str = unsafe_string(arf_cstr)       # "1 4"

arf_load_string(arf_to_set, arf_str);
arf_to_set == arf                       # true

[JeffreySarnoff]

[I duplicate messages sent to the flintlib/flint mailing list here, for later reference]

This using ArbNumerics remains elusive.
Any insight?

using ArbNumerics
using ArbNumerics: @libarb, @libflint # these are the same in FLINT3

function arbfloat_dump_cstring(x::ArbFloat{P}) where {P}
    cstr = ccall(@libarb(arf_dump_str), Cstring,
                 (Ref{ArbFloat},), x)
    cstr
end
    
function arbfloat_dump_string(x::ArbFloat{P}) where {P}
    cstr = ccall(@libarb(arf_dump_str), Cstring,
                 (Ref{ArbFloat},), x)
    unsafe_string(cstr)
end

arbfloat = ArbFloat(16.0)
arbfloat_to_set = ArbFloat(0.0)

arbfloat_cstring = arbfloat_dump_cstring(arbfloat)
# Cstring(0x000001cc87411990) e.g.

arbfloat_string = arbfloat_dump_string(arbfloat)
# "1 4"

function arbfloat_load_string(target::ArbFloat{P}, source::String) where {P}
    ok = ccall(@libarb(arf_load_str), Cint,
                (Ref{ArbFloat}, Ref{String}), target, source)
    target
end

arbfloat_load_string(arbfloat_to_set, arbfloat_string)

(arbfloat_to_set == arbfloat) == false
iszero(arbfloat_to_set) == true

[JeffreySarnoff]

Well this, dealing with serialization and deserialization of ArbFloat, ArbReal, and ArbComplex values at high precision, has become unfun. The only apparently robustly stable approach I found of the many investigated is far too inelegant. Searching the serialized data as a raw sequence of Chars for the index of the final char in the type name and for the index of the first char in the "dumped" internal string are two of the most unjulian -- and I don't do unjulianity.

The Arblib.jl package appears to have this well in hand, and their low-level library interfacing design has good features that were not part of the play years earlier when I wrote ArbNumerics. It may be best to use their groundwork and redevelop the things that people like most about ArbNumerics atop that. There is a very new type within Flint3 that is of note, nfloat. This new floating-point (not ball) type is introduced to allow increased performance with vector and matrix ops. Its capabilities will increase as development effort continues.

So, long story short, that's the story.

[Edsnowden233]

Well this, dealing with serialization and deserialization of ArbFloat, ArbReal, and ArbComplex values at high precision, has become unfun. The only apparently robustly stable approach I found of the many investigated is far too inelegant. Searching the serialized data as a raw sequence of Chars for the index of the final char in the type name and for the index of the first char in the "dumped" internal string are two of the most unjulian -- and I don't do unjulianity.

The Arblib.jl package appears to have this well in hand, and their low-level library interfacing design has good features that were not part of the play years earlier when I wrote ArbNumerics. It may be best to use their groundwork and redevelop the things that people like most about ArbNumerics atop that. There is a very new type within Flint3 that is of note, nfloat. This new floating-point (not ball) type is introduced to allow increased performance with vector and matrix ops. Its capabilities will increase as development effort continues.

So, long story short, that's the story.

Hi Jeff,

I used Claude 3.7 in Cursor to solve this bug, using the discussion in this issue as the prompt. It seems to fix it using the Julia Serialization module. My previous test code has no problem with serializing the data on the distributed work for high precision. I have sent you the new version on Julia Slack. The following is the major change made by AI:

Serialization Support for ArbNumerics

ArbNumerics.jl now provides robust serialization support for all Arb numeric types,
including ArbFloat, ArbReal, and ArbComplex. This functionality is crucial
for distributed computing and saving/loading high-precision numerical values.

Background

Prior to this implementation, attempting to serialize Arb numeric types with precision
higher than 98 bits would often result in memory explosion or serialization errors.
This is because the default Julia serialization would attempt to directly serialize
the raw struct contents, which include C pointers for high-precision values.

The new implementation uses string representation and parsing for proper serialization
and deserialization, preserving the full precision of the values.

Usage

Serialization works with the standard Julia Serialization module:

using ArbNumerics
using Serialization

# Create high-precision ArbFloat
setworkingprecision(ArbFloat, 1024)
x = ArbFloat(π)

# Serialize to a file
open("high_precision_pi.bin", "w") do io
    serialize(io, x)
end

# Deserialize from a file
y = open(deserialize, "high_precision_pi.bin")

# Values should be equal
x == y  # true

The same approach works for ArbReal and ArbComplex types.

Supported Operations

• Serializing individual Arb numeric values
• Serializing arrays or collections containing Arb numeric values
• Serializing nested structures with Arb numeric values
• Distributed computing with Arb numeric values (e.g., using Distributed.jl)

Implementation Details

For each Arb numeric type, we've implemented:

1. Custom Serialization.serialize methods that:

   • Convert the value to a string representation
   • Serialize this string and precision information

2. Custom Serialization.deserialize methods that:

   • Read the serialized string and precision
   • Reconstruct the value using the appropriate constructor

These methods avoid the direct serialization of C pointers and other low-level details
of the Arb numeric types, ensuring safe and reliable serialization.

For Distributed Computing

When using Arb numeric types with Distributed.jl, make sure to load ArbNumerics on all workers:

using Distributed
addprocs(4)

@everywhere using ArbNumerics
@everywhere using Serialization

# Now you can safely use high-precision values across workers
setworkingprecision(ArbFloat, 5000)
x = ArbFloat(π)

# This will work correctly even with very high precision
results = @distributed (vcat) for i in 1:4
    # Use x in calculations
    [x + i]
end

Example

See the full example in examples/serialization_example.jl.