Improve pickling

CHANGES

@@ -1,5 +1,25 @@
 This file describes changes in the IO package.
 
+Changes since 4.10.0
+  - Pickle free and finitely presented groups, their elements and their
+    subgroups
+  - Stop unpickling from evaluating its input: permutations, finite field
+    elements and cyclotomics used to be stored as their printed form and read
+    back with EvalString, so a pickle from an untrusted source could run
+    arbitrary code. They now use formats that need no evaluation, and the old
+    formats are parsed rather than evaluated. Pickles written by earlier
+    versions are still readable; pickles written by this version need IO 4.11
+    or newer to read
+  - Unpickle functions stored by name without evaluating anything, and refuse
+    to evaluate pickled function source unless
+    IO_UnpickleAllowEvalOfFunctions is set to true
+  - Pickle rationals, infinity and -infinity directly
+  - Document `IO_Pickle( ob )` and `IO_Unpickle( str )`, which pickle to and
+    from a string instead of a file. These have existed since 2011 but were
+    never documented
+  - Make `IO_Pickle( ob )` report failure instead of returning a truncated
+    string, and drop its 1 MB preallocation
+
 4.10.0 (2026-07-14)
   - Handle chunked HTTP responses without waiting for the server to close the
     connection

PackageInfo.g

@@ -7,7 +7,7 @@ SetPackageInfo( rec(
 
 PackageName := "IO",
 Subtitle := "Bindings for low level C library I/O routines",
-Version := "4.10.0",
+Version := "4.10.0dev",
 Date := "14/07/2026", # dd/mm/yyyy format
 License := "GPL-3.0-or-later",
 

doc/main.xml

@@ -2050,11 +2050,40 @@ to each other and to other objects.<P/>
         rational functions,
         lists, records, compressed vectors and matrices over finite fields
         (objects are uncompressed in the byte stream but recompressed
-        during unpickling), and straight line programs.
+        during unpickling), straight line programs, and free and finitely
+        presented groups together with their elements and subgroups.
         <P/>
         Self-referential objects built from records and lists are handled
         correctly and are restored completely with the same self-references
         during unpickling.
+        <P/>
+        An element of a free or finitely presented group is only usable
+        next to the group it came from, so pickling one also pickles that
+        group. Objects pickled <E>together</E> share the group and can be
+        combined afterwards, but objects pickled in <E>separate</E> calls
+        cannot: each call rebuilds a group of its own, and &GAP; keeps
+        elements of different groups apart even when the presentations
+        agree. Unpickling never re-uses a group that already exists in the
+        session either. So pickle a list of elements, not one element at a
+        time.
+    </Description>
+</ManSection>
+
+<ManSection>
+    <Oper Name="IO_Pickle" Arg="ob" Label="to a string"/>
+    <Returns> a string or <C>IO&uscore;Error</C> </Returns>
+    <Description>
+        As above, except that the result is returned as a string instead of
+        being written to a <C>File</C> object. Use this to keep pickled
+        objects in memory, or to hand them to code that has nowhere to write
+        a file. If the object cannot be pickled, <C>IO&uscore;Error</C> is
+        returned. Read the string back with
+        <Ref Oper="IO_Unpickle" Label="from a string"/>.
+<Example>
+gap> s := IO_Pickle( [ 1, "two", (3,4) ] );;
+gap> IO_Unpickle( s );
+[ 1, "two", (3,4) ]
+</Example>
     </Description>
 </ManSection>
 
@@ -2073,6 +2102,30 @@ to each other and to other objects.<P/>
     </Description>
 </ManSection>
 
+<ManSection>
+    <Oper Name="IO_Unpickle" Arg="str" Label="from a string"/>
+    <Returns> <C>IO&uscore;Error</C> or a &GAP; object </Returns>
+    <Description>
+        Unpickles the first object stored in the string <A>str</A>, which
+        may be immutable and is not changed. Anything in <A>str</A> after
+        that first object is ignored.
+    </Description>
+</ManSection>
+
+<ManSection>
+    <Var Name="IO_UnpickleAllowEvalOfFunctions"/>
+    <Description>
+        A pickled function is stored as the name of the global variable
+        holding it where possible, and as its source code otherwise.
+        Reading the latter back means evaluating it, which lets a pickle
+        from an untrusted source run arbitrary code, so
+        <Ref Oper="IO_Unpickle"/> refuses to do it and returns
+        <C>IO&uscore;Error</C> unless this variable is set to
+        <K>true</K>. It is <K>false</K> by default. Functions stored by
+        name are unpickled either way.
+    </Description>
+</ManSection>
+
 <ManSection>
     <Func Name="IO_ClearPickleCache" Arg=""/>
     <Returns> Nothing </Returns>
@@ -2154,6 +2207,29 @@ created object or the value <C>IO&uscore;Error</C> if an error occurred.
 They should never go into a break loop, because after leaving the user
 has to call <Ref Func="IO_ClearPickleCache"/> explicitly!
 
+<P/>
+The helpers <C>IO&uscore;GenericObjectPickler</C> and
+<C>IO&uscore;GenericObjectUnpickler</C> cover the common case of an object
+that is rebuilt from a few constituents and then has attributes set on
+it. Those constituents, the <Q>prepickled</Q> objects, are written
+<E>before</E> the object itself is added to the pickle cache, since the
+object cannot be created without them. One consequence is easy to trip
+over: at the top of a stream the cache is empty, so each prepickled
+object is written, and the cache discarded, independently of the next.
+Two prepickled objects that are the same object in memory are therefore
+written twice and come back as two objects when the pickle is the
+outermost one, though not when it sits inside a list or record.
+<P/>
+So a pickler using these helpers must not rely on one prepickled object
+referring to another. Store plain data instead and rebuild the
+references afterwards, as the picklers for finitely presented groups in
+<F>pkg/io/gap/pickle.gi</F> do: they write the external representations
+of words rather than the words themselves, and turn them back into
+elements of a group the unpickler already holds. Note that this is a
+property of the byte format, not just of the implementation — changing
+when the cache is cleared would change the meaning of the
+self-references in every pickle already written.
+
 <P/>
 Perhaps the best way to learn how to extend the framework is to
 study the code for the basic &GAP; objects in the file

gap/pickle.gd

@@ -21,8 +21,11 @@ DeclareGlobalFunction( "IO_ReadSmallInt" );
 DeclareGlobalFunction( "IO_WriteAttribute" );
 DeclareGlobalFunction( "IO_ReadAttribute" );
 DeclareGlobalFunction( "IO_PickleByString" );
-DeclareGlobalFunction( "IO_UnpickleByEvalString" );
+DeclareGlobalFunction( "IO_UnpickleByEvalString" );   # deprecated, see below
 DeclareGlobalFunction( "IO_UnpickleByFunction" );
+DeclareGlobalFunction( "IO_UnpickleByParser" );
+DeclareGlobalFunction( "IO_ParseLegacyExpression" );
+DeclareGlobalFunction( "IO_ParsePermString" );
 DeclareGlobalFunction( "IO_GenericObjectPickler" );
 DeclareGlobalFunction( "IO_GenericObjectUnpickler" );
 
@@ -32,14 +35,25 @@ DeclareOperation( "IO_Pickle", [ IsObject ]);
 DeclareOperation( "IO_Unpickle", [ IsStringRep ]);
 BindGlobal ("IO_Unpicklers", rec() );
 
+# Unpickling the source of a function means evaluating it, which lets a
+# hostile pickle run arbitrary code, so it is refused unless this is set.
+# Functions that are global variables are unpickled by name either way.
+IO_UnpickleAllowEvalOfFunctions := false;
+
 # Here is an overview over the defined tags in this package:
 #
 # CHAR  a character
-# CYCL  a cyclotomic
+# CYCC  a cyclotomic, as its coefficients over the rationals
 # FAIL  fail
 # FALS  false
-# FFEL  a finite field element
+# FFEC  a finite field element, as its coefficients over the prime field
 # FLOT  a Floating point number
+# FPEL  an element of a finitely presented group
+# FPGR  a finitely presented group
+# FPSG  a subgroup of a free or finitely presented group
+# FRAC  a rational number
+# FREG  a free group
+# FREW  an element of a free group
 # FUNC  a GAP function, if it is a global one, only its name is pickled
 # GAPL  a gap in a list (unbound entries)
 # GSLP  a GAP straight line program
@@ -60,11 +74,13 @@ BindGlobal ("IO_Unpicklers", rec() );
 # MREC  a mutable record
 # MRNG  a mutable range
 # MSTR  a mutable string
+# NINF  minus infinity
 # OPER  a GAP operation, only its name is pickled
-# PERM  a permutation
-# PPER  a partial permutation
+# PINF  infinity
 # POLF  an object in the representation IsPolynomialDefaultRep
 # POLY  a Laurent polynomial (or a rational function) deprecated
+# PPER  a partial permutation
+# PRML  a permutation, as its list of images
 # RATF  an object in the representation IsRationalFunctionDefaultRep
 # RSGL  the global random source
 # RSGA  a GAP random source
@@ -76,6 +92,15 @@ BindGlobal ("IO_Unpicklers", rec() );
 # UPOL  an object in the representation IsLaurentPolynomialDefaultRep
 # URFU  an object in the representation IsUnivariateRationalFunctionDefaultRep
 #
+# These tags are only read, never written. They store the printed form of the
+# object, which used to be read back with EvalString; that let a hostile
+# pickle run arbitrary code, so they are parsed now and were replaced by the
+# tags above. Files written before IO 4.11 still use them.
+#
+# CYCL  a cyclotomic
+# FFEL  a finite field element
+# PERM  a permutation
+#
 # Some tags defined in other packages:
 #
 # ICVC  an immutable cvec