[1.5.0 -> 2.1.1] Conversion help with icu::properties.

[rizzen-yazston]

I need help converting the lines below using ICU4X 1.5.0 to the approach used in version 2.1.1.

        let syntax = icu_properties::sets::load_pattern_syntax(data_provider)?;
        let white_space = icu_properties::sets::load_pattern_white_space(data_provider)?;

Where data_provider is either BlobDataProvider or FsDataProvider in the two functions containing the lines above.

[Manishearth]

So these things are now constructed with methods on CodePointSetData.

https://unicode-org.github.io/icu4x/rustdoc/icu/properties/struct.CodePointSetData.html#method.new

But looking at this, we no longer have buffer provider methods on CodePointSetData.

Generally we suggest people use baked data (CodePointSetData::new()) for icu::properties since there's almost no downside: what about your use case requires blob data? Are you trying to download properties data on demand?

Either way, we probably should add a buffer constructor, cc @sffc

[rizzen-yazston]

Originally, ICU4X only had data providers for external data. Later, internal baked data became available, which was a significant feature, particularly when only specific properties were included. However, at the time, the baked data was only a subset of the CLDR data, and if needed, more comprehensive data than the basic data would require the use of external data stored in files. Additionally, data providers allowed for custom data of private regions, which, to the best of my knowledge, is not included in the CLDR.

[rizzen-yazston]

I am a bit stumped on how to use

I believe we need an example (even a partial will so) on how to use CodePointSetData::try_new_unstable(), that is equivalent to this fragment of 1.5.0 code:

                let fs_provider = fs.as_deserializing();
                syntax = Some(load_pattern_syntax(&fs_provider)?);
                white_space = Some(load_pattern_white_space(&fs_provider)?);

where fs is a passed parameter of type &FsDataProvider.

For CodePointSetData::try_new_unstable(), I am not seeing how to obtain separate CodePointSetData instances for syntax and white_space, whereas in ICU4X 1.5.0, there were distinct functions for both property types.

[Manishearth]

You need to use the P generic parameter. CodePointSetData::try_new_unstable::<WhiteSpace>(&provider)

[rizzen-yazston]

I am clearly missing something with this function (most likely related to data_provider: &BlobDataProvider):

    #[cfg(feature = "blob")]
    pub fn try_new_with_buffer_provider(
        ,
    ) -> Result<IcuDataProvider, IcuError> {
        #[cfg(feature = "log")]
        debug!("Creating IcuDataProvider using buffer provider.");

        let grapheme_segmenter =
            GraphemeClusterSegmenter::try_new_with_buffer_provider(data_provider)?;
        let syntax =
            CodePointSetData::try_new_unstable::<PatternSyntax>(&data_provider.as_deserializing())?;
        let white_space = CodePointSetData::try_new_unstable::<PatternWhiteSpace>(
            &data_provider.as_deserializing(),
        )?;
        Ok(IcuDataProvider::Blob(IcuData {
            grapheme_segmenter,
            syntax,
            white_space,
        }))
    }

where both errors are related to the let syntax = ... and let white_space = ... statements.

Providing the error message for one error, the solution for the one statement would also apply to the other statement:

the trait bound PropertyCodePointSet<'de>: serde::Deserialize<'de> is not satisfied
for local types consider adding #[derive(serde::Deserialize)] to your PropertyCodePointSet<'de> type
for types from other crates check whether the crate offers a serde feature flag
the following other types implement trait serde_core::de::Deserialize<'de>:
&’a Path
&’a [u8]
&’a icu_collections::codepointinvliststringlist::CodePointInversionListAndStringListULE
&’a potential_utf::ustr::PotentialUtf8
&’a str
&’a zerovec::ule::tuplevar::Tuple2VarULE<A, B, Format>
&’a zerovec::ule::tuplevar::Tuple3VarULE<A, B, C, Format>
&’a zerovec::ule::tuplevar::Tuple4VarULE<A, B, C, D, Format>
and 218 others
required for DeserializingBufferProvider<'_, BlobDataProvider> to implement DataProvider<PropertyBinaryPatternSyntaxV1> (rustc E0277)

I am having no error with the grapheme segmenter.

[rizzen-yazston]

Found the cause of the issue by taking a wild guess. In Cargo.toml, the line icu_properties was missing , features = [ "serde" ].

Perhaps the crates of ICU4X could include the available features for the crate on the root document page.

[sffc]

Thanks for the feedback!

The docs for GraphemeClusterSegmenter::try_new_with_buffer_provider call out the serde feature, but not the trait AsDeserializingBufferProvider! Fixed in #7408