Add support for Smithy bigInteger and bigDecimal types as string wrappers in aws-smithy-types, allowing users to parse with their preferred big number library.

[AmitKulkarni23]

Motivation and Context

Fixes #312

Smithy defines bigInteger and bigDecimal types for arbitrary-precision numbers, but smithy-rs had TODO placeholders instead of implementations. This prevented users from working with services that use these types.

Description

• Added BigInteger and BigDecimal runtime types in aws-smithy-types as string wrappers
• Implemented JSON serialization/deserialization in codegen
• String-based approach allows users to choose their preferred big number library (e.g., num-bigint, rust_decimal)
• Added unit tests and integration tests with protocol test coverage

Testing

• Added unit tests in SymbolVisitorTest.kt
• Created integration test model big-numbers.smithy with protocol tests
• All codegen-core tests pass
• All codegen-client-test integration tests pass

Checklist

• [ x] For changes to the smithy-rs codegen or runtime crates, I have created a changelog entry Markdown file in the .changelog directory, specifying "client," "server," or both in the applies_to key.

---

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[rcoh]

nice! we need to clean up those representation in aws-smithy-types so that are forward compatible with eventual improvements

[rcoh]

we shouldn't expose methods like this since they will probably be impossible to implement if we eventually switch to using a real internal representation that's not a string

[AmitKulkarni23]

1. Remove as_str() method - it's redundant and limiting
2. Keep AsRef trait - works with any internal representation
3. Update codegen to use .as_ref() instead of .as_str()

^^ Does this work?

[AmitKulkarni23]

Addressed this in the latest revision of PR

[rcoh]

should implement FromStr instead

[AmitKulkarni23]

Ok. Will use FromStr instead of new in next revision.

[AmitKulkarni23]

Addressed this in the latest revision of PR

[rcoh]

we probably need to support this for more than just json protocols. also need protocol tests. does smithy have any protocol tests for these yet?

[AmitKulkarni23]

Ok. I will add serialization and deserialization code for XML, CBOR protocols in the next revision.

[AmitKulkarni23]

does smithy have any protocol tests for these yet?

Protocol tests exist in misc.smithy but BigInteger/BigDecimal are commented out - https://github.com/smithy-lang/smithy-rs/blob/main/codegen-core/common-test-models/misc.smithy#L100. I will uncomment them now that the implementation is complete. However, it seems like misc.smithy only tests JSON. I will look at references and add protocol tests.

[AmitKulkarni23]

Addressed this in the latest revision of PR

[aajtodd]

Also many (all?) of the supported JSON protocols represent these as regular JSON numbers: https://smithy.io/2.0/aws/protocols/aws-json-1_0-protocol.html#shape-serialization

[AmitKulkarni23]

Thanks for the reference. I will use expect_number_or_null instead of expect_string_or_null

[AmitKulkarni23]

Addressed this in the latest revision of PR

[ysaito1001]

For the same reason as

if we eventually switch to using a real internal representation that's not a string

We could consider delaying adding this to leave an option for the future, unless this conversion is required right now.

[AmitKulkarni23]

Ok. Will remove this.

[AmitKulkarni23]

Addressed this in the latest revision of PR

[AmitKulkarni23]

Note to reviewers:

I have added serialization and parsing logic for the following protocols:

• JSON
• CBOR
• XML
• AWS Query
• AWS EC2

Let me know if there are any other protocols.

[AmitKulkarni23]

Question for reviewrs: Default Values for BigInteger/BigDecimal

I've implemented Default trait for both types to support error correction in client codegen:

Context:
ErrorCorrection.kt line 67 generates Some(Default::default()) for all NumberShape types, including BigInteger/BigDecimal, when required fields are missing during deserialization.

Are "0" and "0.0" appropriate defaults for error correction scenarios?

[rcoh]

we should match whatever we do for normal integers

[AmitKulkarni23]

My thought process:

BigInteger/BigDecimal Default implementations match primitive number behavior:

i32::default()  = 0
i64::default()  = 0
f32::default()  = 0
f64::default()  = 0
u32::default()  = 0
u64::default()  = 0
i8::default()   = 0
i16::default()  = 0

• BigInteger::default() returns BigInteger("0") (string "0" representing zero)
• BigDecimal::default() returns BigDecimal("0.0") (string "0.0" representing zero)

All number types default to their zero representation. BigInteger/BigDecimal use string storage for arbitrary precision, but semantically represent the same zero value as primitive numbers.

[rcoh]

Overall looks pretty good. We can decide how we want to handle large numbers in JSON -- currently as you have implemented there will be a loss of precision (but there is no inherent need for that since we control aws-smithy-json and can have it parse a number as a string directly.

[rcoh]

shouldn't we actually use numbers that don't fit into int/decimals?

[AmitKulkarni23]

The test values are limited by Smithy's Java-based model parser, which converts numeric literals to Java Number types. When these are serialized back to strings, Java uses scientific notation for large values.

Example of the parser limitation:

params: {
   bigDec: 123456789012345.123456789
}

After Smithy parses this, the codegen sees: 1.2345678901234512E14 (scientific notation with precision loss)

However, this is only a test limitation - the actual runtime code handles arbitrary precision correctly:

• XML/JSON input is parsed as strings from the wire
• BigDecimal/BigInteger use FromStr to parse directly from those strings
• Serialization writes the string back via .as_ref()
• No precision loss occurs in production code

[rcoh]

gotcha. then we should write an integration test for this against the generated code manually. You can do this by writing a test in kotlin that generates the service, then utilizes the serializers

[AmitKulkarni23]

I have done this as part of https://github.com/smithy-lang/smithy-rs/pull/4418/files#diff-ae00c010398af1c58c9129aef4393407a833d72c73b2063ad28e7754d3ae4eedR385

let input = crate::test_input::BigNumberOpInput::builder().payload(
    crate::test_model::BigNumberData::builder()
        .big_int("12345678901234567890".parse().unwrap())
        .big_dec("3.141592653589793238".parse().unwrap())
        .build()
).build().unwrap();
let serialized = ${format(operationSerializer)}(&input.payload.unwrap()).unwrap();
let output = std::str::from_utf8(&serialized).unwrap();
assert!(output.contains("<bigInt>12345678901234567890</bigInt>"));
assert!(output.contains("<bigDec>3.141592653589793238</bigDec>"));

[AmitKulkarni23]

I am going to remove this unit test which has numbers that don't fit into int/decimals in next revison of the PR

[rcoh]

I assume this is what the spec said for CBOR?

[AmitKulkarni23]

The current implementation uses decoder.str() / encoder.str() (CBOR text strings, Major Type 3) as a temporary approach.

According to the Smithy RPC v2 CBOR spec, BigInteger/BigDecimal should use:

• BigInteger: Major Type 6, tags 2 (unsigned bignum) or 3 (negative bignum)
• BigDecimal: Major Type 6, tag 4 (decimal fraction)

However, aws-smithy-cbor doesn't currently expose methods for these CBOR tags. The underlying minicbor library supports tags (used internally for timestamps), but we'd need to add public methods like encoder.bignum() and encoder.decimal() to properly implement the spec.

Any suggestions on how to address this? How do you recommend I proceed here?

[rcoh]

modify aws-smithy-cbor — you can find the source of it in this repo

[AmitKulkarni23]

Need some direction:

The CBOR spec requires binary encoding (tags 2/3/4), but BigInteger/BigDecimal are string wrappers to avoid choosing a bignum library.

Should we:

1. Keep current text string encoding (non-compliant but simple)
2. Document that BigInteger/BigDecimal don't work with CBOR
3. Add num-bigint dependency to aws-smithy-cbor for spec compliance

[rcoh]

Adding a dependency to aws-smithy-cbor seems OK.

Option 1 is a non-starter — we can't have non-compliant code in smithy-rs.

I would prefer 3, but for simplicity, we could have 2 (but it must FAIL to codegen at runtime, it can't be only a documented feature).

[AmitKulkarni23]

Going ahead with Option 2 - failing codegen at runtime

[rcoh]

I don't think this error is infallible is it? wouldn't this happen if the string wasn't a valid big decimal? this error seems worth preserving?

[AmitKulkarni23]

Should we validate the string is a valid number format?

Options:

1. Keep infallible - Accept any string, let users validate when they parse it
2. Add validation - Check the string is a valid number format, return error if not

What do you recommend?

[rcoh]

ah I see...a bit of a can of worms. I forgot we were basically doing nothing with the numeric values. We can punt this for now.

[rcoh]

I think do actually do this properly you need to add some additional code to aws-smithy-json to parse a number as a string? not sure how hard that would be.

As it is, this isn't terrible, but its not ideal since it defeats the point

[rcoh]

we should match whatever we do for normal integers

[rcoh]

is this correct? I definitely want to see some tests for this.

[rcoh]

nice test! please add something similar for JSON

[rcoh]

bump on this test — we need a test that actually tests that we are preserving E2E precision with JSON.

[AmitKulkarni23]

Agreed. Adding a full end to end Kotlin test for XML protocol that actually serializes and deserializes big numbers in the next commit.

[rcoh]

there appears to be some code that handles E / scientific notation but I don't see any tests of that here

[AmitKulkarni23]

Adding tests for these in the next commit.

[rcoh]

Must fix:

• Must validate inputs in to BigInteger / BigDecimal because we are raw-writing them into JSON. We may want to also improve that API to safe guard that we are only writing "safe" characters?
• Must add a Kotlin test that validates we successfully round trip large values through the serializers (since the protocol tests do not) Can the protocol tests use a number that is actually out of range?
• Few other more minor inline comments

Thanks for your continued hard work on this!

[rcoh]

this isn't larger than u64::max — I guess that's a protocol test limitation? This is 2^53-1 (max safe integer in Javascript), so we aren't really testing that big integers work (e.g. this code would pass even without your changes right?)

[rcoh]

why allow(unused)? parsers should only be generated when they are actually used.

[AmitKulkarni23]

When I was running one build commands, compilation was failing. Rust was complaining that some of the variuables were unused. Therefore, I had added this exception. Will try to reproduce this and add more details here.

[rcoh]

can you just change your parser to match all the other ones and use value?

[AmitKulkarni23]

The nested parsers are called with input as the second parameter. Since the top-level function has value as its parameter, I created the alias let input = value; so we could pass input
to the nested parsers. I will try to incorporate this review comment.

[rcoh]

We need to make this fallible — see other comments.

[AmitKulkarni23]

Agreed. Here is my plan for the next commit:

1. Add validation function is_valid_number_string() that only allows valid JSON number characters:

   • Digits: 0-9
   • Signs: -, +
   • Decimal: .
   • Scientific notation: e, E
   • Rejects JSON special characters: quotes, commas, braces, brackets, etc.

2. Create proper error type:

   #[derive(Debug, Clone, PartialEq, Eq)]
   #[non_exhaustive]
   pub enum BigNumberError {
       InvalidFormat(String),
   }

[rcoh]

this is a semver hazard — it should return an enum marked with #[non_exhaustive] so an error could be added in the future

[AmitKulkarni23]

Understood. Adding the below in the next revision/commit

#[derive(Debug, Clone, PartialEq, Eq)]
#[non_exhaustive]
pub enum BigNumberError {
    InvalidFormat(String),
}

[rcoh]

this impl is probably too hazardous to keep

[AmitKulkarni23]

Here is my understanding of what this comment means:

• From trait must always succeed (infallible)
• With validation, invalid strings should return errors, not panic
• Panicking in From is unexpected and dangerous
• Users should use FromStr instead, which is properly fallible

^^ Assuming that this is right, I am going to remove From<String> implementations for BigInteger and BigDecimal. Users must now use FromStr::from_str() which properly returns Result<T, BigNumberError>.

[rcoh]

empty impl block does nothing

Suggested change

impl BigInteger {}

[AmitKulkarni23]

Agreed. Removing this in next revision/commit

[rcoh]

hmm...there is actually a vulnerability here or at least the possibility to introduce invalid JSON — we are not validating the input to BigInteger and BigDecimal and then we're writing them untrusted directly into the JSON.

We need to validate that they are valid before storing them.

[AmitKulkarni23]

Understood.
Introducing a simple validation function here:

fn is_valid_number_string(s: &str) -> bool {
    if s.is_empty() {
        return false;
    }
    
    s.chars().all(|c| matches!(c, '0'..='9' | '-' | '+' | '.' | 'e' | 'E'))
}

^^ These are all the valid characters that I could think of in any BigNumber

And using it as:

impl std::str::FromStr for BigInteger {
    type Err = BigNumberError;

    fn from_str(s: &str) -> Result<Self, Self::Err> {
        if !is_valid_number_string(s) {
            return Err(BigNumberError::InvalidFormat(s.to_string()));
        }
        Ok(Self(s.to_string()))
    }
}

[rcoh]

big integer is only numbers — the larger set is only for BigDecimal

[rcoh]

bump on this test — we need a test that actually tests that we are preserving E2E precision with JSON.

[rcoh]

why allow(unused)?

[rcoh]

I think we're really close here!

[rcoh]

Suggested change

	"<#{BigDecimal} as ::std::str::FromStr>::from_str(${value.dq()}).unwrap()",
	"<#{BigDecimal} as ::std::str::FromStr>::from_str(${value.dq()}).expect("invalid string for BigDecimal")",

[rcoh]

Suggested change

	"<#{BigInteger} as ::std::str::FromStr>::from_str(${value.dq()}).unwrap()",
	"<#{BigInteger} as ::std::str::FromStr>::from_str(${value.dq()}).expect("Invalid string for big integer")",

[rcoh]

is there an open ticket for this? If not, please open one and then link to the ticket in the error

[AmitKulkarni23]

#4473 - Linking this in the error as well

[rcoh]

I'm really confused — this was value before...why did you need to make it _value?

[rcoh]

ah—I see the issue. Nested functions need access to it now that previously didn't have access to it at all.

[rcoh]

big integer is only numbers — the larger set is only for BigDecimal

[rcoh]

you need to change expect_number to move the check for finite-ness there I think?

[rcoh]

integers have just 0..9 — probably want a tighter restriction there

[AmitKulkarni23]

Agreed. Miss on my part. Changing this in next revision/commit.

[rcoh]

can you really proceed a number with +?

[rcoh]

did this change to infinity? I thought we chose nan?

[rcoh]

ah, I see. the behavior of the parser is Infinity, that's fine.

[aajtodd]

Is +/- infinity a valid value for Smithy bigInteger or bigDecimal shapes ? (e.g. infinity or -infinity)?

[github-actions]

A new generated diff is ready to view.

• AWS SDK (ignoring whitespace)
• Client Test (ignoring whitespace)
• Server Test (ignoring whitespace)
• Server Test Python (ignoring whitespace)
• Server Test Typescript (ignoring whitespace)

A new doc preview is ready to view.