I don't think this behavior is correct. SparkFixupTypes should only fix the type when there is an explicit fix. But SparkFixupTypes doesn't have one so all types are converted to variant. I would expect this to only allow binary, which is probably what we will use to pass this value into Spark 3.5.

Could you please further explain this? Do you mean that we need to restrict VariantType to only convert with BinaryType in SparkFixupTypes? Or are you just suggesting that the name of my test should not be "Correct"?

The cases that are fixed up are in SparkFixupTypes:

  @Override
  protected boolean fixupPrimitive(Type.PrimitiveType type, Type source) {
    switch (type.typeId()) {
      case STRING:
        if (source.typeId() == Type.TypeID.UUID) {
          return true;
        }
        break;
      case BINARY:
        if (source.typeId() == Type.TypeID.FIXED) {
          return true;
        }
        break;
      case TIMESTAMP:
        if (source.typeId() == Type.TypeID.TIMESTAMP) {
          return true;
        }
        break;
      default:
    }
    return false;
  }

Currently, the behavior is to always use the type from schema and never override with the reference schema's type. The test here validates that the type is not fixed, but what we care about are cases where a type does get fixed.

I think the case where a type does get fixed is when we are exposing variant as binary. So I would expect an update to SparkFixupTypes and the base FixupTypes to override binary with variant when variant is in the reference schema. Otherwise, the variant type is never "fixed" and we don't really need a new test suite.