enum deserialization of missing values to null

[bolds07]

My domain class has one field as an enum.

Recently I decided to change the enum values for a better readeability. But know when trying to load users from firebase i get the exception.

Giveaway giv = ds.getValue(Giveaway.class);

Process: com.tomatedigital.instagramgiveawaywinner, PID: 21593
       com.google.firebase.database.DatabaseException: Could not find enum value of com.tomatedigital.giveawaymaster.domain.CommentListType for value "COMMON_NAMES"
           at com.google.firebase.database.core.utilities.encoding.CustomClassMapper.deserializeToEnum(com.google.firebase:firebase-database@@19.2.1:314)
           at com.google.firebase.database.core.utilities.encoding.CustomClassMapper.deserializeToClass(com.google.firebase:firebase-database@@19.2.1:229)
           at com.google.firebase.database.core.utilities.encoding.CustomClassMapper.deserializeToType(com.google.firebase:firebase-database@@19.2.1:178)
           at com.google.firebase.database.core.utilities.encoding.CustomClassMapper.access$100(com.google.firebase:firebase-database@@19.2.1:47)
           at com.google.firebase.database.core.utilities.encoding.CustomClassMapper$BeanMapper.deserialize(com.google.firebase:firebase-database@@19.2.1:592)
           at com.google.firebase.database.core.utilities.encoding.CustomClassMapper$BeanMapper.deserialize(com.google.firebase:firebase-database@@19.2.1:562)
           at com.google.firebase.database.core.utilities.encoding.CustomClassMapper.convertBean(com.google.firebase:firebase-database@@19.2.1:432)
           at com.google.firebase.database.core.utilities.encoding.CustomClassMapper.deserializeToClass(com.google.firebase:firebase-database@@19.2.1:231)
           at com.google.firebase.database.core.utilities.encoding.CustomClassMapper.convertToCustomClass(com.google.firebase:firebase-database@@19.2.1:79)
           at com.google.firebase.database.DataSnapshot.getValue(com.google.firebase:firebase-database@@19.2.1:203)
           at com.tomatedigital.giveawaymaster.firabase.LoginValueEventListener$3.onDataChange(LoginValueEventListener.java:178)
           at com.google.firebase.database.Query$1.onDataChange(com.google.firebase:firebase-database@@19.2.1:179)
           at com.google.firebase.database.core.ValueEventRegistration.fireEvent(com.google.firebase:firebase-database@@19.2.1:75)
           at com.google.firebase.database.core.view.DataEvent.fire(com.google.firebase:firebase-database@@19.2.1:63)
           at com.google.firebase.database.core.view.EventRaiser$1.run(com.google.firebase:firebase-database@@19.2.1:55)
           at android.os.Handler.handleCallback(Handler.java:883)
           at android.os.Handler.dispatchMessage(Handler.java:100)
           at android.os.Looper.loop(Looper.java:214)
           at android.app.ActivityThread.main(ActivityThread.java:7356)
           at java.lang.reflect.Method.invoke(Native Method)
           at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:492)
           at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:930)

COMMON_NAMES was changed to something else.

I completely understand the reason of this exception... my question is "are there any workaround?"...

I wish i could tell firebase to map unexisting enum values to null or any value, so then i could manually deal with them...

Is a very complicated restriction to throw a fatal exception like this on deserialization for a missing value... it means that ALL APPS RUNNING FIREBASE WONT BE ALLOWED TO REMOVE ENUM VALUES FOREVER...

[google-oss-bot]

I couldn't figure out how to label this issue, so I've labeled it for a human to triage. Hang tight.

[ryanjohn1]

I've got this same issue with Firestoe, though I think the most important problem is you can never add a new enum value as any app that hasn't updated to the latest version will throw an exception. I was hoping to add an "unknown" value to my enum and set that as the default when the deserialiser fails to match it to a known value. But I can't see a way to intervene in the deserialisation process to achieve this without catching the error and then deserialising the whole object myself manually but then there is no point in using Firestore's deserialisation for any objects with an enum field.
Ideally, I'd like a way to override a function on an enum class to define a default or be able to override the behaviour of CustomClassMapper.deserializeToEnum so I could catch the exception and provide an alternative before continuing to deserialise the rest of the object.

[dconeybe]

@bolds07 Thank you for bringing this issue to our attention. It is indeed an unfortunate situation that makes organically modifying Enum classes tricky. There are a few mechanisms to help with this sort of thing but none that completely address your situation. Here are some thoughts though:

1. Don’t rename, add, or remove enum elements, ever. - Clearly this is not ideal, and not even really a solution, but this problem could be solved by using a mechanism other than modifying an enum to achieve the desired result.

2. Deserialize to a String instead of an Enum. - By using a String field in the class rather than an Enum the enum parsing logic is bypassed completely, allowing any value. You could then write a custom method (e.g. parseXXX()) to convert the String to an Enum and apply whatever sort of custom mapping you wish.

3. Use the @PropertyName annotation. - By annotating an Enum element with @PropertyName you can use a custom String value to map to that enum element during deserialization. This does not allow for a mixture of old and new enum values in the database. You would have to stick with the old string value in the database so that, old clients will continue to work; however, it would allow you to rename the enum constant without affecting how the data is serialized.

Hopefully one of these suggestions is helpful. Please try it out and let us know.

[dconeybe]

I'm also thinking of how we could modify the Cloud Firestore and/or Realtime Database to handle organically changing enums. I have a few ideas below about which I'd love to hear your ideas. Note that I am NOT saying that this will happen or giving any estimates as to when. I'm just wanting to get your thoughts on these ideas or other ideas that we may consider for the future to solve this problem.

Idea 1 - @FallbackValue annotation in the Enum class

Firestore could provide an annotation named @FallbackValue. This annotation could be applied to zero or one of the enum's elements. If present, then that enum value will be used if the value from the database fails to map to another enum element. Namely, instead of throwing the "DatabaseException: Could not find enum value of..." exception the enum value with this annotation would be used.

Example:

public enum Color {
  @FallbackValue
  unspecified,
  blue,
  red,
  yellow,
}

public static class Pojo {
  public Color color;
}

In this example, if deserializing {"color": "turquoise"} it will produce an instance of Pojo with color set to Color.unspecified. This is because "turquoise" does not map to any of the enum elements and unspecified is annotated with @FallbackValue.

Idea 2 - @FallbackValue annotation in the "POJO" class.

This is similar to Idea 1 above, except that instead of annotating a member of the Enum class, the field/method in the POJO is annotated. This would require declaring a custom @interface annotation for each Enum that desires this behavior; however, it would also allow for different fallback values to be used in different members of a POJO.

Example:

@Retention(RetentionPolicy.RUNTIME)
@Target({ElementType.METHOD, ElementType.FIELD})
@FallbackValue
public @interface ColorFallbackValue {
  Color value();
}

public enum Color {
  blue,
  red,
  yellow,
}

public static class Pojo {
  @ColorFallbackValue(Color.blue)
  public Color foregroundColor;
  @ColorFallbackValue(Color.red)
  public Color backgroundColor;
}

In this example, if deserializing {"foregroundColor": "purple", "backgroundColor": "beige"} it will produce an instance of Pojo with foregroundColor set to Color.blue and backgroundColor set to Color.red. This is because neither "purple" nor "beige" are the names of elements of the Color enum, and therefore the value specified in the field's @ColorFallbackValue is used.

Idea 3 - @Encode/@decode annotations in the Enum class.

Firestore could provide annotations named @Encode and @Decode. Each of these annotations could optionally annotate zero or one method in the Enum class. Then, instead of using the default algorithm to convert between String values and Enum elements, these custom methods would be called.

Example:

  public enum Color {
    undefined,
    blue,
    red,
    yellow;

    @Encode
    public static String encode(Color color) {
      return color.name();
    }

    @Decode
    public static Color decode(String encodedColor) {
      try {
        return Color.valueOf(encodedColor);
      } catch (IllegalArgumentException e) {
        return undefined; 
      }
    }

  }

  public static class Pojo {
    public Color color;
  }

In this example, if deserializing {"color": "turquoise"} it will produce an instance of Pojo with color set to Color.undefined. This is because the method of the Color class that is annotated with @Decode would return Color.undefined because the value "turquoise" is not equal to any of the names of the enum elements.

@bolds07, @ryanjohn1: Please let me know your thoughts.

[bolds07]

I'm happy that someone got to read it... i must say i'vent had success reaching firebase team in the past...

I must say i hadnt think about the option 2 you mentioned and that would probably "workaround" my specific problem but still keep the main problem there. as @ryanjohn1 said if we add any more enum values to a field in a domain class... any user with an old version of the app wont be able to deserialize it and will crash.

so it creates the need of every app have a version control mechanism that forces the users to update in case of a "domain modification" what isnt the best.

From the options you said i like best the number 2.
It is much more flexible. allows the same enum in different classes to have different behavior, also much more easy to read, since when dealing to domain objects i like to read the properties of it right next to the field (not like option 3 and 1 that the annotation would be in the enum file)

you said:
"I'm also thinking of how we could modify the Cloud Firestore and/or Realtime Database to handle organically changing enums"

If i can add some thoughts is: dont limitate to enums!

i always missed something like "setLenient(true)" to firebase... deserialize whatever you can... if you cannot deserialize something just let it be the way it is....

I know is a much more weird scenario to have a string to become int... but remember that all kinds of projects goes to firebase and with very different levels of maturity and some projects can really change on the way.

I personally NEVER load entiry collections from firebase... is very risky to try to load something like Map<String,Something>... i always do an iterative deserialization with try{}catch{} on client side... so i can either fix some invalid fields (case im aware of the problems) or just delete the node from database and apologise to the user.

if i can add my personal experience... i've never had any android courses in life, when i first heard of firebase i was looking for a free sql database 1.5 years ago... today i've 180k registered users and 7k daily sessions. all this needed to be replaned thousands of times on the run
most people start with idea and give shape to it during the run. give tools for us to be able to reshape it.

[dconeybe]

All excellent points. Thank you for your thoughtful response. I'll take this information into account and discuss with the team; however, I don't expect any work to occur on this in the short term and cannot make any promises of future work. From what I can tell from your description of the issue, the @PropertyName annotation should address your specific issue of renaming enum values. I'm going to close this ticket since I think that the issue at hand has been resolved (feel free to re-open if that is not true).

[bolds07]

Is it possible to leave this issue open?

You can label it as feature request, this way people with related problems can add thoughts to the topic and we can keep track of the updates... I know it can take time or even never be included in backlog but still this issue is a reference point to the topic...

[dconeybe]

Sure, I will do that. Good idea.