Skip to content

Commit c3a718a

Browse files
authored
[venice-common] Fix missing SchemaAdapter.adaptToSchema() in (linkedin#2640)
Fix schema adaptation in setElementsToRemoveFromListField setElementsToRemoveFromListField in UpdateBuilderImpl does not call SchemaAdapter.adaptToSchema() on elementsToRemove, unlike the symmetric setElementsToAddToListField method which does. This causes Avro serialization failures when input records have extra fields (e.g., from an evolved schema) that do not match the Venice WriteOp schema. The root cause exception is: java.lang.ClassCastException: class java.lang.Long cannot be cast to class java.util.List during serialization, because unadapted records with mismatched schemas are placed into the setDiff field. Add the missing SchemaAdapter.adaptToSchema() call in setElementsToRemoveFromListField, making it consistent with setElementsToAddToListField, setEntriesToAddToMapField, and setNewFieldValue.
1 parent 129c3a9 commit c3a718a

2 files changed

Lines changed: 24 additions & 1 deletion

File tree

internal/venice-common/src/main/java/com/linkedin/venice/writer/update/UpdateBuilderImpl.java

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -72,7 +72,9 @@ public UpdateBuilder setElementsToRemoveFromListField(String listFieldName, List
7272
validateFieldType(Validate.notNull(listFieldName), Schema.Type.ARRAY);
7373
validateFieldNotSet(listFieldName);
7474
if (!elementsToRemove.isEmpty()) {
75-
getOrCreateListMergeRecord(listFieldName).put(WriteComputeConstants.SET_DIFF, elementsToRemove);
75+
getOrCreateListMergeRecord(listFieldName).put(
76+
WriteComputeConstants.SET_DIFF,
77+
SchemaAdapter.adaptToSchema(getCorrespondingValueFieldSchema(listFieldName), elementsToRemove));
7678
collectionMergeFieldNameSet.add(listFieldName);
7779
}
7880
return this;

internal/venice-common/src/test/java/com/linkedin/venice/writer/update/UpdateBuilderImplTest.java

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -116,6 +116,27 @@ public void testUpdateEvolvedSubfieldRemoveFieldsNotInUpdateSchema() {
116116
Assert.assertEquals(recordMapMergeRecord.get("mapDiff"), Collections.emptyList());
117117
}
118118

119+
@Test
120+
public void testUpdateEvolvedSubfieldRemoveFromListFieldAdaptsSchema() {
121+
UpdateBuilder builder = new UpdateBuilderImpl(UPDATE_SCHEMA);
122+
123+
List<GenericRecord> writeRecordArrayToRemove = new ArrayList<>();
124+
writeRecordArrayToRemove.add(createEvolvedRecordForListField(1, "testName"));
125+
126+
List<GenericRecord> expectedRecordArrayToRemove = new ArrayList<>();
127+
expectedRecordArrayToRemove.add(createRecordForListField(1));
128+
129+
builder.setNewFieldValue("name", "Lebron James");
130+
builder.setElementsToRemoveFromListField("recordArray", writeRecordArrayToRemove);
131+
132+
GenericRecord updateRecord = builder.build();
133+
134+
Assert.assertTrue(updateRecord.get("recordArray") instanceof GenericRecord);
135+
GenericRecord listMergeRecord = (GenericRecord) updateRecord.get("recordArray");
136+
Assert.assertEquals(listMergeRecord.get("setUnion"), Collections.emptyList());
137+
Assert.assertEquals(listMergeRecord.get("setDiff"), expectedRecordArrayToRemove);
138+
}
139+
119140
@Test
120141
public void testUpdateEvolvedSubfieldFillDefaultsForUnspecifiedFields() {
121142
UpdateBuilder builder = new UpdateBuilderImpl(EVOLVED_UPDATE_SCHEMA);

0 commit comments

Comments
 (0)