After upgrading from 3.2.0 to 3.2.1 attributes with default values get reset to None after a refresh

[canoon]

-> % pip install pynamodb==3.2.0
Collecting pynamodb==3.2.0
  Downloading pynamodb-3.2.0-py2.py3-none-any.whl (102kB)
    100% |████████████████████████████████| 102kB 5.0MB/s
Requirement already satisfied: botocore>=1.2.0 in ./ve/lib/python3.6/site-packages (from pynamodb==3.2.0)
Requirement already satisfied: six in ./ve/lib/python3.6/site-packages (from pynamodb==3.2.0)
Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in ./ve/lib/python3.6/site-packages (from pynamodb==3.2.0)
Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in ./ve/lib/python3.6/site-packages (from botocore>=1.2.0->pynamodb==3.2.0)
Requirement already satisfied: docutils>=0.10 in ./ve/lib/python3.6/site-packages (from botocore>=1.2.0->pynamodb==3.2.0)
Installing collected packages: pynamodb
  Found existing installation: pynamodb 3.1.0
    Uninstalling pynamodb-3.1.0:
      Successfully uninstalled pynamodb-3.1.0
Successfully installed pynamodb-3.2.0
-> % python3 test.py
temp_value:  test
refreshing
temp_value:  test
refreshing
-> % pip install pynamodb==3.2.1
Collecting pynamodb==3.2.1
  Using cached pynamodb-3.2.1-py2.py3-none-any.whl
Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in ./ve/lib/python3.6/site-packages (from pynamodb==3.2.1)
Requirement already satisfied: six in ./ve/lib/python3.6/site-packages (from pynamodb==3.2.1)
Requirement already satisfied: botocore>=1.2.0 in ./ve/lib/python3.6/site-packages (from pynamodb==3.2.1)
Requirement already satisfied: docutils>=0.10 in ./ve/lib/python3.6/site-packages (from botocore>=1.2.0->pynamodb==3.2.1)
Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in ./ve/lib/python3.6/site-packages (from botocore>=1.2.0->pynamodb==3.2.1)
Installing collected packages: pynamodb
  Found existing installation: pynamodb 3.2.0
    Uninstalling pynamodb-3.2.0:
      Successfully uninstalled pynamodb-3.2.0
Successfully installed pynamodb-3.2.1
-> % python3 test.py
temp_value:  test
refreshing
temp_value:  None
refreshing
Traceback (most recent call last):
  File "test.py", line 35, in <module>
    t.refresh()
  File "./ve/lib/python3.6/site-packages/pynamodb/models.py", line 446, in refresh
    args, kwargs = self._get_save_args(attributes=False)
  File "./ve/lib/python3.6/site-packages/pynamodb/models.py", line 1159, in _get_save_args
    serialized = self._serialize(null_check=null_check)
  File "./ve/lib/python3.6/site-packages/pynamodb/models.py", line 1283, in _serialize
    serialized = self._serialize_value(attr, value, null_check)
  File "./ve/lib/python3.6/site-packages/pynamodb/models.py", line 1315, in _serialize_value
    raise ValueError("Attribute '{0}' cannot be None".format(attr.attr_name))
ValueError: Attribute 'temp_value' cannot be None
-> % cat test.py
import pynamodb.models
import pynamodb.attributes
import uuid


class Temp(pynamodb.models.Model):
    class Meta:
        table_name = 'temp'
        region = 'us-east-1'

    temp_key = pynamodb.attributes.UnicodeAttribute(hash_key=True)


class Temp2(pynamodb.models.Model):
    class Meta:
        table_name = 'temp'
        region = 'us-east-1'

    temp_key = pynamodb.attributes.UnicodeAttribute(hash_key=True)
    temp_value = pynamodb.attributes.UnicodeAttribute(default="test")


key = str(uuid.uuid4())

# save initial schema
t1 = Temp(key)
t1.save()

t = Temp2.get(key)
print('temp_value: ', t.temp_value)
print("refreshing")
t.refresh()
print('temp_value: ', t.temp_value)
print("refreshing")
t.refresh()

It's probably as a result of this PR #388

[ikonst]

cc @jpinner-lyft any idea what went wrong w.r.t default and refresh?

[jpinner-lyft]

The default values only get set during AttributeContainer initialization, not during refresh. I'm not sure if this is a bug or expected behavior. The linked PR fixes "refresh" to set all attribute values to whatever is in the database (as opposed to ignoring unknown attribute values). I would maybe ask @garrettheel what he thinks the behavior of defaults should be after a refresh where the attribute isn't in the database.

[jpinner-lyft]

If this is decided to be a bug, the "fix" would be to add the replace _deserialize() in refresh() with:

self.attribute_values = {}
self._set_defaults()
self._deserialize(item_data)

[garrettheel]

@jpinner-lyft given the inability to store null in the database, I'd agree that this is indeed unexpected behavior and warrants a fix like the one you propose

[samopus1io]

Will this ever be addressed?

We rely on adding new BoolAttributes with default=True It has always worked (getting True when no value was explicitly set), but we were surprised to hit a production issue where using refresh() would reset the value to None and then throwing an exception on subsequent save because None was not allowed.

Note we just migrated from 3.3.3 to 4.3.3 (not sure if its relevant here)