fix: Fix to_pandas() on empty enum Series did not preserve enum dictionary

fix: Fix to_pandas() on empty enum Series did not preserve enum dictionary #3967