Using `transformers==4.49.0` leads to incorrect results

[oisinturbitt]

Hi There,

Thank you very much for the paper and the models!

Just want to flag an issue I have found when using this model with the latest versions of the transformers library (4.49.0). Using it results in a performance degradation versus when using 4.48.*.

[AbrahamSanders]

I noticed that when transformers >= 4.49.0 I get the following warning on loading the codec, i.e. some weights are not initialized from the checkpoint. This would explain the performance degradation.

Some weights of the model checkpoint at HKUSTAudio/xcodec2 were not used when initializing XCodec2Model: ['CodecEnc.conv_blocks.1.block.0.block.0.act.beta', 'CodecEnc.conv_blocks.1.block.0.block.2.act.beta', 'CodecEnc.conv_blocks.1.block.1.block.0.act.beta', 'CodecEnc.conv_blocks.1.block.1.block.2.act.beta', 'CodecEnc.conv_blocks.1.block.2.block.0.act.beta', 'CodecEnc.conv_blocks.1.block.2.block.2.act.beta', 'CodecEnc.conv_blocks.1.block.3.act.beta', 'CodecEnc.conv_blocks.2.block.0.block.0.act.beta', 'CodecEnc.conv_blocks.2.block.0.block.2.act.beta', 'CodecEnc.conv_blocks.2.block.1.block.0.act.beta', 'CodecEnc.conv_blocks.2.block.1.block.2.act.beta', 'CodecEnc.conv_blocks.2.block.2.block.0.act.beta', 'CodecEnc.conv_blocks.2.block.2.block.2.act.beta', 'CodecEnc.conv_blocks.2.block.3.act.beta', 'CodecEnc.conv_blocks.3.block.0.block.0.act.beta', 'CodecEnc.conv_blocks.3.block.0.block.2.act.beta', 'CodecEnc.conv_blocks.3.block.1.block.0.act.beta', 'CodecEnc.conv_blocks.3.block.1.block.2.act.beta', 'CodecEnc.conv_blocks.3.block.2.block.0.act.beta', 'CodecEnc.conv_blocks.3.block.2.block.2.act.beta', 'CodecEnc.conv_blocks.3.block.3.act.beta', 'CodecEnc.conv_blocks.4.block.0.block.0.act.beta', 'CodecEnc.conv_blocks.4.block.0.block.2.act.beta', 'CodecEnc.conv_blocks.4.block.1.block.0.act.beta', 'CodecEnc.conv_blocks.4.block.1.block.2.act.beta', 'CodecEnc.conv_blocks.4.block.2.block.0.act.beta', 'CodecEnc.conv_blocks.4.block.2.block.2.act.beta', 'CodecEnc.conv_blocks.4.block.3.act.beta', 'CodecEnc.conv_blocks.5.block.0.block.0.act.beta', 'CodecEnc.conv_blocks.5.block.0.block.2.act.beta', 'CodecEnc.conv_blocks.5.block.1.block.0.act.beta', 'CodecEnc.conv_blocks.5.block.1.block.2.act.beta', 'CodecEnc.conv_blocks.5.block.2.block.0.act.beta', 'CodecEnc.conv_blocks.5.block.2.block.2.act.beta', 'CodecEnc.conv_blocks.5.block.3.act.beta', 'CodecEnc.conv_final_block.0.act.beta']
- This IS expected if you are initializing XCodec2Model from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing XCodec2Model from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of XCodec2Model were not initialized from the model checkpoint at HKUSTAudio/xcodec2 and are newly initialized: ['CodecEnc.conv_blocks.1.block.0.block.0.act.bias', 'CodecEnc.conv_blocks.1.block.0.block.2.act.bias', 'CodecEnc.conv_blocks.1.block.1.block.0.act.bias', 'CodecEnc.conv_blocks.1.block.1.block.2.act.bias', 'CodecEnc.conv_blocks.1.block.2.block.0.act.bias', 'CodecEnc.conv_blocks.1.block.2.block.2.act.bias', 'CodecEnc.conv_blocks.1.block.3.act.bias', 'CodecEnc.conv_blocks.2.block.0.block.0.act.bias', 'CodecEnc.conv_blocks.2.block.0.block.2.act.bias', 'CodecEnc.conv_blocks.2.block.1.block.0.act.bias', 'CodecEnc.conv_blocks.2.block.1.block.2.act.bias', 'CodecEnc.conv_blocks.2.block.2.block.0.act.bias', 'CodecEnc.conv_blocks.2.block.2.block.2.act.bias', 'CodecEnc.conv_blocks.2.block.3.act.bias', 'CodecEnc.conv_blocks.3.block.0.block.0.act.bias', 'CodecEnc.conv_blocks.3.block.0.block.2.act.bias', 'CodecEnc.conv_blocks.3.block.1.block.0.act.bias', 'CodecEnc.conv_blocks.3.block.1.block.2.act.bias', 'CodecEnc.conv_blocks.3.block.2.block.0.act.bias', 'CodecEnc.conv_blocks.3.block.2.block.2.act.bias', 'CodecEnc.conv_blocks.3.block.3.act.bias', 'CodecEnc.conv_blocks.4.block.0.block.0.act.bias', 'CodecEnc.conv_blocks.4.block.0.block.2.act.bias', 'CodecEnc.conv_blocks.4.block.1.block.0.act.bias', 'CodecEnc.conv_blocks.4.block.1.block.2.act.bias', 'CodecEnc.conv_blocks.4.block.2.block.0.act.bias', 'CodecEnc.conv_blocks.4.block.2.block.2.act.bias', 'CodecEnc.conv_blocks.4.block.3.act.bias', 'CodecEnc.conv_blocks.5.block.0.block.0.act.bias', 'CodecEnc.conv_blocks.5.block.0.block.2.act.bias', 'CodecEnc.conv_blocks.5.block.1.block.0.act.bias', 'CodecEnc.conv_blocks.5.block.1.block.2.act.bias', 'CodecEnc.conv_blocks.5.block.2.block.0.act.bias', 'CodecEnc.conv_blocks.5.block.2.block.2.act.bias', 'CodecEnc.conv_blocks.5.block.3.act.bias', 'CodecEnc.conv_final_block.0.act.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.

[apoorvcodes]

It was a standard issue. I tried debugging the codec because it wasn't producing any codes, then switched to version 4.48 of Transformers and now it works perfectly.

[JunZhan2000]

Hello, may I ask which aspect of performance has declined? I'm using 4.50.2 and haven't noticed any significant issues.

[AbrahamSanders]

Hello, may I ask which aspect of performance has declined? I'm using 4.50.2 and haven't noticed any significant issues.

The issue seems to affect encoding only. Decoding works fine. It seems the weights for the encoder portion of the model are not properly initialized from the checkpoint on transformers >= 4.49.0.

[JunZhan2000]

Hello, may I ask which aspect of performance has declined? I'm using 4.50.2 and haven't noticed any significant issues.

The issue seems to affect encoding only. Decoding works fine. It seems the weights for the encoder portion of the model are not properly initialized from the checkpoint on transformers >= 4.49.0.

Thank you for your reply. I'd like to ask what effects you've observed on the encoding? Has the performance degraded, or is it unable to generate tokens?

[zhenye234]

Maybe you can try the original ckpts without pip install ( I mean without import xcodec2), https://github.com/zhenye234/X-Codec-2.0#setup https://huggingface.co/HKUSTAudio/xcodec2/blob/main/ckpt/epoch%3D4-step%3D1400000.ckpt

[AbrahamSanders]

Maybe you can try the original ckpts without pip install ( I mean without import xcodec2), https://github.com/zhenye234/X-Codec-2.0#setup https://huggingface.co/HKUSTAudio/xcodec2/blob/main/ckpt/epoch%3D4-step%3D1400000.ckpt

Thank you for your reply! Using the repo directly wouldn't work for me due to the version pins in requirements.txt. However, with a bit of investigation I was able to isolate the issue and come up with a workaround:

The issue

The version of vq/activations.py distributed with the pip install package has the following code:

  # initialize alpha
  self.alpha_logscale = alpha_logscale
  if self.alpha_logscale: # log scale alphas initialized to zeros
      self.alpha = Parameter(torch.zeros(in_features) * alpha)
      self.bias = Parameter(torch.zeros(in_features) * alpha)
  else: # linear scale alphas initialized to ones
      self.alpha = Parameter(torch.ones(in_features) * alpha)
      self.bias = Parameter(torch.ones(in_features) * alpha)

however in both the repo and the checkpoint, self.bias is called self.beta. This worked prior to transformers 4.49.0 because there was code that replaced "beta" with "bias" in the parameter names on state dict load. This logic was changed in 4.49.0 under the assumption that this change only needed to be done for LayerNorm parameters -
In src/transformers/modeling_utils.py in the method def _fix_state_dict_key_on_load(key):

        if "beta" in key:
            return key.replace("beta", "bias")

was changed to:

        if key.endswith("LayerNorm.beta"):
            return key.replace("LayerNorm.beta", "LayerNorm.bias"), True

The full diff can be found in this PR.

The workaround

So, the workaround for the pip install package is simple:

Instead of loading the model with

from xcodec2.modeling_xcodec2 import XCodec2Model

codec_model = XCodec2Model.from_pretrained("HKUSTAudio/xcodec2")

we can load it this way, which works with the latest transformers version:

from xcodec2.modeling_xcodec2 import XCodec2Model
from xcodec2.configuration_bigcodec import BigCodecConfig
from safetensors import safe_open
from huggingface_hub import hf_hub_download

model_name = "HKUSTAudio/xcodec2"
ckpt_path = hf_hub_download(repo_id=model_name, filename="model.safetensors")
ckpt = {}
with safe_open(ckpt_path, framework="pt", device="cpu") as f:
    for k in f.keys():
        ckpt[k.replace(".beta", ".bias")] = f.get_tensor(k)
codec_config = BigCodecConfig.from_pretrained(model_name)
codec_model = XCodec2Model.from_pretrained(None, config=codec_config, state_dict=ckpt)

@zhenye234 If you have a chance, would you be able to update the model checkpoint on the HuggingFace hub to use bias instead of beta?

[AbrahamSanders]

Hello, may I ask which aspect of performance has declined? I'm using 4.50.2 and haven't noticed any significant issues.

The issue seems to affect encoding only. Decoding works fine. It seems the weights for the encoder portion of the model are not properly initialized from the checkpoint on transformers >= 4.49.0.

Thank you for your reply. I'd like to ask what effects you've observed on the encoding? Has the performance degraded, or is it unable to generate tokens?

Encoding an audio and decoding it would result in just noise. Decoding codes previously saved using transformers==4.48.3 works fine, indicating the issue is only with the encoder.