Why is the decoder non-causal

[gk-mixar]

codeline
According to this code line the decoder is non-causal that implies its bidiretional attention which implies its seeing the future tokens to generate the current token which is not typical in decoders.
Can you clarify this issue ??

[splionar]

@gk-mixar If I am not mistaken, the causal handle on that script does not have any effect:

TreeMeshGPT/model/custom_transformers_training.py

Lines 1260 to 1269 in 3582158

	LATENT_LEN = 2048
	def custom(b, h, q_idx, kv_idx):
	mask_latent = kv_idx < LATENT_LEN
	mask_causal = q_idx >= kv_idx
	return mask_causal | mask_latent
	block_mask = create_block_mask(custom, B=None, H=None, Q_LEN=q.shape[-2], KV_LEN=k.shape[-2])
	out = flex_attention(q, k, v, block_mask=block_mask)

It uses FlexAttention, where the first 2048 tokens (point clound condition) is non-causal, and the subsequent tokens for mesh generation are causal.