[BUG: Breaking] Marker is 20x+ slower since v1.9.0+ in Mac

[axsaucedo]

🧨 Describe the Bug

Marker is at least 20x slower in Mac since since version v1.9.0.

If using Marker on a mac install pip install marker==1.8.0 as the fix.

📄 Input Document

This works with any PDF when run in Mac with M1+ chip.

📤 Output Trace / Stack Trace

It may be due to this warning (but may not be the case):

2025-12-13 15:17:09,820 [WARNING] surya: `TableRecEncoderDecoderModel` is not compatible with mps backend. Defaulting to cpu instead

⚙️ Environment

Please fill in all relevant details:

• Marker version: Any version 1.9.0+
• Operating System: Mac

✅ Expected Behavior

What did you expect Marker to do?

Not be 20x slower in newer versions.

📟 Command or Code Used

Any command works. Here's an example:

marker --output_dir ./output/ ./

[mntolia]

Can confirm its very slow on my mac m4 pro.
Had to downgrade it to make it performant

[uts58]

Same for me, I thought it was normal

[axn236]

Found the root cause. In v0.16.0, the devs added a block in surya/table_rec/loader.py that forces TableRecEncoderDecoderModel to fall back to CPU when MPS is detected - presumably to stop an underlying incompatibility. The actual issue is that PyTorch's MPS backend silently produces incorrect results from scaled_dot_product_attention when operating on non-contiguous tensors during batch processing.

The fix is to add .contiguous() calls before SDPA operations and use float32 on MPS (same speed as bfloat16, but more reliable). Tested on M4 MacBook Pro (16GB), PyTorch 2.9.1, Surya 0.17.0, Python 3.13. Full MPS support with batching restored.

Seven files require modification:

surya/common/adetr/decoder.py (Table Recognition)

The ADETR decoder lacks the MPS-compatible autocast handling present in the Qwen2 decoder. In SuryaADETRDecoderRotaryEmbedding.forward() (~line 78):

device_type = x.device.type
device_type = (
    device_type
    if isinstance(device_type, str) and device_type != "mps"
    else "cpu"
)
with torch.autocast(device_type=device_type, enabled=False):
    # existing freqs calculation...

Add .contiguous() before both SDPA calls in SuryaADETRDecoderSdpaCrossAttention.forward() and SuryaADETRDecoderSdpaAttention.forward():

query_states = query_states.contiguous()
key_states = key_states.contiguous()
value_states = value_states.contiguous()
if causal_mask is not None:
    causal_mask = causal_mask.contiguous()

Fix cache functions (_update_static_cache, _update_dynamic_cache) to return contiguous tensors:

return k_out.contiguous(), v_out.contiguous()

surya/table_rec/loader.py

Replace the MPS block with float32 enforcement:

if device == "mps":
    dtype = torch.float32

surya/table_rec/__init__.py

Re-enable MPS batching (set to 8 to match CPU), add .contiguous() after encoder output and torch.stack:

encoder_hidden_states = self.model.encoder(pixel_values=batch_pixel_values).last_hidden_state
encoder_hidden_states = encoder_hidden_states.contiguous()

row_encoder_hidden_states = torch.stack(row_encoder_hidden_states).contiguous()

surya/common/surya/encoder/__init__.py (Foundation Model - Layout, OCR)

Both Qwen2_5_VLVisionXLASdpaAttention.forward() and Qwen2_5_VLVisionSdpaAttention.forward() have q, k, v from reshape().permute().unbind() then transpose() - non-contiguous. Add .contiguous() after each transpose before SDPA:

q = q.transpose(1, 2).contiguous()
k = k.transpose(1, 2).contiguous()
v = v.transpose(1, 2).contiguous()

surya/foundation/loader.py (Foundation Model loader)

Use float32 on MPS (same speed as bfloat16, more reliable):

if device == "mps":
    dtype = torch.float32

surya/common/donut/encoder.py (Detection)

transpose_for_scores returns non-contiguous tensor. Add .contiguous():

def transpose_for_scores(self, x):
    new_x_shape = x.size()[:-1] + (
        self.num_attention_heads,
        self.attention_head_size,
    )
    x = x.view(new_x_shape)
    return x.permute(0, 2, 1, 3).contiguous()

Note: transpose_kv_for_scores already has .contiguous().

surya/detection/loader.py (Detection Model loader)

Use float32 on MPS:

if device == "mps":
    dtype = torch.float32

Tensors from transpose(), view(), permute(), cache slicing, torch.cat(), and torch.stack() are not guaranteed contiguous. MPS SDPA fails silently on these. After these changes, MPS batch output matches CPU exactly.

[sp-cega]

Found the root cause. In v0.16.0, the devs added a block in surya/table_rec/loader.py that forces TableRecEncoderDecoderModel to fall back to CPU when MPS is detected - presumably to stop an underlying incompatibility. The actual issue is that PyTorch's MPS backend silently produces incorrect results from scaled_dot_product_attention when operating on non-contiguous tensors during batch processing.

The fix is to add .contiguous() calls before SDPA operations and use float32 on MPS (same speed as bfloat16, but more reliable). Tested on M4 MacBook Pro (16GB), PyTorch 2.9.1, Surya 0.17.0, Python 3.13. Full MPS support with batching restored.

Seven files require modification:

surya/common/adetr/decoder.py (Table Recognition)

The ADETR decoder lacks the MPS-compatible autocast handling present in the Qwen2 decoder. In SuryaADETRDecoderRotaryEmbedding.forward() (~line 78):

device_type = x.device.type
device_type = (
device_type
if isinstance(device_type, str) and device_type != "mps"
else "cpu"
)
with torch.autocast(device_type=device_type, enabled=False):
# existing freqs calculation...
Add .contiguous() before both SDPA calls in SuryaADETRDecoderSdpaCrossAttention.forward() and SuryaADETRDecoderSdpaAttention.forward():

query_states = query_states.contiguous()
key_states = key_states.contiguous()
value_states = value_states.contiguous()
if causal_mask is not None:
causal_mask = causal_mask.contiguous()
Fix cache functions (_update_static_cache, _update_dynamic_cache) to return contiguous tensors:

return k_out.contiguous(), v_out.contiguous()
surya/table_rec/loader.py

Replace the MPS block with float32 enforcement:

if device == "mps":
dtype = torch.float32
surya/table_rec/__init__.py

Re-enable MPS batching (set to 8 to match CPU), add .contiguous() after encoder output and torch.stack:

encoder_hidden_states = self.model.encoder(pixel_values=batch_pixel_values).last_hidden_state
encoder_hidden_states = encoder_hidden_states.contiguous()
row_encoder_hidden_states = torch.stack(row_encoder_hidden_states).contiguous()
surya/common/surya/encoder/__init__.py (Foundation Model - Layout, OCR)

Both Qwen2_5_VLVisionXLASdpaAttention.forward() and Qwen2_5_VLVisionSdpaAttention.forward() have q, k, v from reshape().permute().unbind() then transpose() - non-contiguous. Add .contiguous() after each transpose before SDPA:

q = q.transpose(1, 2).contiguous()
k = k.transpose(1, 2).contiguous()
v = v.transpose(1, 2).contiguous()
surya/foundation/loader.py (Foundation Model loader)

Use float32 on MPS (same speed as bfloat16, more reliable):

if device == "mps":
dtype = torch.float32
surya/common/donut/encoder.py (Detection)

transpose_for_scores returns non-contiguous tensor. Add .contiguous():

def transpose_for_scores(self, x):
new_x_shape = x.size()[:-1] + (
self.num_attention_heads,
self.attention_head_size,
)
x = x.view(new_x_shape)
return x.permute(0, 2, 1, 3).contiguous()
Note: transpose_kv_for_scores already has .contiguous().

surya/detection/loader.py (Detection Model loader)

Use float32 on MPS:

if device == "mps":
dtype = torch.float32
Tensors from transpose(), view(), permute(), cache slicing, torch.cat(), and torch.stack() are not guaranteed contiguous. MPS SDPA fails silently on these. After these changes, MPS batch output matches CPU exactly.

I implemented these changes and the warning indeed went away (thanks!). The runtime did not change though, and Marker is still extremely slow (relative to previous versions). The results are better, but not by much.

[papatistos]

@sp-cega when you say

The runtime did not change though, and Marker is still extremely slow (relative to previous versions).

can you clarify whether it is slow even though it now uses the GPU or because it still uses the CPU?