Support channels_last format in portable upsample kernels #9526

GregoryComer · 2025-03-23T03:45:34Z

Summary:
Support channels_last input format in portable CPU upsample_bilinear2d and upsample_nearest2d kernels. This is useful for resize-in-model patterns when the user wants to pass inputs in channels_last format. It also (theoretically) allows for more effective auto-vectorization when vectorizing along the channels dim when there are a larger number of channels.

I considered generalizing the kernel to handle arbitrary dim order, but having a specialized channels last version allows for traversing the output in contiguous order. I could add a separate, arbitrarily-strided variant, but we can take that as a follow-up if needed.

To accomplish this, this PR makes the following changes:

Update check_upsample_2d_common_args to relax the dim order restriction. It now allows for both default and channels_last dim order and verifies that the output dim order matches the input.
In the upsample kernels (bilinear and nearest), split out NCHW and NHWC variants. The NHWC variant interchanges the loop order as to maintain contiguous output accesses.
Add test coverage to ensure ATen numerical parity.

Differential Revision: D71690379

cc @larryliu0820 @manuelcandales

pytorch-bot · 2025-03-23T03:45:38Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/9526

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 2 New Failures

As of commit 627f274 with merge base 012f120 ():

NEW FAILURES - The following jobs have failed:

pull / unittest / macos / macos-job (gh)
backends/xnnpack/test/ops/test_check_quant_params.py::TestCheckQuantParams::test_inject_invalid_zp
pull / unittest-editable / macos / macos-job (gh)
backends/xnnpack/test/passes/test_convert_to_linear.py::TestConvertToLinear::test_fp32_convert_to_linear

This comment was automatically generated by Dr. CI and updates every 15 minutes.

facebook-github-bot · 2025-03-23T03:45:49Z

This pull request was exported from Phabricator. Differential Revision: D71690379

Summary: Support channels_last input format in portable CPU upsample_bilinear2d and upsample_nearest2d kernels. This is useful for resize-in-model patterns when the user wants to pass inputs in channels_last format. It also (theoretically) allows for more effective auto-vectorization when vectorizing along the channels dim when there are a larger number of channels. I considered generalizing the kernel to handle arbitrary dim order, but having a specialized channels last version allows for traversing the output in contiguous order. I could add a separate, arbitrarily-strided variant, but we can take that as a follow-up if needed. To accomplish this, this PR makes the following changes: - Update `check_upsample_2d_common_args` to relax the dim order restriction. It now allows for both default and channels_last dim order and verifies that the output dim order matches the input. - In the upsample kernels (bilinear and nearest), split out NCHW and NHWC variants. The NHWC variant interchanges the loop order as to maintain contiguous output accesses. - Add test coverage to ensure ATen numerical parity. Differential Revision: D71690379

facebook-github-bot · 2025-03-23T06:19:02Z

This pull request was exported from Phabricator. Differential Revision: D71690379

Summary: Support channels_last input format in portable CPU upsample_bilinear2d and upsample_nearest2d kernels. This is useful for resize-in-model patterns when the user wants to pass inputs in channels_last format. It also (theoretically) allows for more effective auto-vectorization when vectorizing along the channels dim when there are a larger number of channels. I considered generalizing the kernel to handle arbitrary dim order, but having a specialized channels last version allows for traversing the output in contiguous order. I could add a separate, arbitrarily-strided variant, but we can take that as a follow-up if needed. To accomplish this, this PR makes the following changes: - Update `check_upsample_2d_common_args` to relax the dim order restriction. It now allows for both default and channels_last dim order and verifies that the output dim order matches the input. - In the upsample kernels (bilinear and nearest), split out NCHW and NHWC variants. The NHWC variant interchanges the loop order as to maintain contiguous output accesses. - Add test coverage to ensure ATen numerical parity. Differential Revision: D71690379

swolchok · 2025-03-24T16:33:11Z

never worked with channels_last so probably not the right person to review this. ping me if I need to just learn it for this review

Summary: Support channels_last input format in portable CPU upsample_bilinear2d and upsample_nearest2d kernels. This is useful for resize-in-model patterns when the user wants to pass inputs in channels_last format. It also (theoretically) allows for more effective auto-vectorization when vectorizing along the channels dim when there are a larger number of channels. I considered generalizing the kernel to handle arbitrary dim order, but having a specialized channels last version allows for traversing the output in contiguous order. I could add a separate, arbitrarily-strided variant, but we can take that as a follow-up if needed. To accomplish this, this PR makes the following changes: - Update `check_upsample_2d_common_args` to relax the dim order restriction. It now allows for both default and channels_last dim order and verifies that the output dim order matches the input. - In the upsample kernels (bilinear and nearest), split out NCHW and NHWC variants. The NHWC variant interchanges the loop order as to maintain contiguous output accesses. - Add test coverage to ensure ATen numerical parity. Differential Revision: D71690379

facebook-github-bot · 2025-03-25T12:25:28Z

This pull request was exported from Phabricator. Differential Revision: D71690379

Summary: Pull Request resolved: pytorch#9526 Support channels_last input format in portable CPU upsample_bilinear2d and upsample_nearest2d kernels. This is useful for resize-in-model patterns when the user wants to pass inputs in channels_last format. It also (theoretically) allows for more effective auto-vectorization when vectorizing along the channels dim when there are a larger number of channels. I considered generalizing the kernel to handle arbitrary dim order, but having a specialized channels last version allows for traversing the output in contiguous order. I could add a separate, arbitrarily-strided variant, but we can take that as a follow-up if needed. To accomplish this, this PR makes the following changes: - Update `check_upsample_2d_common_args` to relax the dim order restriction. It now allows for both default and channels_last dim order and verifies that the output dim order matches the input. - In the upsample kernels (bilinear and nearest), split out NCHW and NHWC variants. The NHWC variant interchanges the loop order as to maintain contiguous output accesses. - Add test coverage to ensure ATen numerical parity. Differential Revision: D71690379

mcr229 · 2025-03-25T20:32:53Z

does this have an affect on binary size?

GregoryComer · 2025-03-25T20:33:54Z

does this have an affect on binary size?

Good question. It should be pretty small, but I can run a BSB check internally to verify.

manuelcandales · 2025-03-25T21:04:33Z

kernels/portable/cpu/op_upsample_nearest2d.cpp

@@ -79,11 +104,25 @@ Tensor& upsample_nearest2d_vec_out(
  const auto kernel_scale_w = area_pixel_compute_scale<double>(
      in.sizes()[3], out.sizes()[3], false, scale_w);

+  if (tensor_is_default_dim_order(in)) {


I don't think there is a need to add this if/else here.
I think you should move this if/else login inside a new upsample_nearest2d_kernel_impl.

@manuelcandales Sure. Just to double check, is there a better way to dispatch on dim order other than explicit if checks? It looks like, in theory, there is selective build integration. Though I'm not clear on if that's actually implemented.

Not as far as I know. I don't think we have that machinery developed. As a matter of fact, I believe this is the first portable op branching out on dim order.
The few ops that currently support channel last in the portable library, are not dispatching specific code for that. In those cases the code is written in a dim order agnostic way, maybe only requiring that all inputs have the same dim order

Summary: Pull Request resolved: pytorch#9526 Support channels_last input format in portable CPU upsample_bilinear2d and upsample_nearest2d kernels. This is useful for resize-in-model patterns when the user wants to pass inputs in channels_last format. It also (theoretically) allows for more effective auto-vectorization when vectorizing along the channels dim when there are a larger number of channels. I considered generalizing the kernel to handle arbitrary dim order, but having a specialized channels last version allows for traversing the output in contiguous order. I could add a separate, arbitrarily-strided variant, but we can take that as a follow-up if needed. To accomplish this, this PR makes the following changes: - Update `check_upsample_2d_common_args` to relax the dim order restriction. It now allows for both default and channels_last dim order and verifies that the output dim order matches the input. - In the upsample kernels (bilinear and nearest), split out NCHW and NHWC variants. The NHWC variant interchanges the loop order as to maintain contiguous output accesses. - Add test coverage to ensure ATen numerical parity. Differential Revision: D71690379

manuelcandales · 2025-03-25T21:08:07Z

kernels/portable/cpu/op_upsample_bilinear2d.cpp

-        upsample_bilinear2d_kernel_impl<CTYPE>(
-            in, align_corners, kernel_scale_h, kernel_scale_w, out);
-      });
+  if (executorch::runtime::tensor_is_default_dim_order(in)) {


see my comment above about not doing the if/else here. It is duplicating the dispatch code.

GregoryComer requested review from JacobSzwejbka, lucylq, swolchok and manuelcandales as code owners March 23, 2025 03:45

facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Mar 23, 2025

facebook-github-bot added the fb-exported label Mar 23, 2025

GregoryComer added module: kernels Issues related to kernel libraries and utilities, and code under kernels/ triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module labels Mar 23, 2025

GregoryComer force-pushed the export-D71690379 branch from 1bfb530 to fb62a3b Compare March 23, 2025 06:18

GregoryComer force-pushed the export-D71690379 branch from fb62a3b to e2f9e3b Compare March 23, 2025 06:38

GregoryComer force-pushed the export-D71690379 branch from e2f9e3b to 3b28a15 Compare March 23, 2025 07:57

GregoryComer added the release notes: ops & kernels Changes to the opset and any new / changed kernel implementations label Mar 24, 2025

GregoryComer force-pushed the export-D71690379 branch from 3b28a15 to c8a3356 Compare March 24, 2025 07:16

swolchok removed their request for review March 24, 2025 16:32

GregoryComer force-pushed the export-D71690379 branch from c8a3356 to 2f3bd98 Compare March 25, 2025 12:21

GregoryComer force-pushed the export-D71690379 branch from 2f3bd98 to 4e4003f Compare March 25, 2025 12:25

manuelcandales reviewed Mar 25, 2025

View reviewed changes

GregoryComer force-pushed the export-D71690379 branch from 4e4003f to 627f274 Compare March 25, 2025 21:05

manuelcandales approved these changes Mar 25, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support channels_last format in portable upsample kernels #9526

Support channels_last format in portable upsample kernels #9526

GregoryComer commented Mar 23, 2025 •

edited by pytorch-bot bot

Loading

pytorch-bot bot commented Mar 23, 2025 •

edited

Loading

facebook-github-bot commented Mar 23, 2025

facebook-github-bot commented Mar 23, 2025

swolchok commented Mar 24, 2025

facebook-github-bot commented Mar 25, 2025

mcr229 commented Mar 25, 2025

GregoryComer commented Mar 25, 2025

manuelcandales Mar 25, 2025 •

edited

Loading

GregoryComer Mar 25, 2025

manuelcandales Mar 25, 2025

manuelcandales Mar 25, 2025

Support channels_last format in portable upsample kernels #9526

Are you sure you want to change the base?

Support channels_last format in portable upsample kernels #9526

Conversation

GregoryComer commented Mar 23, 2025 • edited by pytorch-bot bot Loading

pytorch-bot bot commented Mar 23, 2025 • edited Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/9526

❌ 2 New Failures

facebook-github-bot commented Mar 23, 2025

facebook-github-bot commented Mar 23, 2025

swolchok commented Mar 24, 2025

facebook-github-bot commented Mar 25, 2025

mcr229 commented Mar 25, 2025

GregoryComer commented Mar 25, 2025

manuelcandales Mar 25, 2025 • edited Loading

Choose a reason for hiding this comment

GregoryComer Mar 25, 2025

Choose a reason for hiding this comment

manuelcandales Mar 25, 2025

Choose a reason for hiding this comment

manuelcandales Mar 25, 2025

Choose a reason for hiding this comment

GregoryComer commented Mar 23, 2025 •

edited by pytorch-bot bot

Loading

pytorch-bot bot commented Mar 23, 2025 •

edited

Loading

manuelcandales Mar 25, 2025 •

edited

Loading