[Transformations][CPU] Introduce Convolution fusion with bias #29076

aobolensk · 2025-02-19T18:21:13Z

Details:

Move convolution with bias transformation from CPU graph optimizer to transformations

Tickets:

N/A

t-jankowski · 2025-02-28T13:51:04Z

src/core/shape_inference/include/convolution_shape_inference.hpp

@@ -51,5 +54,8 @@ std::vector<TRShape> shape_infer(const TOp* op,
    return output_shapes;
 }
 }  // namespace v1
+
+using v1::shape_infer;


Could using be avoided in .hpp file? How complex is alternative solution?
fyi @praasz

It should be removed or have some local scope only like: function, code block etc.

Adjusted, the only drawback is the change of 1 line in GPU plugin

CuriousPanCake · 2025-03-03T10:11:02Z

Are there going to be any tests added for the added functionality?

rkazants · 2025-03-06T10:20:48Z

src/common/transformations/include/ov_ops/convolution_biased.hpp

+namespace op {
+namespace internal {
+
+class TRANSFORMATIONS_API ConvolutionBiased : public ov::op::util::ConvolutionFwdPropBase {


TRANSFORMATIONS_API is usually used for exporting transformations, not operations.

Hm, this is strange. I can see TRANSFORMATIONS_API macro usage in this context across many files in src/common/transformations/include/ov_ops dir. Do they also use that in incorrect way?

I see, then we may re-do this in separate PR. Operation is not a transformation:)

It should be TRANSFORMATIONS_API as it regards build target, not class category.

t-jankowski

Ok for core part.

dmitry-gorokhov · 2025-03-11T13:12:21Z

@aobolensk General comment for this PR:
I would say I totally don't see any sence in separate ConvolutionBiased operation. Internal opset serves as opset aligned with HW plugins needs:

Several operations from external opset are executed via single node (so the expectation is to have single internal op)
Operations are extended with additional semantics (like quantization or weights compression params).

Item 1 is especially important to avoid cartesian product of operation types in bounds of item 2.
Please look at #26239 as an example of aligned internal opset extension.

So applying it for Convolution I would say we need to have 2 Internal operations:

internal::Convolution (with bias and groups (to cover GroupedConvolution case))
internal::ConvolutionQuantized (internal::Convolution + quantization params)

GPU plugin already support such semantics inplemented as intel_gpu::op::Convolution (src/plugins/intel_gpu/include/intel_gpu/op/convolution.hpp).
My proposal is to extract GPU impl (both operation and corresponding transformation) into common internal opset and reuse between CPU and GPU. This should be done in 2 separate stage: first will be internal::Convolution and second - internal::ConvolutionQuantized.

praasz · 2025-03-13T08:20:21Z

src/common/transformations/src/ov_ops/convolution.cpp

+#include "itt.hpp"
+#include "openvino/op/util/precision_sensitive_attribute.hpp"
+
+using namespace std;


praasz · 2025-03-13T08:20:58Z

src/common/transformations/src/ov_ops/convolution.cpp

+        const auto& bias_et = get_input_element_type(2);
+        result_et = bias_et;


Suggested change

const auto& bias_et = get_input_element_type(2);

result_et = bias_et;

result_et = get_input_element_type(2);

praasz · 2025-03-13T08:23:48Z

src/common/transformations/src/ov_ops/convolution.cpp

+
+using namespace std;
+
+namespace ov {


Suggested change

namespace ov {

namespace ov::op::internal {

Just optional detail to consider

praasz · 2025-03-13T08:24:40Z

src/common/transformations/src/ov_ops/convolution.cpp

+
+    const auto output_shapes = op::shape_infer(this, input_shapes, m_pads_begin, m_pads_end);
+    set_output_type(0, result_et, output_shapes[0]);
+    set_num_spatial(num_spatial, input_shapes);


Should be set if value is undefined?

Could you please clarify what value do you mean here?

In case when num_spatial == util::num_spatial_undefined?
Maybe it should be set only when is different than num_spatial_undefined?

...ormations/src/transformations/op_conversions/convert_convolution_to_convolution_internal.cpp

src/core/shape_inference/include/internal_convolution_shape_inference.hpp

praasz · 2025-03-13T09:05:03Z

src/core/shape_inference/include/internal_convolution_shape_inference.hpp

+template <class TOp,
+          class TShape,
+          class TRShape = result_shape_t<TShape>,
+          typename std::enable_if<std::is_same<TOp, internal::Convolution>::value>::type* = nullptr>


the enable_if shoudl not be required just use internal::Convolution as type

src/core/shape_inference/include/convolution_shape_inference.hpp

...lugins/intel_cpu/tests/unit/shape_inference_test/convolution_biased_shape_inference_test.cpp

src/plugins/intel_cpu/src/utils/debug_caps_config.cpp

praasz · 2025-03-17T12:50:56Z

src/common/transformations/include/ov_ops/convolution.hpp

@@ -0,0 +1,57 @@
+// Copyright (C) 2025 Intel Corporation


Suggested change

// Copyright (C) 2025 Intel Corporation

// Copyright (C) 2018-2025 Intel Corporation

github-actions · 2025-04-14T00:27:23Z

This PR will be closed in a week because of 2 weeks of no activity.

praasz · 2025-04-22T06:50:12Z

...ons/src/transformations/op_conversions/convert_group_convolution_to_convolution_internal.cpp

+        }
+
+        int64_t groups = -1;
+        auto weights_shape = gconv->get_input_partial_shape(1);


Suggested change

auto weights_shape = gconv->get_input_partial_shape(1);

const auto& weights_shape = gconv->get_input_partial_shape(1);

praasz · 2025-04-22T06:51:53Z

src/common/transformations/src/transformations/op_conversions/convolution_bias_fusion.cpp

+#include "ov_ops/convolution.hpp"
+#include "transformations/utils/utils.hpp"
+
+static inline std::vector<size_t> getNormalizedDimsBySize(const std::vector<size_t>& dims, size_t ndims) {


Suggested change

static inline std::vector<size_t> getNormalizedDimsBySize(const std::vector<size_t>& dims, size_t ndims) {

static inline std::vector<size_t> getNormalizedDimsBySize(const ov::Shape& dims, size_t ndims) {

praasz · 2025-04-22T06:53:02Z

src/common/transformations/src/transformations/op_conversions/convolution_bias_fusion.cpp

+        ov::NodeVector new_ops;
+
+        std::shared_ptr<ov::Node> final_bias = bias;
+        auto add_shape = add->get_output_partial_shape(0);


Suggested change

auto add_shape = add->get_output_partial_shape(0);

const auto& add_shape = add->get_output_partial_shape(0);

praasz · 2025-04-22T06:54:04Z

src/common/transformations/src/transformations/op_conversions/convolution_bias_fusion.cpp

+
+ov::pass::ConvolutionBiasFusion::ConvolutionBiasFusion() {
+    MATCHER_SCOPE(ConvolutionBiasFusion);
+    using namespace ov::pass::pattern;


Can be removed?

aobolensk added do_not_review do_not_merge labels Feb 19, 2025

github-actions bot added category: CPU OpenVINO CPU plugin category: transformations OpenVINO Runtime library - Transformations labels Feb 19, 2025

aobolensk force-pushed the fuse-bias branch from 1eefa01 to fa96b37 Compare February 20, 2025 14:22

github-actions bot added the category: Core OpenVINO Core (aka ngraph) label Feb 21, 2025

aobolensk force-pushed the fuse-bias branch 9 times, most recently from 2c87699 to ce6fcca Compare February 27, 2025 19:51

aobolensk removed do_not_review do_not_merge labels Feb 27, 2025

aobolensk marked this pull request as ready for review February 27, 2025 20:05

aobolensk requested review from a team as code owners February 27, 2025 20:05

aobolensk requested review from itikhono and removed request for a team February 27, 2025 20:05

mlukasze requested review from jane-intel and CuriousPanCake February 28, 2025 05:42

t-jankowski reviewed Feb 28, 2025

View reviewed changes

aobolensk force-pushed the fuse-bias branch from 98f76d9 to fbf6810 Compare March 6, 2025 10:17

rkazants reviewed Mar 6, 2025

View reviewed changes

aobolensk requested review from zKulesza and removed request for a team March 10, 2025 09:09

github-actions bot added the category: docs OpenVINO documentation label Mar 10, 2025

aobolensk force-pushed the fuse-bias branch from 9a8744f to 7e62a5e Compare March 10, 2025 09:11

t-jankowski approved these changes Mar 10, 2025

View reviewed changes

aobolensk force-pushed the fuse-bias branch 6 times, most recently from 447e18b to 14089a2 Compare March 12, 2025 15:13

praasz reviewed Mar 13, 2025

View reviewed changes

aobolensk force-pushed the fuse-bias branch from 14089a2 to f0ec8fd Compare March 14, 2025 13:28

praasz reviewed Mar 17, 2025

View reviewed changes

src/plugins/intel_cpu/src/utils/debug_caps_config.cpp Outdated Show resolved Hide resolved

aobolensk force-pushed the fuse-bias branch 2 times, most recently from 9cbf58e to 53607e1 Compare March 17, 2025 08:11

github-actions bot removed the category: GPU OpenVINO GPU plugin label Mar 17, 2025

praasz reviewed Mar 17, 2025

View reviewed changes

aobolensk force-pushed the fuse-bias branch 3 times, most recently from a2bf6e8 to 97c687e Compare March 21, 2025 08:36

github-actions bot added the Stale label Apr 14, 2025

aobolensk force-pushed the fuse-bias branch 3 times, most recently from acfbc8d to b5bd157 Compare April 17, 2025 17:22

fuse bias

b5bd157

github-actions bot removed the Stale label Apr 18, 2025

praasz reviewed Apr 22, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Transformations][CPU] Introduce Convolution fusion with bias #29076

[Transformations][CPU] Introduce Convolution fusion with bias #29076

aobolensk commented Feb 19, 2025 •

edited

Loading

t-jankowski Feb 28, 2025

praasz Mar 3, 2025

aobolensk Mar 7, 2025

CuriousPanCake commented Mar 3, 2025

rkazants Mar 6, 2025

aobolensk Mar 6, 2025

rkazants Mar 6, 2025

aobolensk Mar 6, 2025

t-jankowski Mar 6, 2025

t-jankowski left a comment

dmitry-gorokhov commented Mar 11, 2025 •

edited

Loading

praasz Mar 13, 2025

aobolensk Mar 17, 2025

praasz Mar 13, 2025

aobolensk Mar 17, 2025

praasz Mar 13, 2025

aobolensk Mar 21, 2025

praasz Mar 13, 2025

aobolensk Mar 21, 2025

praasz Mar 24, 2025

praasz Mar 13, 2025

praasz Mar 17, 2025

aobolensk Mar 17, 2025

github-actions bot commented Apr 14, 2025

praasz Apr 22, 2025

praasz Apr 22, 2025

praasz Apr 22, 2025

praasz Apr 22, 2025

		const auto& bias_et = get_input_element_type(2);
		result_et = bias_et;

	const auto& bias_et = get_input_element_type(2);
	result_et = bias_et;
	result_et = get_input_element_type(2);

	// Copyright (C) 2025 Intel Corporation
	// Copyright (C) 2018-2025 Intel Corporation

	auto weights_shape = gconv->get_input_partial_shape(1);
	const auto& weights_shape = gconv->get_input_partial_shape(1);

	static inline std::vector<size_t> getNormalizedDimsBySize(const std::vector<size_t>& dims, size_t ndims) {
	static inline std::vector<size_t> getNormalizedDimsBySize(const ov::Shape& dims, size_t ndims) {

	auto add_shape = add->get_output_partial_shape(0);
	const auto& add_shape = add->get_output_partial_shape(0);

[Transformations][CPU] Introduce Convolution fusion with bias #29076

Are you sure you want to change the base?

[Transformations][CPU] Introduce Convolution fusion with bias #29076

Conversation

aobolensk commented Feb 19, 2025 • edited Loading

Details:

Tickets:

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

CuriousPanCake commented Mar 3, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

t-jankowski left a comment

Choose a reason for hiding this comment

dmitry-gorokhov commented Mar 11, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Apr 14, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aobolensk commented Feb 19, 2025 •

edited

Loading

dmitry-gorokhov commented Mar 11, 2025 •

edited

Loading