feat: Enhance Suggestion Feature with Cosine Similarity Approach

[ebinzack15]

Is your feature request related to a problem? Please describe.

Yes, this feature request addresses limitations in the current "suggestion" feature of the bpmn.io modeler, as documented in issues [#32](https://github.com/bpmn-io/bpmn-ai/issues/32) and [#24](https://github.com/bpmn-io/bpmn-ai/issues/24). The existing solution, which uses OpenAI function calling to suggest Connector templates based on a BPMN element’s name and type, has the following problems:

• Issue Suggestions don't work for Start Events bpmn-io/refactorings#32: Template IDs exceeding 64 characters cause 400 errors, preventing suggestions for Start Events.
• Issue Research using fine-tuned, smaller model for better performance and lower cost bpmn-io/refactorings#24: GPT-3.5 with function calling performs poorly for generic element names (e.g., "Send message") or when multiple templates apply, reducing the discoverability of out-of-the-box (OOTB) Connector templates.

---

Describe the solution you'd like

We propose a new Cosine Similarity Approach to suggest relevant Connector templates when a user clicks the "Show suggestions" button for a selected BPMN element (e.g., Task, Event). This solution avoids reliance on large language models (LLMs) like OpenAI and involves a two-step process:

1. Filtering by Element Type:
   • Load a predefined list of templates.
   • Filter the list to include only templates applicable to the selected element’s type (e.g., "bpmn:Task").
2. Semantic Similarity with Cosine Similarity:
   • Use a pre-trained SentenceTransformer model (multi-qa-mpnet-base-dot-v1) to encode:
     • The element’s name (e.g., "Send email") into a high-dimensional vector.
     • The combined name and description fields of each filtered template into vectors.
   • Compute the cosine similarity between the element’s name vector and each template’s vector.
   • Return the top 5 templates with the highest similarity scores, applying a threshold cutoff to ensure relevance.

This solution has been implemented and integrated into a forked repository at , providing more accurate and reliable suggestions compared to the current OpenAI-based approach.

---

Describe alternatives you've considered

We’ve explored and experimented with several alternative approaches to enhance the suggestion feature:

• Dot Product Similarity:
  • Replaces cosine similarity with dot product for potentially faster computation.
  • Trade-off: Less normalized than cosine similarity, which may affect consistency.
• TF-IDF Based Similarity:
  • Uses TF-IDF vectors instead of transformer embeddings to calculate similarity.
  • Trade-off: Simpler but less semantically rich than SentenceTransformer embeddings.
• Expanded Information during Check:
  • Includes additional template details (e.g., group labels, property labels/descriptions) in the embedding text for richer context.
  • Trade-off: May introduce impurity, reducing suggestion precision.

These alternatives are implemented and available for review in the experimental folder of our forked repository .

---

Additional context

To demonstrate the effectiveness of the Cosine Similarity Approach, we compared its performance against the existing OpenAI solution using test cases. The table below highlights the differences in suggestion relevance:

Test Case ID	Input Name	Input Type	OpenAI Suggestion	Cosine Suggestions
1	Send email	bpmn:Task	SendGrid Outbound Connector	SendGrid Outbound Connector (21.09), Microsoft Teams Outbound Connector (17.07), Amazon SQS Outbound Connector (16.11), Amazon SNS Outbound connector (15.61), Microsoft Office 365 Mail Connector (15.04)
2	Send message	bpmn:Task	Slack Outbound Connector	SendGrid Outbound Connector (20.71), RabbitMQ Outbound Connector (20.53), Amazon SNS Outbound connector (19.94), Microsoft Teams Outbound Connector (19.74), Slack Outbound Connector (19.63)
3	Setup project	bpmn:Task	Asana Outbound Connector	Asana Outbound Connector (19.76), GitLab Outbound Connector (16.18)
4	Call REST API	bpmn:Task	REST Outbound Connector	REST Outbound Connector (25.73), Salesforce Outbound Connector (20.63), Camunda Operate Outbound connector (18.05), Twilio Outbound Connector (15.43), AWS Lambda Outbound Connector (15.24)
5	Get distance from Google Maps	bpmn:Task	Google Maps Platform Outbound Connector	Google Maps Platform Outbound Connector (19.95)
6	Send message on Slack	bpmn:Task	Slack Outbound Connector	Slack Outbound Connector (26.76), RabbitMQ Outbound Connector (21.22), Amazon SQS Outbound Connector (19.40), Microsoft Teams Outbound Connector (18.41), Amazon SNS Outbound connector (18.37)
7	Receive order	bpmn:StartEvent	error	Amazon SQS Message Start Event Connector (17.11), Amazon SQS Start Event Connector (16.30), Amazon EventBridge Message Start Event Connector (15.06)
8	Wait for approval	bpmn:IntermediateCatchEvent	error	Amazon SQS Intermediate Message Catch Event connector (15.48)
9	GitHub issue	bpmn:BoundaryEvent	GitHub Webhook Boundary Event Connector	GitHub Webhook Boundary Event Connector (20.53)
10	Create Slack channel	bpmn:Task	Slack Outbound Connector	Slack Outbound Connector (28.21), RabbitMQ Outbound Connector (17.73), GitLab Outbound Connector (15.66), Microsoft Teams Outbound Connector (15.53), GitHub Outbound Connector (15.29)
11	Email sender	bpmn:Task	SendGrid Outbound Connector	SendGrid Outbound Connector (18.41), Easy Post Outbound Connector (15.10)
12	Order received	bpmn:StartEvent	error	Amazon SQS Message Start Event Connector (15.88), Amazon SQS Start Event Connector (15.50)
13	Approval wait	bpmn:IntermediateCatchEvent	error	Amazon SQS Intermediate Message Catch Event connector (15.78)
14	Notification sender	bpmn:IntermediateThrowEvent	error	Amazon SQS Intermediate Message Catch Event connector (17.61), Slack Webhook Intermediate Catch Event Connector (15.50), RabbitMQ Intermediate Catch Event Connector (15.06)
15	Send email to customer	bpmn:Task	SendGrid Outbound Connector	SendGrid Outbound Connector (19.53), Amazon SQS Outbound Connector (17.09), Amazon SNS Outbound connector (16.32), Microsoft Teams Outbound Connector (16.11), Microsoft Office 365 Mail Connector (15.15)
16	Make API request	bpmn:Task	REST Outbound Connector	REST Outbound Connector (22.79), Salesforce Outbound Connector (21.34), Camunda Operate Outbound connector (19.80), AWS Lambda Outbound Connector (16.06), Automation Anywhere Outbound Connector (15.97)
17	Email	bpmn:IntermediateCatchEvent	error	No suggestions
18	Receive Email	bpmn:IntermediateCatchEvent	error	Amazon SQS Intermediate Message Catch Event connector (16.37), RabbitMQ Intermediate Catch Event Connector (15.26), Twilio Intermediate Catch Event Connector (15.09)
19	Approval	bpmn:Task	No suggestions	No suggestions
20	Approval	bpmn:IntermediateCatchEvent	error	No suggestions
21	Send Email Notification	bpmn:Task	SendGrid Outbound Connector	SendGrid Outbound Connector (21.07), Amazon SQS Outbound Connector (18.16), Amazon SNS Outbound connector (17.59), Microsoft Teams Outbound Connector (17.52), Slack Outbound Connector (16.17)
22	Fire John	bpmn:Task	Slack Outbound Connector	No suggestions
23	SEND EMAIL	bpmn:Task	SendGrid Outbound Connector	SendGrid Outbound Connector (21.09), Microsoft Teams Outbound Connector (17.07), Amazon SQS Outbound Connector (16.11), Amazon SNS Outbound connector (15.61), Microsoft Office 365 Mail Connector (15.04)
24	Call external API to get data	bpmn:Task	REST Outbound Connector	Salesforce Outbound Connector (23.89), Camunda Operate Outbound connector (23.01), REST Outbound Connector (22.84), Twilio Outbound Connector (18.61), AWS Lambda Outbound Connector (17.35)
25	Call Google Maps API	bpmn:Task	Google Maps Platform Outbound Connector	Google Maps Platform Outbound Connector (18.92), Salesforce Outbound Connector (17.58), REST Outbound Connector (17.55), Camunda Operate Outbound connector (15.88), AWS Lambda Outbound Connector (15.05)
26	Call some API	bpmn:Task	REST Outbound Connector	Salesforce Outbound Connector (22.36), REST Outbound Connector (22.04), Camunda Operate Outbound connector (19.29), AWS Lambda Outbound Connector (17.18), Automation Anywhere Outbound Connector (16.68)
27	Process payment	bpmn:Task	Salesforce Outbound Connector	Salesforce Outbound Connector (15.00)
28	User registration	bpmn:StartEvent	error	No suggestions
29	Timer-based reminder	bpmn:IntermediateCatchEvent	error	No suggestions
30	Log system event	bpmn:IntermediateThrowEvent	error	Amazon SQS Intermediate Message Catch Event connector (15.62), RabbitMQ Intermediate Catch Event Connector (15.26)
31	Compensate transaction	bpmn:BoundaryEvent	No suggestions	No suggestions
32	Send email with attachment	bpmn:Task	SendGrid Outbound Connector	SendGrid Outbound Connector (20.03), Microsoft Office 365 Mail Connector (15.01)
33	Wait for customer approval	bpmn:IntermediateCatchEvent	error	Amazon SQS Intermediate Message Catch Event connector (15.33)
34	Error handling mechanism	bpmn:BoundaryEvent	No suggestions	No suggestions
35	Compensate failed transaction	bpmn:BoundaryEvent	Webhook Boundary Event Connector	No suggestions
36	Log important event	bpmn:IntermediateThrowEvent	error	No suggestions
37	Start process on system boot	bpmn:StartEvent	error	Kafka Message Start Event Connector (15.27)
38	User logs in successfully	bpmn:StartEvent	error	No suggestions
39	Timer event after 24 hours	bpmn:IntermediateCatchEvent	error	No suggestions
40	Escalate to manager	bpmn:IntermediateThrowEvent	error	No suggestions
41	Validate user input, store results in DB, then trigger notification email	bpmn:Task	REST Outbound Connector	No suggestions
41	Validate user input, store results in DB, then trigger notification email	bpmn:Task	SQL Database Connector	No suggestions
41	Validate user input, store results in DB, then trigger notification email	bpmn:Task	SendGrid Outbound Connector	No suggestions
42	Initiate Payment Process	bpmn:StartEvent	error	Amazon SQS Start Event Connector (15.34)
43	Data Synchronization Timeout	bpmn:IntermediateThrowEvent	error	No suggestions
44	Boundary Timer for SLA breach	bpmn:BoundaryEvent	No suggestions	Amazon SQS Boundary Event Connector (19.38), HTTP Polling Boundary Catch Event Connector (19.07), Kafka Boundary Event Connector (18.70), Webhook Boundary Event Connector (17.01), Slack Webhook Boundary Event Connector (16.87)
45	Review & Approve Contract Documents	bpmn:Task	Google Drive Outbound Connector	Automation Anywhere Outbound Connector (15.04)
46	Wait for 2 hours	bpmn:IntermediateCatchEvent	error	Amazon SQS Intermediate Message Catch Event connector (15.35)
47	Callback to External System with JSON Payload	bpmn:Task	REST Outbound Connector	Salesforce Outbound Connector (22.08), REST Outbound Connector (21.69), Camunda Operate Outbound connector (18.87), AWS Lambda Outbound Connector (17.80), RabbitMQ Outbound Connector (17.12)
48		bpmn:Task	No suggestions	No suggestions
49		bpmn:StartEvent	No suggestions	No suggestions
50	boundary_event_12345!?$	bpmn:BoundaryEvent	No suggestions	RabbitMQ Boundary Event Connector (19.96), Amazon SQS Boundary Event Connector (19.85), Webhook Boundary Event Connector (18.13), Kafka Boundary Event Connector (17.97), Amazon EventBridge Boundary Event Connector (17.67)
51	Fire-and-Forget Notification	bpmn:IntermediateThrowEvent	error	Amazon SQS Intermediate Message Catch Event connector (17.11)
52	Escalate to Human Operator if No Response in 24h	bpmn:IntermediateThrowEvent	error	No suggestions
53	Check XYZ System’s Health and Retry Connection	bpmn:Task	REST Outbound Connector	Camunda Operate Outbound connector (16.80), Blue Prism Outbound Connector (16.34), Google Sheets Outbound Connector (15.34), Kafka Outbound Connector (15.19), Amazon SQS Outbound Connector (15.12)
54	非常に長い名前のタスク - 国際化テスト (Internationalization Test)	bpmn:Task	No suggestions	No suggestions
55	Multiple Approvals from Different Departments Required	bpmn:IntermediateCatchEvent	error	No suggestions
56	--------------------	bpmn:Task	No suggestions	No suggestions
57	Generate Monthly Financial Reports & Send PDF to CFO	bpmn:Task	Google Sheets Outbound Connector	No suggestions
57	Generate Monthly Financial Reports & Send PDF to CFO	bpmn:Task	SendGrid Outbound Connector	No suggestions
58	Initialize Process Flow For Quarterly Audit	bpmn:StartEvent	error	Amazon SQS Start Event Connector (15.09)
59	Boundary Error Catching For Database Exception	bpmn:BoundaryEvent	No suggestions	HTTP Polling Boundary Catch Event Connector (18.84), Amazon SQS Boundary Event Connector (18.56), Webhook Boundary Event Connector (18.03), RabbitMQ Boundary Event Connector (17.83), Twilio Boundary Event Connector (16.91)
60	Sensor Trigger - Catch Motion Input	bpmn:IntermediateCatchEvent	error	Amazon SQS Intermediate Message Catch Event connector (17.55), Webhook Intermediate Event Connector (16.39), HTTP Polling Intermediate Catch Event Connector (16.37), Kafka Intermediate Catch Event Connector (16.25), Amazon EventBridge Intermediate Catch Event Connector (15.80)
61	Signal Throw to Downstream Systems	bpmn:IntermediateThrowEvent	error	Amazon SQS Intermediate Message Catch Event connector (18.18), Amazon EventBridge Intermediate Catch Event Connector (17.63), RabbitMQ Intermediate Catch Event Connector (17.28), Slack Webhook Intermediate Catch Event Connector (16.50), Kafka Intermediate Catch Event Connector (16.20)

The Cosine Similarity Approach consistently provides a broader, more relevant set of suggestions, addressing the limitations of generic names and providing multiple recommendatons improving discoverability of Connector templates.

While the author mentioned interest in training a small LLM model, our analysis indicates that the Cosine Similarity Approach is effective and efficient for this use case. We haven’t conducted a feasibility study on fine-tuning an LLM for this usecase yet, but it could be explored as a future enhancement.

About Us:

We are a small team of developers based in Bangalore, passionate about solving compelling problems using AI and making AI technologies accessible to all.

[sujithatzackriya]

Hey Ebin, Good document. Please find my comments below

1. Please add why fine tuning is not feasible as this requires a good dataset and context to be provided. Fine-tuned models tend to hallucinate and these general purpose base models are not good at suggesting things. Even if we manage to fine-tune a model, the results will not be consistant. Cosine similarity solves this problem. Also, multi-qa-mpnet-base-dot-v1 model is half the size of even the smallest LLMs. so the memory required to run the suggested model is less compared to a fine-tuned model.
2. Please add the next steps and ask suggestions on how to proceed. As of now, we have a python backend for running our Ai backend and the integration with given bpmn.js sample is done. If we create a PR to the refactoring repo, is it going to be useful / How can we contribute?

[sujithatzackriya]

Suggestion

Is your feature request related to a problem? Please describe.

Yes, We hope to solve 2 out of the 3 issues raised in the refactoring repository with this feature request. This feature request addresses limitations in the current "suggestion" feature of the bpmn.io modeler, as documented in issues [#32](https://github.com/bpmn-io/bpmn-ai/issues/32) and [#24](https://github.com/bpmn-io/bpmn-ai/issues/24). The existing solution, which uses OpenAI function calling to suggest Connector templates based on a BPMN element’s name and type, has the following problems:

[ebinzack15]

Is your feature request related to a problem? Please describe.

Yes, we hope to solve 2 out of the 3 issues raised in the refactoring repository with this feature request. Specifically, this feature request addresses limitations in the current "suggestion" feature of the bpmn.io modeler, as documented in issues #32 and #24. The existing solution, which uses OpenAI function calling to suggest Connector templates based on a BPMN element’s name and type, has the following problems:

• Issue Suggestions don't work for Start Events bpmn-io/refactorings#32: Template IDs exceeding 64 characters cause 400 errors, preventing suggestions for Start Events.
• Issue Research using fine-tuned, smaller model for better performance and lower cost bpmn-io/refactorings#24: GPT-3.5 with function calling performs poorly for generic element names (e.g., "Send message") or when multiple templates apply, reducing the discoverability of out-of-the-box (OOTB) Connector templates.

---

Describe the solution you'd like

We propose a new Cosine Similarity Approach to suggest relevant Connector templates when a user clicks the "Show suggestions" button for a selected BPMN element (e.g., Task, Event). This solution avoids reliance on large language models (LLMs) like OpenAI and involves a two-step process:

1. Filtering by Element Type:
   • Load a predefined list of templates.
   • Filter the list to include only templates applicable to the selected element’s type (e.g., "bpmn:Task").
2. Semantic Similarity with Cosine Similarity:
   • Use a pre-trained SentenceTransformer model (multi-qa-mpnet-base-dot-v1) to encode:
     • The element’s name (e.g., "Send email") into a high-dimensional vector.
     • The combined name and description fields of each filtered template into vectors.
   • Compute the cosine similarity between the element’s name vector and each template’s vector.
   • Return the top 5 templates with the highest similarity scores, applying a threshold cutoff to ensure relevance.

This solution has been implemented and integrated into a forked repository at , providing more accurate and reliable suggestions compared to the current OpenAI-based approach.

---

Describe alternatives you've considered

We’ve explored and experimented with several alternative approaches to enhance the suggestion feature:

• Dot Product Similarity:
  • Replaces cosine similarity with dot product for potentially faster computation.
  • Trade-off: Less normalized than cosine similarity, which may affect consistency.
• TF-IDF Based Similarity:
  • Uses TF-IDF vectors instead of transformer embeddings to calculate similarity.
  • Trade-off: Simpler but less semantically rich than SentenceTransformer embeddings.
• Expanded Information during Check:
  • Includes additional template details (e.g., group labels, property labels/descriptions) in the embedding text for richer context.
  • Trade-off: May introduce impurity, reducing suggestion precision.

These alternatives are implemented and available for review in the experimental folder of our forked repository .

---

Additional context

To demonstrate the effectiveness of the Cosine Similarity Approach, we compared its performance against the existing OpenAI solution using test cases. The table below highlights the differences in suggestion relevance:

Test Case ID	Input Name	Input Type	OpenAI Suggestion	Cosine Suggestions
1	Send email	bpmn:Task	SendGrid Outbound Connector	SendGrid Outbound Connector (21.09), Microsoft Teams Outbound Connector (17.07), Amazon SQS Outbound Connector (16.11), Amazon SNS Outbound connector (15.61), Microsoft Office 365 Mail Connector (15.04)
2	Send message	bpmn:Task	Slack Outbound Connector	SendGrid Outbound Connector (20.71), RabbitMQ Outbound Connector (20.53), Amazon SNS Outbound connector (19.94), Microsoft Teams Outbound Connector (19.74), Slack Outbound Connector (19.63)
3	Setup project	bpmn:Task	Asana Outbound Connector	Asana Outbound Connector (19.76), GitLab Outbound Connector (16.18)
4	Call REST API	bpmn:Task	REST Outbound Connector	REST Outbound Connector (25.73), Salesforce Outbound Connector (20.63), Camunda Operate Outbound connector (18.05), Twilio Outbound Connector (15.43), AWS Lambda Outbound Connector (15.24)
5	Get distance from Google Maps	bpmn:Task	Google Maps Platform Outbound Connector	Google Maps Platform Outbound Connector (19.95)
6	Send message on Slack	bpmn:Task	Slack Outbound Connector	Slack Outbound Connector (26.76), RabbitMQ Outbound Connector (21.22), Amazon SQS Outbound Connector (19.40), Microsoft Teams Outbound Connector (18.41), Amazon SNS Outbound connector (18.37)
7	Receive order	bpmn:StartEvent	error	Amazon SQS Message Start Event Connector (17.11), Amazon SQS Start Event Connector (16.30), Amazon EventBridge Message Start Event Connector (15.06)
8	Wait for approval	bpmn:IntermediateCatchEvent	error	Amazon SQS Intermediate Message Catch Event connector (15.48)
9	GitHub issue	bpmn:BoundaryEvent	GitHub Webhook Boundary Event Connector	GitHub Webhook Boundary Event Connector (20.53)
10	Create Slack channel	bpmn:Task	Slack Outbound Connector	Slack Outbound Connector (28.21), RabbitMQ Outbound Connector (17.73), GitLab Outbound Connector (15.66), Microsoft Teams Outbound Connector (15.53), GitHub Outbound Connector (15.29)
11	Email sender	bpmn:Task	SendGrid Outbound Connector	SendGrid Outbound Connector (18.41), Easy Post Outbound Connector (15.10)
12	Order received	bpmn:StartEvent	error	Amazon SQS Message Start Event Connector (15.88), Amazon SQS Start Event Connector (15.50)
13	Approval wait	bpmn:IntermediateCatchEvent	error	Amazon SQS Intermediate Message Catch Event connector (15.78)
14	Notification sender	bpmn:IntermediateThrowEvent	error	Amazon SQS Intermediate Message Catch Event connector (17.61), Slack Webhook Intermediate Catch Event Connector (15.50), RabbitMQ Intermediate Catch Event Connector (15.06)
15	Send email to customer	bpmn:Task	SendGrid Outbound Connector	SendGrid Outbound Connector (19.53), Amazon SQS Outbound Connector (17.09), Amazon SNS Outbound connector (16.32), Microsoft Teams Outbound Connector (16.11), Microsoft Office 365 Mail Connector (15.15)
16	Make API request	bpmn:Task	REST Outbound Connector	REST Outbound Connector (22.79), Salesforce Outbound Connector (21.34), Camunda Operate Outbound connector (19.80), AWS Lambda Outbound Connector (16.06), Automation Anywhere Outbound Connector (15.97)
17	Email	bpmn:IntermediateCatchEvent	error	No suggestions
18	Receive Email	bpmn:IntermediateCatchEvent	error	Amazon SQS Intermediate Message Catch Event connector (16.37), RabbitMQ Intermediate Catch Event Connector (15.26), Twilio Intermediate Catch Event Connector (15.09)
19	Approval	bpmn:Task	No suggestions	No suggestions
20	Approval	bpmn:IntermediateCatchEvent	error	No suggestions
21	Send Email Notification	bpmn:Task	SendGrid Outbound Connector	SendGrid Outbound Connector (21.07), Amazon SQS Outbound Connector (18.16), Amazon SNS Outbound connector (17.59), Microsoft Teams Outbound Connector (17.52), Slack Outbound Connector (16.17)
22	Fire John	bpmn:Task	Slack Outbound Connector	No suggestions
23	SEND EMAIL	bpmn:Task	SendGrid Outbound Connector	SendGrid Outbound Connector (21.09), Microsoft Teams Outbound Connector (17.07), Amazon SQS Outbound Connector (16.11), Amazon SNS Outbound connector (15.61), Microsoft Office 365 Mail Connector (15.04)
24	Call external API to get data	bpmn:Task	REST Outbound Connector	Salesforce Outbound Connector (23.89), Camunda Operate Outbound connector (23.01), REST Outbound Connector (22.84), Twilio Outbound Connector (18.61), AWS Lambda Outbound Connector (17.35)
25	Call Google Maps API	bpmn:Task	Google Maps Platform Outbound Connector	Google Maps Platform Outbound Connector (18.92), Salesforce Outbound Connector (17.58), REST Outbound Connector (17.55), Camunda Operate Outbound connector (15.88), AWS Lambda Outbound Connector (15.05)
26	Call some API	bpmn:Task	REST Outbound Connector	Salesforce Outbound Connector (22.36), REST Outbound Connector (22.04), Camunda Operate Outbound connector (19.29), AWS Lambda Outbound Connector (17.18), Automation Anywhere Outbound Connector (16.68)
27	Process payment	bpmn:Task	Salesforce Outbound Connector	Salesforce Outbound Connector (15.00)
28	User registration	bpmn:StartEvent	error	No suggestions
29	Timer-based reminder	bpmn:IntermediateCatchEvent	error	No suggestions
30	Log system event	bpmn:IntermediateThrowEvent	error	Amazon SQS Intermediate Message Catch Event connector (15.62), RabbitMQ Intermediate Catch Event Connector (15.26)
31	Compensate transaction	bpmn:BoundaryEvent	No suggestions	No suggestions
32	Send email with attachment	bpmn:Task	SendGrid Outbound Connector	SendGrid Outbound Connector (20.03), Microsoft Office 365 Mail Connector (15.01)
33	Wait for customer approval	bpmn:IntermediateCatchEvent	error	Amazon SQS Intermediate Message Catch Event connector (15.33)
34	Error handling mechanism	bpmn:BoundaryEvent	No suggestions	No suggestions
35	Compensate failed transaction	bpmn:BoundaryEvent	Webhook Boundary Event Connector	No suggestions
36	Log important event	bpmn:IntermediateThrowEvent	error	No suggestions
37	Start process on system boot	bpmn:StartEvent	error	Kafka Message Start Event Connector (15.27)
38	User logs in successfully	bpmn:StartEvent	error	No suggestions
39	Timer event after 24 hours	bpmn:IntermediateCatchEvent	error	No suggestions
40	Escalate to manager	bpmn:IntermediateThrowEvent	error	No suggestions
41	Validate user input, store results in DB, then trigger notification email	bpmn:Task	REST Outbound Connector	No suggestions
41	Validate user input, store results in DB, then trigger notification email	bpmn:Task	SQL Database Connector	No suggestions
41	Validate user input, store results in DB, then trigger notification email	bpmn:Task	SendGrid Outbound Connector	No suggestions
42	Initiate Payment Process	bpmn:StartEvent	error	Amazon SQS Start Event Connector (15.34)
43	Data Synchronization Timeout	bpmn:IntermediateThrowEvent	error	No suggestions
44	Boundary Timer for SLA breach	bpmn:BoundaryEvent	No suggestions	Amazon SQS Boundary Event Connector (19.38), HTTP Polling Boundary Catch Event Connector (19.07), Kafka Boundary Event Connector (18.70), Webhook Boundary Event Connector (17.01), Slack Webhook Boundary Event Connector (16.87)
45	Review & Approve Contract Documents	bpmn:Task	Google Drive Outbound Connector	Automation Anywhere Outbound Connector (15.04)
46	Wait for 2 hours	bpmn:IntermediateCatchEvent	error	Amazon SQS Intermediate Message Catch Event connector (15.35)
47	Callback to External System with JSON Payload	bpmn:Task	REST Outbound Connector	Salesforce Outbound Connector (22.08), REST Outbound Connector (21.69), Camunda Operate Outbound connector (18.87), AWS Lambda Outbound Connector (17.80), RabbitMQ Outbound Connector (17.12)
48		bpmn:Task	No suggestions	No suggestions
49		bpmn:StartEvent	No suggestions	No suggestions
50	boundary_event_12345!?$	bpmn:BoundaryEvent	No suggestions	RabbitMQ Boundary Event Connector (19.96), Amazon SQS Boundary Event Connector (19.85), Webhook Boundary Event Connector (18.13), Kafka Boundary Event Connector (17.97), Amazon EventBridge Boundary Event Connector (17.67)
51	Fire-and-Forget Notification	bpmn:IntermediateThrowEvent	error	Amazon SQS Intermediate Message Catch Event connector (17.11)
52	Escalate to Human Operator if No Response in 24h	bpmn:IntermediateThrowEvent	error	No suggestions
53	Check XYZ System’s Health and Retry Connection	bpmn:Task	REST Outbound Connector	Camunda Operate Outbound connector (16.80), Blue Prism Outbound Connector (16.34), Google Sheets Outbound Connector (15.34), Kafka Outbound Connector (15.19), Amazon SQS Outbound Connector (15.12)
54	非常に長い名前のタスク - 国際化テスト (Internationalization Test)	bpmn:Task	No suggestions	No suggestions
55	Multiple Approvals from Different Departments Required	bpmn:IntermediateCatchEvent	error	No suggestions
56	--------------------	bpmn:Task	No suggestions	No suggestions
57	Generate Monthly Financial Reports & Send PDF to CFO	bpmn:Task	Google Sheets Outbound Connector	No suggestions
57	Generate Monthly Financial Reports & Send PDF to CFO	bpmn:Task	SendGrid Outbound Connector	No suggestions
58	Initialize Process Flow For Quarterly Audit	bpmn:StartEvent	error	Amazon SQS Start Event Connector (15.09)
59	Boundary Error Catching For Database Exception	bpmn:BoundaryEvent	No suggestions	HTTP Polling Boundary Catch Event Connector (18.84), Amazon SQS Boundary Event Connector (18.56), Webhook Boundary Event Connector (18.03), RabbitMQ Boundary Event Connector (17.83), Twilio Boundary Event Connector (16.91)
60	Sensor Trigger - Catch Motion Input	bpmn:IntermediateCatchEvent	error	Amazon SQS Intermediate Message Catch Event connector (17.55), Webhook Intermediate Event Connector (16.39), HTTP Polling Intermediate Catch Event Connector (16.37), Kafka Intermediate Catch Event Connector (16.25), Amazon EventBridge Intermediate Catch Event Connector (15.80)
61	Signal Throw to Downstream Systems	bpmn:IntermediateThrowEvent	error	Amazon SQS Intermediate Message Catch Event connector (18.18), Amazon EventBridge Intermediate Catch Event Connector (17.63), RabbitMQ Intermediate Catch Event Connector (17.28), Slack Webhook Intermediate Catch Event Connector (16.50), Kafka Intermediate Catch Event Connector (16.20)

The Cosine Similarity Approach consistently provides a broader, more relevant set of suggestions, addressing the limitations of generic names and providing multiple recommendations improving the discoverability of Connector templates.

While the author mentioned interest in training a small LLM model, our analysis indicates that the Cosine Similarity Approach is effective and efficient for this use case.

We haven't conducted a feasibility study on fine-tuning an LLM yet, but based on our preliminary assessment, fine-tuning might not be beneficial. Fine-tuning requires a substantial, high-quality dataset and sufficient context, both of which are challenging to provide consistently in this scenario. Moreover, fine-tuned LLMs often produce hallucinations and inconsistent outputs, especially for recommendation-type tasks. Even with successful fine-tuning, the suggestions would likely remain unreliable. In contrast, the proposed Cosine Similarity Approach avoids these pitfalls by using semantic similarity directly. Additionally, the multi-qa-mpnet-base-dot-v1 SentenceTransformer model is approximately half the size of even the smallest LLMs, requiring significantly less memory and computational resources.

We currently have a Python backend running the proposed AI model, integrated with the provided bpmn.js sample. We’re considering creating a Pull Request to the refactoring repository. Would a PR with our solution be helpful, or could you advise us on how we might best contribute this enhancement?

About Us:

We are a small team of developers based in Bangalore, passionate about solving compelling problems using AI and making AI technologies accessible to all.

[ebinzack15]

Known Issues

• With the introduction of cosine similarity, the implementation now relies on a local JSON dump file containing a list of 70 Connector templates due to errors encountered when using the built-in BPMN method.
• Manual selection and application of templates currently do not function.
• The predefined "Notify on Slack" task in the demo does not work.
• Re-suggestion and re-application of templates after applying suggestions are not operational.

This section fits just after the Additional context.

@sujithatzackriya

[ebinzack15]

Proposed Changes

This pull request enhances the Connector template suggestion feature in the BPMN modeler by replacing the existing OpenAI-based solution with a local Python backend leveraging semantic similarity (cosine similarity). This addresses known limitations such as:

• Errors caused by template IDs exceeding 64 characters ([Issue #32](Suggestions don't work for Start Events bpmn-io/refactorings#32)).
• Poor suggestion quality for generic element names like "Send message" ([Issue #24](Research using fine-tuned, smaller model for better performance and lower cost bpmn-io/refactorings#24)).

Approach

The new solution uses a pre-trained SentenceTransformer model (multi-qa-mpnet-base-dot-v1) to semantically match the selected BPMN element's name with relevant Connector templates. The process involves:

1. Filtering templates by the selected BPMN element type (e.g., bpmn:Task).
2. Encoding the element's name and template descriptions into semantic vectors.
3. Computing dot product similarity between vectors to identify the most relevant templates.
4. Returning the top 5 suggestions, applying a relevance threshold.

This approach provides more accurate, relevant, and cost-effective suggestions without relying on external APIs or large language models.

Changes Made

• Removed OpenAI Integration: Deleted OpenAI-related files and test cases.
• Added Local Python Backend: Implemented suggestion logic using cosine similarity in the suggestion_backend folder.
• API Integration: Updated frontend code to call the local backend API instead of OpenAI.
• Template Loading: Switched from built-in BPMN methods to loading templates directly from a JSON file (all_elements.json).
• Experimental Folder: Included an experimental folder documenting similarity approaches (cosine similarity, dot product, TF-IDF, expanded context embeddings). This folder is for reference only and does not affect the main application.
• Updated README: Added setup instructions for the new backend.

Visual Demo

Screenshot of suggestion feature

Demo Video

Steps to Try Out

Backend Setup (Docker)

cd suggestion_backend
docker compose up

Backend available at: http://localhost:8000/

Backend Setup (Local)

cd suggestion_backend
pip install -r requirements.txt
uvicorn app:app --host 0.0.0.0 --port 8000

Backend available at: http://localhost:8000/

Frontend Setup

npm install
npm start

Testing and Validation

We conducted comparative tests demonstrating that the cosine similarity approach consistently outperforms the previous OpenAI-based solution, especially for generic or ambiguous element names. Detailed comparison results and analysis are available in the linked enhancement issue: [Enhance Suggestion Feature with Cosine Similarity Approach #33](bpmn-io#33).

Benefits

• Improved Accuracy: Provides more relevant and reliable suggestions.
• Reduced Cost & Complexity: Eliminates dependency on external OpenAI APIs.
• Performance & Efficiency: Uses a lightweight, efficient semantic embedding model.

Key Files to Review

• suggestion_backend/app.py: Core implementation of the Python backend.
• lib/refactorings/providers/local-api/LocalAPIProvider.js: Updated API call to the new backend.

Known Issues & Future Improvements

• Currently relies on a static JSON file (all_elements.json) due to issues with built-in BPMN template loading methods.
• Manual selection and application of templates are not functional.
• Predefined "Notify on Slack" demo task does not work.
• Re-suggestion after applying a template is not functional.
• Docker image size (~10GB) has not been optimized.

[ebinzack15]

@sujithatzackriya please review above PR.

below is the author's PR format.

### Proposed Changes

<!--

Add relevant context (issue fixed or related to),
a capture of the UI changes (if any) as well as
steps to try out your changes.

- ->

### Checklist

To ensure you provided everything we need to look at your PR:

- [ ]  **Brief textual description** of the changes present
- [ ]  **Visual demo** attached
- [ ]  **Steps to try out** present, i.e. [using the `@bpmn-io/sr` tool](https://github.com/bpmn-io/sr)
- [ ]  Related issue linked via `Closes {LINK_TO_ISSUE}` or `Related to {LINK_TO_ISSUE}`

<!--

Thanks for creating this pull request! ❤️

- ->