Skip to content
Merged
Show file tree
Hide file tree
Changes from 11 commits
Commits
Show all changes
54 commits
Select commit Hold shift + click to select a range
db7d221
Create convertor.py
edwinjosechittilappilly Apr 23, 2025
0d8388b
Merge branch 'main' into convert_component
edwinjosechittilappilly Apr 24, 2025
52d667c
[autofix.ci] apply automated fixes
autofix-ci[bot] Apr 24, 2025
a12ef7f
Merge branch 'main' into convert_component
edwinjosechittilappilly Apr 24, 2025
4658c14
convert component
edwinjosechittilappilly Apr 24, 2025
bf30499
Merge branch 'convert_component' of https://github.com/langflow-ai/la…
edwinjosechittilappilly Apr 24, 2025
c3164f9
[autofix.ci] apply automated fixes
autofix-ci[bot] Apr 24, 2025
588fc41
add Type_conversion base class with dispatchers for performance based…
edwinjosechittilappilly Apr 29, 2025
765d40f
fix lint issues
edwinjosechittilappilly Apr 29, 2025
e65c60e
add type_convertor test
edwinjosechittilappilly Apr 29, 2025
87ef657
[autofix.ci] apply automated fixes
autofix-ci[bot] Apr 29, 2025
3dd3307
update tests
edwinjosechittilappilly Apr 29, 2025
2c0ba21
Merge branch 'convert_component' of https://github.com/langflow-ai/la…
edwinjosechittilappilly Apr 29, 2025
6ab11ff
Merge branch 'main' into convert_component
edwinjosechittilappilly Apr 29, 2025
00a4b16
Merge branch 'convert_component' of https://github.com/langflow-ai/la…
edwinjosechittilappilly Apr 29, 2025
7ff74ae
fix tests
edwinjosechittilappilly May 9, 2025
ec023e2
Merge branch 'main' into convert_component
edwinjosechittilappilly May 9, 2025
c02b3ef
Merge branch 'main' into convert_component
edwinjosechittilappilly May 12, 2025
c555554
update with auto conversion methods
edwinjosechittilappilly May 15, 2025
2c97cab
Merge branch 'main' into convert_component
edwinjosechittilappilly May 16, 2025
db08285
update function to component file
edwinjosechittilappilly May 19, 2025
b4e83cc
Merge branch 'main' into convert_component
edwinjosechittilappilly May 20, 2025
5343410
Merge branch 'main' into convert_component
edwinjosechittilappilly May 22, 2025
b779bd3
Merge branch 'main' into convert_component
edwinjosechittilappilly May 22, 2025
78b0701
Merge branch 'main' into convert_component
edwinjosechittilappilly May 29, 2025
d00eb5c
feat: enhance input validation for Data, DataFrame, and Message types
ogabrielluiz Apr 2, 2025
357d1e3
test: add unit tests for DataInput, MessageInput, and DataFrameInput …
ogabrielluiz Apr 2, 2025
f0ff9df
updated changes to use type classes
edwinjosechittilappilly May 29, 2025
13a8d70
[autofix.ci] apply automated fixes
autofix-ci[bot] May 29, 2025
08136b9
add convert logic
edwinjosechittilappilly May 29, 2025
7705c72
update to converter
edwinjosechittilappilly May 29, 2025
bd01612
update converts
edwinjosechittilappilly May 29, 2025
61370c1
Update converter.py
edwinjosechittilappilly May 29, 2025
d1a62ef
Merge branch 'main' into convert_component
edwinjosechittilappilly May 29, 2025
6e108ae
[autofix.ci] apply automated fixes
autofix-ci[bot] May 29, 2025
484faf6
revert converter.py
edwinjosechittilappilly May 30, 2025
54d66f5
Update inputs.py
edwinjosechittilappilly May 30, 2025
33f467b
Update test_inputs.py
edwinjosechittilappilly May 30, 2025
8613a67
update to logic
edwinjosechittilappilly May 30, 2025
9475ff6
Update test_type_convertor_component.py
edwinjosechittilappilly May 30, 2025
19dcfdc
update converter
edwinjosechittilappilly May 30, 2025
f7ef291
Merge branch 'main' into convert_component
edwinjosechittilappilly May 30, 2025
5953a6b
[autofix.ci] apply automated fixes
autofix-ci[bot] May 30, 2025
9d2ffae
Merge branch 'main' into convert_component
carlosrcoelho Jun 2, 2025
3903d7b
refactor: rename conversion functions for clarity
ogabrielluiz Jun 2, 2025
835b7ce
fix: add TYPE_CHECKING for conditional imports in message.py
ogabrielluiz Jun 2, 2025
821f9f0
refactor: simplify data conversion methods in Message class
ogabrielluiz Jun 2, 2025
5a2f9f6
refactor: enhance DataFrame methods for clarity and type safety
ogabrielluiz Jun 2, 2025
b7314f0
refactor: streamline Data class methods for improved clarity
ogabrielluiz Jun 2, 2025
53af8db
refactor: simplify conversion method calls by removing redundant argu…
ogabrielluiz Jun 2, 2025
84ed405
rename test file
ogabrielluiz Jun 2, 2025
dc498f6
refactor: remove obsolete test file for data conversion
ogabrielluiz Jun 2, 2025
619cd6a
refactor: add support for converting dictionary to DataFrame
ogabrielluiz Jun 2, 2025
1d8fc2c
Merge branch 'main' into convert_component
Yukiyukiyeah Jun 2, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Empty file.
37 changes: 37 additions & 0 deletions src/backend/base/langflow/base/processing/type_conversion.py
Copy link
Contributor

@ogabrielluiz ogabrielluiz May 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have a PR (#7412) that does something similar. What we need are classmethods (which I think we have most of them).

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, should we build a component on top of this ?
Or should we keep this component on hold?

Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
import json

from langflow.schema import Data, DataFrame, Message

# Type conversion dispatchers
_message_converters = {
Message: lambda msg: Message(text=msg.get_text()),
Data: lambda data: Message(text=json.dumps(data.data)),
DataFrame: lambda df: Message(text=df.to_markdown(index=False)),
}

_data_converters = {
Message: lambda msg: Data(data=msg.data),
Copy link

Copilot AI Apr 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The lambda for Message in _data_converters assumes a 'data' attribute on Message, but the tests expect the message text to be used (e.g., {"text": "Hello"}). Consider updating the lambda to use msg.get_text() or msg.text.

Suggested change
Message: lambda msg: Data(data=msg.data),
Message: lambda msg: Data(data=msg.get_text()),

Copilot uses AI. Check for mistakes.
Data: lambda data: data,
DataFrame: lambda df: Data(data={"records": df.to_dict(orient="records")}),
}

_dataframe_converters = {
DataFrame: lambda df: df,
Data: lambda data: DataFrame([dict(data.data) if data.data else {}]),
Message: lambda msg: DataFrame([dict(msg.data) if msg.data else {}]),
Copy link

Copilot AI Apr 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The lambda for Message in _dataframe_converters uses msg.data, but Message objects do not have a data attribute. Update this lambda to extract the text from the Message instead, for example by using msg.get_text().

Suggested change
Message: lambda msg: DataFrame([dict(msg.data) if msg.data else {}]),
Message: lambda msg: DataFrame([{"text": msg.get_text()} if msg.get_text() else {}]),

Copilot uses AI. Check for mistakes.
}


def get_message_converter() -> dict:
"""Get the message conversion dispatcher."""
return _message_converters


def get_data_converter() -> dict:
"""Get the data conversion dispatcher."""
return _data_converters


def get_dataframe_converter() -> dict:
"""Get the dataframe conversion dispatcher."""
return _dataframe_converters
2 changes: 2 additions & 0 deletions src/backend/base/langflow/components/processing/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@
from .regex import RegexExtractorComponent
from .select_data import SelectDataComponent
from .split_text import SplitTextComponent
from .type_converter import TypeConverterComponent
from .update_data import UpdateDataComponent

__all__ = [
Expand All @@ -34,5 +35,6 @@
"RegexExtractorComponent",
"SelectDataComponent",
"SplitTextComponent",
"TypeConverterComponent",
"UpdateDataComponent",
]
134 changes: 134 additions & 0 deletions src/backend/base/langflow/components/processing/convertor.py
Copy link
Contributor

@ogabrielluiz ogabrielluiz May 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I still think the conversion logic should be in methods. Soon we will need to use these in other places and once we move to serializers it will be troublesome to phase these out.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Earlier, the converter was at the base of processing.
Can I move to generic component utils in the langflow base?

Or should we add this to the core component logic so it can be utilised across all other components?

Original file line number Diff line number Diff line change
@@ -0,0 +1,134 @@
import json
from typing import Any

from langflow.base.processing.type_conversion import (
get_data_converter,
get_dataframe_converter,
get_message_converter,
)
from langflow.custom import Component
from langflow.io import HandleInput, Output, TabInput
from langflow.schema import Data, DataFrame
from langflow.schema.message import Message


class TypeConverterComponent(Component):
display_name = "Type Convert"
description = "Convert between different types (Message, Data, DataFrame)"
icon = "repeat"

# Class-level conversion dispatchers
_message_converters = get_message_converter()
_data_converters = get_data_converter()
_dataframe_converters = get_dataframe_converter()

inputs = [
HandleInput(
name="input_data",
display_name="Input",
input_types=["Message", "Data", "DataFrame"],
info="Accept Message, Data or DataFrame as input",
required=True,
),
TabInput(
name="output_type",
display_name="Output Type",
options=["Message", "Data", "DataFrame"],
info="Select the desired output data type",
real_time_refresh=True,
value="Message",
),
]

outputs = [Output(display_name="Message Output", name="message_output", method="convert_to_message")]

def update_outputs(self, frontend_node: dict, field_name: str, field_value: Any) -> dict:
"""Dynamically show only the relevant output based on the selected output type."""
if field_name == "output_type":
# Start with empty outputs
frontend_node["outputs"] = []

# Add only the selected output type
if field_value == "Message":
frontend_node["outputs"].append(
Output(display_name="Message Output", name="message_output", method="convert_to_message").to_dict()
)
elif field_value == "Data":
frontend_node["outputs"].append(
Output(display_name="Data Output", name="data_output", method="convert_to_data").to_dict()
)
elif field_value == "DataFrame":
frontend_node["outputs"].append(
Output(
display_name="DataFrame Output", name="dataframe_output", method="convert_to_dataframe"
).to_dict()
)

return frontend_node

def _safe_convert(self, data: Any) -> str:
"""Safely convert input data to string."""
try:
if isinstance(data, str):
return data
if isinstance(data, Message):
return data.get_text()
if isinstance(data, Data):
return json.dumps(data.data)
if isinstance(data, DataFrame):
# Remove empty rows
data = data.dropna(how="all")
# Remove empty lines in each cell
data = data.replace(r"^\s*$", "", regex=True)
# Replace multiple newlines with a single newline
data = data.replace(r"\n+", "\n", regex=True)
return data.to_markdown(index=False)
return str(data)
except (ValueError, TypeError, AttributeError) as e:
msg = f"Error converting data: {e!s}"
raise ValueError(msg) from e

def convert_to_message(self) -> Message:
"""Convert input data to string with proper error handling."""
result = ""
if isinstance(self.input_data, list):
result = "\n".join([self._safe_convert(item) for item in self.input_data])
else:
result = self._safe_convert(self.input_data)
self.log(f"Converted to string with length: {len(result)}")
message = Message(text=result)
self.status = message
return message

def convert_to_data(self) -> Data:
"""Convert input to Data type."""
input_data = self.input_data

converter = self._data_converters.get(type(input_data))
if converter:
try:
return converter(input_data)
except (ValueError, TypeError, AttributeError) as e:
self.log(f"Error converting to Data: {e!s}")
return Data(data={"text": str(input_data)})

# Default fallback
return Data(data={"value": str(input_data)})

def convert_to_dataframe(self) -> DataFrame:
"""Convert input to DataFrame type."""
input_data = self.input_data
converter = self._dataframe_converters.get(type(input_data))
if converter:
try:
return converter(input_data)
except (ValueError, TypeError, AttributeError) as e:
self.log(f"Error converting to DataFrame: {e!s}")
import pandas as pd

return DataFrame(pd.DataFrame({"value": [str(input_data)]}))

# Default fallback
import pandas as pd

return DataFrame(pd.DataFrame({"value": [str(input_data)]}))
Original file line number Diff line number Diff line change
@@ -0,0 +1,132 @@
import pandas as pd
import pytest
from langflow.components.processing.convertor import TypeConverterComponent
from langflow.schema import Data, DataFrame, Message

from tests.base import ComponentTestBaseWithoutClient


class TestTypeConverterComponent(ComponentTestBaseWithoutClient):
@pytest.fixture
def component_class(self):
"""Return the component class to test."""
return TypeConverterComponent

@pytest.fixture
def file_names_mapping(self):
"""Return an empty list since this component doesn't have version-specific files."""
return []

# Message to other types
def test_message_to_message(self, component_class):
"""Test converting Message to Message."""
component = component_class(input_data=Message(text="Hello World"), output_type="Message")
result = component.convert_to_message()
assert isinstance(result, Message)
assert result.text == "Hello World"

def test_message_to_data(self, component_class):
"""Test converting Message to Data."""
component = component_class(input_data=Message(text="Hello"), output_type="Data")
result = component.convert_to_data()
assert isinstance(result, Data)
assert result.data == {"text": "Hello"}

def test_message_to_dataframe(self, component_class):
"""Test converting Message to DataFrame."""
component = component_class(input_data=Message(text="Hello"), output_type="DataFrame")
result = component.convert_to_dataframe()
assert isinstance(result, DataFrame)
assert isinstance(result.data, pd.DataFrame)
assert "value" in result.data.columns
assert result.data.iloc[0]["value"] == "Hello"

# Data to other types
def test_data_to_message(self, component_class):
"""Test converting Data to Message."""
component = component_class(input_data=Data(data={"text": "Hello World"}), output_type="Message")
result = component.convert_to_message()
assert isinstance(result, Message)
assert result.text == "{'text': 'Hello World'}"
Copy link

Copilot AI Apr 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The test for converting Data to Message expects single quotes in the output, but using json.dumps in the conversion produces double quotes. Update the expected output to match the json.dumps format or adjust the conversion logic accordingly.

Suggested change
assert result.text == "{'text': 'Hello World'}"
assert result.text == '{"text": "Hello World"}'

Copilot uses AI. Check for mistakes.

def test_data_to_data(self, component_class):
"""Test converting Data to Data."""
component = component_class(input_data=Data(data={"key": "value"}), output_type="Data")
result = component.convert_to_data()
assert isinstance(result, Data)
assert result.data == {"key": "value"}

def test_data_to_dataframe(self, component_class):
"""Test converting Data to DataFrame."""
component = component_class(input_data=Data(data={"key": "value"}), output_type="DataFrame")
result = component.convert_to_dataframe()
assert isinstance(result, DataFrame)
assert isinstance(result.data, pd.DataFrame)
assert "value" in result.data.columns
assert result.data.iloc[0]["value"] == "{'key': 'value'}"

# DataFrame to other types
def test_dataframe_to_message(self, component_class):
"""Test converting DataFrame to Message."""
test_df = pd.DataFrame({"col1": ["Hello"], "col2": ["World"]})
component = component_class(input_data=DataFrame(test_df), output_type="Message")
result = component.convert_to_message()
assert isinstance(result, Message)
assert result.text == "| col1 | col2 |\n|--------|--------|\n| Hello | World |"

def test_dataframe_to_data(self, component_class):
"""Test converting DataFrame to Data."""
test_df = pd.DataFrame({"col1": ["Hello"]})
component = component_class(input_data=DataFrame(test_df), output_type="Data")
result = component.convert_to_data()
assert isinstance(result, Data)
assert isinstance(result.data, dict)

def test_dataframe_to_dataframe(self, component_class):
"""Test converting DataFrame to DataFrame."""
test_df = pd.DataFrame({"col1": ["Hello"]})
component = component_class(input_data=DataFrame(test_df), output_type="DataFrame")
result = component.convert_to_dataframe()
assert isinstance(result, DataFrame)
assert isinstance(result.data, pd.DataFrame)
assert "col1" in result.data.columns
assert result.data.iloc[0]["col1"] == "Hello"

# Additional helper tests
def test_safe_convert(self, component_class):
"""Test the _safe_convert method."""
component = component_class(input_data=Message(text="Hello"), output_type="Message")

# Test with Message
result = component._safe_convert(Message(text="Hello"))
assert result == "Hello"

# Test with Data
result = component._safe_convert(Data(data={"text": "Hello"}))
assert result == "{'text': 'Hello'}"

# Test with DataFrame
test_df = pd.DataFrame({"col1": ["Hello"]})
result = component._safe_convert(DataFrame(test_df))
assert "| col1 |" in result
assert "| Hello |" in result

def test_update_outputs(self, component_class):
"""Test the update_outputs method."""
component = component_class(input_data=Message(text="Hello"), output_type="Message")
frontend_node = {"outputs": []}

# Test with Message output
updated = component.update_outputs(frontend_node, "output_type", "Message")
assert len(updated["outputs"]) == 1
assert updated["outputs"][0]["name"] == "message_output"

# Test with Data output
updated = component.update_outputs(frontend_node, "output_type", "Data")
assert len(updated["outputs"]) == 1
assert updated["outputs"][0]["name"] == "data_output"

# Test with DataFrame output
updated = component.update_outputs(frontend_node, "output_type", "DataFrame")
assert len(updated["outputs"]) == 1
assert updated["outputs"][0]["name"] == "dataframe_output"
Loading