Skip to content

[Explore vis]add data transformation panel #11944

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ruanyl merged 8 commits into from
May 26, 2026
Merged

[Explore vis]add data transformation panel #11944

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
848449b
feat: add data transformation panel
Qxisylolo May 12, 2026
e4b09e6
read tab from url and save id instead of label
Qxisylolo May 15, 2026
a9e4214
make transform panel a shared component
Qxisylolo May 18, 2026
5766222
add to metric page
Qxisylolo May 21, 2026
ec60b3e
optimize code
Qxisylolo May 22, 2026
1fe50fd
solve comments
Qxisylolo May 25, 2026
13501e0
solve comments-2
Qxisylolo May 25, 2026
a6f9deb
Merge branch 'main' into feat/add_data_transformation
Qxisylolo May 25, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions ...ugins/explore/public/application/in_context_vis_editor/component/save_vis_button.test.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,26 @@ jest.mock('./save_vis_modal', () => ({
SaveVisModal: () => <div data-test-subj="save-vis-modal">Modal</div>,
}));

jest.mock('../../../components/data_transformations', () => {
const mockObs = {
subscribe: jest.fn().mockReturnValue({ unsubscribe: jest.fn() }),
getValue: jest.fn().mockReturnValue([]),
};
const mockMapObs = {
subscribe: jest.fn().mockReturnValue({ unsubscribe: jest.fn() }),
getValue: jest.fn().mockReturnValue(new Map()),
};
return {
useTransformationService: jest.fn().mockReturnValue({
clearPipeline: jest.fn(),
pipeline$: mockObs,
getPipeline$: () => mockObs,
stageSchemas$: mockMapObs,
}),
TransformPanel: () => null,
};
});

jest.mock('@osd/i18n', () => ({
i18n: {
translate: jest.fn((key, options) => options.defaultMessage),
Expand Down Expand Up @@ -67,6 +87,7 @@ const buildVisualizationBuilder = () => ({
visualizationBuilderForEditor: {
visConfig$: { value: { type: 'bar', styles: {}, axesMapping: { x: 'field' } } },
isVisDirty$: new BehaviorSubject(false),
getTransformationService: jest.fn().mockReturnValue({ pipeline$: new BehaviorSubject([]) }),
},
});

Expand Down
11 changes: 11 additions & 0 deletions src/plugins/explore/public/application/in_context_vis_editor/component/save_vis_button.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@ export const SaveVisButton = () => {
const { queryEditorState, datasetView } = useQueryBuilderState();
const { visualizationBuilderForEditor: visualizationBuilder } = useVisualizationBuilder();
const visConfig = visualizationBuilder.visConfig$.value;
const transformationService = visualizationBuilder.getTransformationService();

const { services } = useOpenSearchDashboards<ExploreServices>();

Expand Down Expand Up @@ -125,6 +126,14 @@ export const SaveVisButton = () => {
isTitleDuplicateConfirmed,
onTitleDuplicate,
}: OnSaveProps) => {
const pipeline = transformationService.pipeline$.getValue();
const serializedPipeline = pipeline.map((instance) => ({
definitionId: instance.definition_id,
instanceId: instance.instance_id,

@ruanyl ruanyl May 24, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why need to persist instance_id? It seems it's purely a runtime identifier

@Qxisylolo Qxisylolo May 25, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, we don't need to save it, will delete it

config: instance.config,
hide: instance.hide,
}));

const axesMapping = visConfig?.axesMapping;
const currentTitle = savedExploreToSave.title;
savedExploreToSave.title = newTitle;
Expand All @@ -137,6 +146,7 @@ export const SaveVisButton = () => {
splitField: visConfig?.splitField,
splitLayout: visConfig?.splitLayout,
showSplitLabel: visConfig?.showSplitLabel,
dataTransformationJSON: JSON.stringify(serializedPipeline),

@ruanyl ruanyl May 24, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Curious why does it need JSON.stringify() for serializedPipeline?

@Qxisylolo Qxisylolo May 25, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don’t need to double-stringify this, I assume it is the leftover code from the initial changes.

});
savedExploreToSave.version = 1;

Expand Down Expand Up @@ -196,6 +206,7 @@ export const SaveVisButton = () => {
toastNotifications,
isPromptMode,
queryEditorState.lastExecutedTranslatedQuery,
transformationService,
]
);

Expand Down
14 changes: 11 additions & 3 deletions src/plugins/explore/public/application/in_context_vis_editor/component/top_nav.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,13 +70,21 @@ export const TopNav = ({ setHeaderActionMenu = () => {}, savedExplore }: TopNavP
if (payload?.dateRange) {
queryBuilder.updateQueryEditorState({ dateRange: payload.dateRange });
}
// update current query text
if (queryEditorState.editorMode !== EditorMode.Prompt) {
// update current query text only if on Query tab and not in Prompt mode
if (
queryEditorState.editorMode !== EditorMode.Prompt &&
queryEditorState.activeBottomPanelTab === 'QUERY_TAB'
) {
queryBuilder.updateQueryState({ query: getEditorText() });
}
await queryBuilder.onQueryExecutionSubmit();
},
[queryBuilder, queryEditorState.editorMode, getEditorText]
[
queryBuilder,
queryEditorState.editorMode,
queryEditorState.activeBottomPanelTab,
getEditorText,
]
);

const handleQueryCancel = useCallback(() => {
Expand Down
25 changes: 24 additions & 1 deletion ...ation/in_context_vis_editor/component/visualization_editor_bottom_left_container.test.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,6 @@
* SPDX-License-Identifier: Apache-2.0
*/

import React from 'react';
import { render, screen } from '@testing-library/react';
import { ResizableQueryPanelAndVisualization } from './visualization_editor_bottom_left_container';
import { QueryExecutionStatus } from '../../utils/state_management/types';
Expand All @@ -12,6 +11,26 @@ import { useVisualizationBuilder } from '../hooks/use_visualization_builder';

jest.mock('../hooks/use_query_builder_state', () => ({ useQueryBuilderState: jest.fn() }));
jest.mock('../hooks/use_visualization_builder', () => ({ useVisualizationBuilder: jest.fn() }));

jest.mock('../../../components/data_transformations', () => {
const mockObs = {
subscribe: jest.fn().mockReturnValue({ unsubscribe: jest.fn() }),
getValue: jest.fn().mockReturnValue([]),
};
const mockMapObs = {
subscribe: jest.fn().mockReturnValue({ unsubscribe: jest.fn() }),
getValue: jest.fn().mockReturnValue(new Map()),
};
return {
useTransformationService: jest.fn().mockReturnValue({
clearPipeline: jest.fn(),
pipeline$: mockObs,
getPipeline$: () => mockObs,
stageSchemas$: mockMapObs,
}),
TransformPanel: () => null,
};
});
jest.mock('../../../components/query_panel/utils/use_search_context', () => ({
useSearchContext: jest.fn().mockReturnValue({}),
}));
Expand Down Expand Up @@ -44,8 +63,12 @@ jest.mock('@osd/i18n', () => ({
}));

const mockQueryEditorState$ = { getValue: jest.fn() };
const mockResultState$ = { subscribe: jest.fn().mockReturnValue({ unsubscribe: jest.fn() }) };
const mockQueryBuilder = {
queryEditorState$: mockQueryEditorState$,
resultState$: mockResultState$,
updateQueryEditorState: jest.fn(),
setOnDatasetChanged: jest.fn(),
};

const buildState = (options: Record<string, any>) => ({
Expand Down
175 changes: 126 additions & 49 deletions ...pplication/in_context_vis_editor/component/visualization_editor_bottom_left_container.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,8 @@
* SPDX-License-Identifier: Apache-2.0
*/

import React, { useEffect, useCallback } from 'react';
import React, { useEffect, useCallback, useState } from 'react';
import { useObservable } from 'react-use';
import moment from 'moment';
import { i18n } from '@osd/i18n';
import {
Expand All @@ -12,6 +13,9 @@ import {
EuiTitle,
EuiErrorBoundary,
EuiIcon,
EuiTabs,
EuiTab,
EuiPanel,
} from '@elastic/eui';
import { TimeRange } from 'src/plugins/data/common';
import { QueryExecutionStatus } from '../../utils/state_management/types';
Expand All @@ -24,7 +28,15 @@ import { useQueryBuilderState } from '../hooks/use_query_builder_state';
import { ErrorCodeBlock } from '../../../components/tabs/error_guard/error_code_block';
import { EditorPanel } from './editor_panel';
import { useVisualizationBuilder } from '../hooks/use_visualization_builder';

import '../visualization_editor.scss';
import {
ITransformationService,
useTransformationService,
TransformPanel,
} from '../../../components/data_transformations';

type ActiveTab = 'QUERY_TAB' | 'TRANSFORM_TAB';

const errorDefaultTitle = i18n.translate('explore.errorPanel.defaultTitle', {
defaultMessage: 'An error occurred while executing the query',
Expand All @@ -36,9 +48,36 @@ const typeText = i18n.translate('explore.errorPanel.type', {
defaultMessage: 'Type',
});

const queryTabLabel = i18n.translate('explore.bottomPanel.queryTab', {
defaultMessage: 'Query',
});
const transformTabLabel = i18n.translate('explore.bottomPanel.transformTab', {
defaultMessage: 'Transform',
});

export const ResizableQueryPanelAndVisualization = () => {
const { queryBuilder, queryEditorState } = useQueryBuilderState();
const queryStatus = queryEditorState.queryStatus;
const [activeTab, setActiveTab] = useState<ActiveTab>(
queryEditorState.activeBottomPanelTab ?? 'QUERY_TAB'
);
const { visualizationBuilderForEditor } = useVisualizationBuilder();

const transformServices = useTransformationService(visualizationBuilderForEditor, {
onPipelineChange: useCallback(() => {
queryBuilder.updateQueryEditorState({ isQueryEditorDirty: true });
}, [queryBuilder]),
});

useEffect(() => {
queryBuilder.setOnDatasetChanged(() => {
transformServices.clearPipeline();
});
}, [queryBuilder, transformServices]);

useEffect(() => {
queryBuilder.updateQueryEditorState({ activeBottomPanelTab: activeTab });
}, [activeTab, queryBuilder]);

const renderVis = () => {
if (queryStatus.status === QueryExecutionStatus.NO_RESULTS) {
Expand Down Expand Up @@ -88,21 +127,52 @@ export const ResizableQueryPanelAndVisualization = () => {
</EuiErrorBoundary>
);
}
return <VisualizationContainer />;
return <VisualizationContainer transformationService={transformServices} />;
};
return (
<EuiResizableContainer direction="vertical">
{(EuiResizablePanel, EuiResizableButton) => {
return (
<>
<EuiResizablePanel initialSize={70} minSize="60%" paddingSize="none">
<EuiResizablePanel initialSize={70} minSize="0%" paddingSize="none">
{renderVis()}
</EuiResizablePanel>

<EuiResizableButton />

<EuiResizablePanel initialSize={30} minSize="20%" paddingSize="none" hasBorder={false}>
<QueryPanel queryEditorState$={queryBuilder.queryEditorState$} />
<EuiPanel
paddingSize="none"
className="multiTabsPanel"
style={{
height: '100%',
display: 'flex',
flexDirection: 'column',
}}
>
<EuiTabs size="s">
<EuiTab
isSelected={activeTab === 'QUERY_TAB'}
onClick={() => setActiveTab('QUERY_TAB')}
data-test-subj="queryPanelTab"
>
{queryTabLabel}
</EuiTab>
<EuiTab
isSelected={activeTab === 'TRANSFORM_TAB'}
onClick={() => setActiveTab('TRANSFORM_TAB')}
data-test-subj="transformPanelTab"
>
{transformTabLabel}
</EuiTab>
</EuiTabs>

{activeTab === 'QUERY_TAB' ? (
<QueryPanel queryEditorState$={queryBuilder.queryEditorState$} />
) : (
<TransformPanel transformationService={transformServices} />
)}
</EuiPanel>
</EuiResizablePanel>
</>
);
Expand All @@ -111,52 +181,59 @@ export const ResizableQueryPanelAndVisualization = () => {
);
};

export const VisualizationContainer = React.memo(() => {
const searchContext = useSearchContext();
export const VisualizationContainer = React.memo(
({ transformationService }: { transformationService: ITransformationService }) => {
const searchContext = useSearchContext();

const { visualizationBuilderForEditor: visualizationBuilder } = useVisualizationBuilder();
const { resultState: results, queryBuilder } = useQueryBuilderState();
const { visualizationBuilderForEditor: visualizationBuilder } = useVisualizationBuilder();
const { queryBuilder } = useQueryBuilderState();

useEffect(() => {
if (results) {
const rows = results.hits?.hits || [];
const fieldSchema = results.fieldSchema || [];
const resultState = useObservable(queryBuilder.resultState$, undefined);

visualizationBuilder.handleData(rows, fieldSchema);
}
}, [visualizationBuilder, results]);

const onSelectTimeRange = useCallback(
(timeRange?: TimeRange) => {
if (timeRange) {
queryBuilder.updateQueryEditorState({
dateRange: {
from: moment(timeRange.from).toISOString(),
to: moment(timeRange.to).toISOString(),
},
});

queryBuilder.clearResultState();
queryBuilder.updateQueryEditorState({
queryStatus: {
status: QueryExecutionStatus.UNINITIALIZED,
elapsedMs: undefined,
startTime: undefined,
error: undefined,
},
});
queryBuilder.executeQuery();
}
},
[queryBuilder]
);
const pipeline = useObservable(
transformationService.getPipeline$(),
transformationService.pipeline$.getValue()
);

return (
<EditorPanel>
{visualizationBuilder.renderVisualization({
timeRange: searchContext?.timeRange,
onSelectTimeRange,
})}
</EditorPanel>
);
});
useEffect(() => {
if (!resultState) return;
const rows = resultState.hits?.hits || [];
const fieldSchema = resultState.fieldSchema || [];
visualizationBuilder.handleData(rows, fieldSchema);
}, [visualizationBuilder, resultState, pipeline]);

const onSelectTimeRange = useCallback(
(timeRange?: TimeRange) => {
if (timeRange) {
queryBuilder.updateQueryEditorState({
dateRange: {
from: moment(timeRange.from).toISOString(),
to: moment(timeRange.to).toISOString(),
},
});

queryBuilder.clearResultState();
queryBuilder.updateQueryEditorState({
queryStatus: {
status: QueryExecutionStatus.UNINITIALIZED,
elapsedMs: undefined,
startTime: undefined,
error: undefined,
},
});
queryBuilder.executeQuery();
}
},
[queryBuilder]
);

return (
<EditorPanel>
{visualizationBuilder.renderVisualization({
timeRange: searchContext?.timeRange,
onSelectTimeRange,
})}
</EditorPanel>
);
}
);
3 changes: 2 additions & 1 deletion ...e/public/application/in_context_vis_editor/component/visualization_editor_query_panel.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ import { QueryExecutionStatus } from '../../utils/state_management/types';
import { QueryPanelWidgets } from './query_panel_widget';
import { QueryPanelEditor } from './query_editor';
import { QueryPanelGeneratedQuery } from './generated_query_panel';
import '../visualization_editor.scss';

export const QueryPanel = ({
queryEditorState$,
Expand All @@ -27,7 +28,7 @@ export const QueryPanel = ({
<EuiPanel
paddingSize="s"
borderRadius="none"
className="exploreInContexQueryPanel"
className="visualizationEditorTabPanel"
style={{ height: '100%' }}
>
<QueryPanelWidgets />
Expand Down
Loading
Loading