Need help for a broken pipeline... probably after a Hop update

[dave-csc]

Hello,

I need your help to figure out what has been changed in Hop to make a pipeline of mine not working anymore.

The pipeline composition is attached at the end of the post, the archive contains 3 files:

• anti-normalizer.hpl, the "main" pipeline: it should provide a sort of emulation of the Row Normalizer transform, with the added value that you don't have to provide the full schema to the transform, but only the fields you would like to use for grouping
• _inj-anti-normalizer.hpl, the "injected" pipeline: this pipeline is injected with the data to normalize and the schema/metadata that is evaluated dinamically
• _test-anti-normalizer.hpl, the "testing" pipeline: it generates some fake data to demonstrate the functionality

This composition worked well in Hop 2.10: the output for the test case is correctly generated with a RecordID grouping column, and the pairs elementName and elementValue.

Starting from Hop 2.11 the same composition is broken. During its execution the following error, regarding the Data grid transform in the injected pipeline, is reported:

2025/04/03 16:28:03 - Data grid.0 - ERROR: Unexpected error
2025/04/03 16:28:03 - Data grid.0 - ERROR: java.lang.IndexOutOfBoundsException: Index 0 out of bounds for length 0
2025/04/03 16:28:03 - Data grid.0 - 	at java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:64)
2025/04/03 16:28:03 - Data grid.0 - 	at java.base/jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:70)
2025/04/03 16:28:03 - Data grid.0 - 	at java.base/jdk.internal.util.Preconditions.checkIndex(Preconditions.java:266)
2025/04/03 16:28:03 - Data grid.0 - 	at java.base/java.util.Objects.checkIndex(Objects.java:359)
2025/04/03 16:28:03 - Data grid.0 - 	at java.base/java.util.ArrayList.get(ArrayList.java:427)
2025/04/03 16:28:03 - Data grid.0 - 	at org.apache.hop.pipeline.transforms.datagrid.DataGrid.processRow(DataGrid.java:80)
2025/04/03 16:28:03 - Data grid.0 - 	at org.apache.hop.pipeline.transform.RunThread.run(RunThread.java:54)
2025/04/03 16:28:03 - Data grid.0 - 	at java.base/java.lang.Thread.run(Thread.java:833)
2025/04/03 16:28:03 - Data grid.0 - Finished processing (I=0, O=0, R=0, W=0, U=0, E=1)

What has been changed in Hop in order to make the pipeline fail, and what can I do to make a fix?

Hint for the search: I started developing this mini-project slightly before reporting issue #4525 (you can notice that similar components are involved). The issue has been solved, but probably has broken something else...

---

Attached files:
anti-normalizer.zip

[bamaer]

First thing I noticed is that your MDI transform points to ${HOP_CONFIG_FOLDER}/projects/_root/datanorm/_inj-anti-normalizer.hpl as the child pipeline. As the name implies, ${HOP_CONFIG_FOLDER} should be a folder to store configuration. Using it for project files is not what I would consider a best practice, if not just tricky.

Have you tried specifying a target file name for the pipeline after injection (Options -> Option target file (hpl after injection)). Do you see any issues in the data grid if you uncheck the Run resulting pipeline and open the generated pipeline after the injection?

[dave-csc]

Have you tried specifying a target file name for the pipeline after injection (Options -> Option target file (hpl after injection)). Do you see any issues in the data grid if you uncheck the Run resulting pipeline and open the generated pipeline after the injection?

Hi @bamaer,

I'll explain the path question below (I exported directly my pipelines, for your tests you need to tweak the paths in the "testing" and "main" pipeline of course), first I tried to do this test and a significant difference occurred in the resulting debug pipeline:

• with Hop 2.10 the Data grid transform is correctly populated, both in the "Meta" and the "Data" sections
• with Hop 2.12 only the "Meta" section is populated, the "Data" section contains 5 empty rows (as if their contents have not been injected)

First thing I noticed is that your MDI transform points to ${HOP_CONFIG_FOLDER}/projects/_root/datanorm/_inj-anti-normalizer.hpl as the child pipeline. As the name implies, ${HOP_CONFIG_FOLDER} should be a folder to store configuration. Using it for project files is not what I would consider a best practice, if not just tricky.

The ${HOP_CONFIG_FOLDER} has been moved away from its default location, so the configuration used is the same for all the installed versions of Hop. However, in the default location a project subfolder indeed exists, and seems to be intended to store project files (default and samples are there): so why it should be risky to keep a similar one off the default location?

Thanks for your support

[hansva]

Hi @dave-csc,

I found the issue. The root cause is that the second field in the data section for the data grid has to be filled in.
Trying to find a way to resolve this. But setting it to a constant "," will solve your issue.

This was added to make it possible to inject multiple columns to a datagrid.

[dave-csc]

Hi @hansva,

your suggestion worked! Thank you so much :)

Should I report this in the issues?

[hansva]

yes, feel free to create a ticket. I can't find an easy solution at the moment