VFS - Connect on remote hdfs

[oudine13]

Hi,
I would like to create file transformation pipelines by downloading input files from remote HDFS and storing outputs file on the same remote HDFS. Kerberos is used to connect to my hadoop cluster.

On my local machine, I did the following steps:

• installed kerberos and obtained a key,
• set the environment variables (JAVA_HOME, HADOOP_CONF_DIR and HADOOP_OPTS).
• test read/write on remote HDFS with CLI command: hdfs dfs -ls hdfs://clusterhadoop/user => success
• add environment variables in hop/config/hop-config.json
• add hadoop-common-3.1.1.jar and hadoop-hdfs-3.1.1.jar files to the hdfs/lib directory created in hop/plugins/tech/hdfs/
• add hdfs in plugin explorer, with main class [org.apache.hadoop.fs.hdfs.HdfsFileSystem] and path to jar librairies above.

After all that, I still can't see the contents of my HDFS from the CSV File Input or TextFIle Input. I get this error:

Error browsing to location: 'hdfs://clusterhadoop/user/toto.csv'
FileNotFolderException: Could not list the contents of "file:///home/ubuntu/hop/hdfs:/clusterhadoop/user" because it is not a folder.
Root cause: FileNotFolderException: Could not list the contents of "file:///home/ubuntu/hop/hdfs:/clusterhadoop/user" because it is not a folder.

org.apache.commons.vfs2.FileNotFolderException: Could not list the contents of "file:///home/ubuntu/hop/hdfs:/clusterhadoop/user" because it is not a folder.
	at org.apache.commons.vfs2.provider.AbstractFileObject.getChildren(AbstractFileObject.java:1107)
	at org.apache.hop.ui.core.vfs.HopVfsFileDialog.populateFolder(HopVfsFileDialog.java:941)
	at org.apache.hop.ui.core.vfs.HopVfsFileDialog.refreshBrowser(HopVfsFileDialog.java:898)
	at org.apache.hop.ui.core.vfs.HopVfsFileDialog.refreshAll(HopVfsFileDialog.java:1278)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
	at org.apache.hop.ui.core.gui.BaseGuiWidgets.lambda$getListener$1(BaseGuiWidgets.java:213)
	at org.eclipse.swt.widgets.EventTable.sendEvent(EventTable.java:91)
	at org.eclipse.swt.widgets.Display.sendEvent(Display.java:5857)
	at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1617)
	at org.eclipse.swt.widgets.Display.runDeferredEvents(Display.java:5067)
	at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:4519)
	at org.apache.hop.ui.core.vfs.HopVfsFileDialog.open(HopVfsFileDialog.java:584)
	at org.apache.hop.ui.core.dialog.BaseDialog.presentFileDialog(BaseDialog.java:278)
	at org.apache.hop.ui.core.dialog.BaseDialog.presentFileDialog(BaseDialog.java:172)
	at org.apache.hop.pipeline.transforms.fileinput.text.TextFileInputDialog.lambda$open$8(TextFileInputDialog.java:484)
	at org.eclipse.swt.widgets.EventTable.sendEvent(EventTable.java:91)
	at org.eclipse.swt.widgets.Display.sendEvent(Display.java:5857)
	at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1617)
	at org.eclipse.swt.widgets.Display.runDeferredEvents(Display.java:5067)
	at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:4519)
	at org.apache.hop.ui.core.dialog.BaseDialog.defaultShellHandling(BaseDialog.java:528)
	at org.apache.hop.pipeline.transforms.fileinput.text.TextFileInputDialog.open(TextFileInputDialog.java:501)
	at org.apache.hop.ui.hopgui.file.pipeline.delegates.HopGuiPipelineTransformDelegate.editTransform(HopGuiPipelineTransformDelegate.java:195)
	at org.apache.hop.ui.hopgui.file.pipeline.HopGuiPipelineGraph.editTransform(HopGuiPipelineGraph.java:3368)
	at org.apache.hop.ui.hopgui.file.pipeline.HopGuiPipelineGraph.editTransform(HopGuiPipelineGraph.java:2210)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
	at org.apache.hop.core.gui.plugin.action.GuiActionLambdaBuilder.lambda$createLambda$0(GuiActionLambdaBuilder.java:97)
	at org.apache.hop.ui.hopgui.context.GuiContextUtil.lambda$handleActionSelection$0(GuiContextUtil.java:182)
	at org.eclipse.swt.widgets.RunnableLock.run(RunnableLock.java:40)
	at org.eclipse.swt.widgets.Synchronizer.runAsyncMessages(Synchronizer.java:132)
	at org.eclipse.swt.widgets.Display.runAsyncMessages(Display.java:5042)
	at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:4522)
	at org.apache.hop.ui.hopgui.HopGui.open(HopGui.java:478)
	at org.apache.hop.ui.hopgui.HopGui.main(HopGui.java:356)

Can anyone help me? Thanks in advance for your time.

[bamaer]

did you add your own HDFS plugin?
HDFS is supported by Hop VFS, which uses Apache Commons VFS. Also, there's no need to add hadoop jars afaik, they're already in lib/core.
Not sure if the Hop VFS implementation for HDFS works with Kerberos though...

[oudine13]

I added these 2 plugins hadoop-common-3.1.1.jar, hadoop-hdfs-3.1.1.jar in hop/plugins/tech/hdfs/lib. if it's wrong, could you tell me more, and explain how to proceed?

Even if kerberos is not supported by Hop VFS, i can create workflow where first action will run bash script with kinit.

thx

[oudine13]

I take a lot of time to test many configurations and read many documentations, eliminate potential errors from the kerberos configuration, by setting up a test connection to another HDFS without Kerberos authentication, but i've still been unable to resolve that. I still have the same output error. However when I tested another VFS file system types (http, ftp, gz, ...) Apache hop recognizes them and didn't prepend them with file///.

I start having a doubt on resolving this issue when i saw a youtube video that broadcast a discussion session between Apache hop developers and his community, when someone explained the same difficults than me to get and put files on hdfs. One of the Apache hop developers acknowledged that the documentation on configuration, the various options and the use of hdfs was not sufficient and said that it would be updated as has been done for other vfs like azure, aws... Even though the video is about 3 years old, there's nothing online or on the Apache hop website. No more recent communication on this subject.

I really really appreciate your help. Thank you.

[hansva]

I have taken a look at the code, this is where we register the supported filetypes. hdfs currently isn't included. It would require some extra development to enable hdfs. While the codechange might be minimal, testing it will require some more work.

[oudine13]

ok. thank you @hansva

[ZhaoRuidong]

Any progress?