feat: Add get_hdfs_by_full_path_with_config for configurable HDFS connections#27
feat: Add get_hdfs_by_full_path_with_config for configurable HDFS connections#27drexler-sky wants to merge 2 commits into
Conversation
|
@yahoNanJing Could you please take a look at this PR? Thanks! |
|
also cc @Kontinuation @comphead @parthchandra |
| pub fn connect_with_config( | ||
| nn: &str, | ||
| config: &HashMap<String, String>, | ||
| ) -> Result<Arc<Self>, HdfsErr> { |
There was a problem hiding this comment.
The main entry for getting an Arc<HdfsFs> is get_hdfs_by_full_path. It caches already connected file system handles and will reuse them when getting another HdfsFs using uri with cached host. HdfsFs handles were never closed, since the Rust binding does not call hdfsDisconnect at all.
The caller of connect_with_config has to deal with caching and reusing already connected HdfsFs, otherwise we'll leave lots of connected Hadoop FileSystem objects in the JVM.
There was a problem hiding this comment.
Thanks for your comment! I’ve updated the implementation to use get_hdfs_by_full_path_with_config, so it now follows the same caching logic as get_hdfs_by_full_path and reuses existing HdfsFs handles. Let me know if there’s anything else I should adjust.
drexler-sky
changed the title
|
Gentle ping @yahoNanJing, when you have a chance, could you please take a look at this? |
2 participants
This PR introduces a new
get_hdfs_by_full_path_with_configmethod to HdfsFs, enabling programmatic configuration of HDFS connections through a HashMap of key-value pairs.