add support to azure data lake gen 2 storage

[Anhui-tqhuang]

Related issue:

• Support hierarchical namespaced Azure storage accounts grafana/loki#19903

Azure Data Lake Gen 2 Storage is a bit different from the pervious azure object storage account, inside a filesystem (container), azure recognize a directory as a file as well.

Given the following example, assume we have the following files on storage:

a/b/c
a/b/d
e/f
g

When we list objects on old storage account, we will get:

a/b/c
a/b/d
e/f
g

When we list objects on azure data lake gen 2, we will get:

a
a/b
a/b/c
a/b/d
e
e/f
g

Note

name to directory in azure data lake gen 2 doesn't have the suffix /

as a result, when loki or other components which import thanos object storage package, they will meet errors like when using azure data lake gen 2

level=error ts=2025-11-18T04:07:17.55822398Z caller=index_set.go:308 table-name=index_20409 msg="sync failed, retrying it" err="open /var/loki/index_cache/index_20409/fake: is a directory"
open /var/loki/index_cache/index_20409/fake: is a directory
error initialising module: store
github.com/grafana/dskit/modules.(*Manager).initModule
	/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:138
github.com/grafana/dskit/modules.(*Manager).InitModuleServices
	/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:108
github.com/grafana/loki/v3/pkg/loki.(*Loki).Run
	/src/loki/pkg/loki/loki.go:531
main.main
	/src/loki/cmd/loki/main.go:129
runtime.main
	/usr/local/go/src/runtime/proc.go:283
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1700
level=info ts=2025-11-18T04:07:17.567952634Z caller=util.go:77 table-name=index_20409 file-name=fake msg="downloaded file" total_time=9.212318ms
level=error ts=2025-11-18T04:07:17.568410124Z caller=index_set.go:308 table-name=index_20409 msg="sync failed, retrying it" err="open /var/loki/index_cache/index_20409/fake: is a directory"
level=error ts=2025-11-18T04:07:17.56842428Z caller=index_set.go:104 table-name=index_20409 msg="failed to initialize table, cleaning it up" table=index_20409 err="open /var/loki/index_cache/index_20409/fake: is a directory"
level=error ts=2025-11-18T04:07:17.568892675Z caller=log.go:223 msg="error running loki" err="open /var/loki/index_cache/index_20409/fake: is a directory\nerror initialising module: store\ngithub.com/grafana/dskit/modules.(*Manager).initModule\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:138\ngithub.com/grafana/dskit/modules.(*Manager).InitModuleServices\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:108\ngithub.com/grafana/loki/v3/pkg/loki.(*Loki).Run\n\t/src/loki/pkg/loki/loki.go:531\nmain.main\n\t/src/loki/cmd/loki/main.go:129\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:283\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1700"

cc @bwplotka @fpetkovski