INTERNAL Error when linking a custom built including iceberg, aws and httpfs

[mlafeldt]

Note

This issue was moved from marcboeker#433

Original issue by @EthanBlackburn

I built duckdb on main (a35d8b6d5ba121f2601b917e1a7732207c944ae5) using these commands

mkdir -p build/release && cd build/release

cmake ../.. \
  -DCMAKE_BUILD_TYPE=Release \
  -DVCPKG_BUILD=1 \
  -DCMAKE_TOOLCHAIN_FILE=$HOME/vcpkg/scripts/buildsystems/vcpkg.cmake \
  -DDUCKDB_EXTENSION_CONFIGS="$HOME/duckdb/extension_config.cmake" \
  -DAVRO_INCLUDE_DIR=$HOME/duckdb/vcpkg_installed/arm64-osx/include  -DAVRO_LIBRARY=$HOME/duckdb/vcpkg_installed/arm64-osx/lib/libavro.a \
  -DENABLE_EXTENSION_AUTOLOADING=1 \
  -DENABLE_EXTENSION_AUTOINSTALL=1 -DEXTENSION_STATIC_BUILD=1

make -j8

The build succeeds, and I can run the built duckdb binary just fine.

I then followed the directions in README.md to link duckdb dynamically

CGO_ENABLED=1 CGO_LDFLAGS="-lduckdb -L/Users/ethanblackburn/duckdb/build/release/src/" go build -tags=duckdb_use_lib .

DYLD_LIBRARY_PATH=/Users/ethanblackburn/duckdb/build/release/src ./mybinary

I have this code to setup the db

var execDir string
execDir, err = os.Executable()
if err != nil {
			return
}
execDir = path.Dir(execDir)

// Create a directory that should be writable within the sandbox
duckDBDir := path.Join(execDir, "duckdb_data")
err = os.MkdirAll(duckDBDir, 0755)
if err != nil {
	return
}

client, err = sqlx.Connect("duckdb", "")
if err != nil {
	return
}
client.Mapper = reflectx.NewMapperTagFunc("json", nil, nil)

_, err = client.Exec(fmt.Sprintf(`
			SET home_directory='%s'; 
			SET extension_directory='%s';

			LOAD iceberg; 
			LOAD aws;
			LOAD httpfs;

			CREATE SECRET (
				TYPE s3,
				PROVIDER credential_chain
			);

			ATTACH '%s'
			AS s3_tables_db (
				TYPE iceberg,
				ENDPOINT_TYPE s3_tables
			);

			USE s3_tables_db.debug_data;
		`, duckDBDir, duckDBDir, bucketArn))
		if err != nil {
			return
		}

rows, err := dbClient.Query("select count(*) from my_table;")
	if err != nil {
		fmt.Fprintf(os.Stderr, "Failed to query database: %v", err)
		os.Exit(1)
	}

The query runs into this error

Failed to query database: INTERNAL Error: optional_idx cannot be initialized with an invalid index

Stack Trace:

0        duckdb::Exception::Exception(duckdb::ExceptionType, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) + 64
1        duckdb::InternalException::InternalException(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) + 20
2        duckdb::optional_idx::optional_idx(unsigned long long) + 96
3        duckdb::ThreadContext::ThreadContext(duckdb::ClientContext&) + 124
4        duckdb::AvroScan::AvroScan(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, duckdb::ClientContext&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) + 1084
5        duckdb::TemplatedUniqueIf<duckdb::AvroScan, true>::templated_unique_single_t duckdb::make_uniq<duckdb::AvroScan, char const (&) [16], duckdb::ClientContext&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>&>(char const (&) [16], duckdb::ClientContext&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>&) + 192
6        duckdb::IcebergMultiFileList::InitializeFiles(std::__1::lock_guard<std::__1::mutex>&) + 1416
7        duckdb::IcebergMultiFileList::Bind(duckdb::vector<duckdb::LogicalType, true>&, duckdb::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, true>&) + 164
8        duckdb::IcebergMultiFileReader::Bind(duckdb::MultiFileOptions&, duckdb::MultiFileList&, duckdb::vector<duckdb::LogicalType, true>&, duckdb::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, true>&, duckdb::MultiFileReaderBindData&) + 88
9        duckdb::MultiFileFunction<duckdb::ParquetMultiFileInfo>::MultiFileBindInternal(duckdb::ClientContext&, duckdb::unique_ptr<duckdb::MultiFileReader, std::__1::default_delete<duckdb::MultiFileReader>, true>, duckdb::shared_ptr<duckdb::MultiFileList, true>, duckdb::vector<duckdb::LogicalType, true>&, duckdb::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, true>&, duckdb::MultiFileOptions, duckdb::unique_ptr<duckdb::BaseFileReaderOptions, std::__1::default_delete<duckdb::BaseFileReaderOptions>, true>) + 676
10       duckdb::MultiFileFunction<duckdb::ParquetMultiFileInfo>::MultiFileBind(duckdb::ClientContext&, duckdb::TableFunctionBindInput&, duckdb::vector<duckdb::LogicalType, true>&, duckdb::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, true>&) + 688
11       duckdb::ICTableEntry::GetScanFunction(duckdb::ClientContext&, duckdb::unique_ptr<duckdb::FunctionData, std::__1::default_delete<duckdb::FunctionData>, true>&) + 4544
12       duckdb::Binder::Bind(duckdb::BaseTableRef&) + 1096
13       duckdb::Binder::Bind(duckdb::TableRef&) + 100
14       duckdb::Binder::BindNode(duckdb::SelectNode&) + 68
15       duckdb::Binder::BindNode(duckdb::QueryNode&) + 140
16       duckdb::Binder::Bind(duckdb::QueryNode&) + 176
17       duckdb::Planner::CreatePlan(duckdb::SQLStatement&) + 152
18       duckdb::ClientContext::CreatePreparedStatementInternal(duckdb::ClientContextLock&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, duckdb::unique_ptr<duckdb::SQLStatement, std::__1::default_delete<duckdb::SQLStatement>, true>, duckdb::optional_ptr<std::__1::unordered_map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, duckdb::BoundParameterData, duckdb::CaseInsensitiveStringHashFunction, duckdb::CaseInsensitiveStringEquality, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const, duckdb::BoundParameterData>>>, true>) + 512
19       duckdb::ClientContext::CreatePreparedStatement(duckdb::ClientContextLock&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, duckdb::unique_ptr<duckdb::SQLStatement, std::__1::default_delete<duckdb::SQLStatement>, true>, duckdb::optional_ptr<std::__1::unordered_map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, duckdb::BoundParameterData, duckdb::CaseInsensitiveStringHashFunction, duckdb::CaseInsensitiveStringEquality, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const, duckdb::BoundParameterData>>>, true>, duckdb::PreparedStatementMode) + 1024
20       std::__1::__function::__func<duckdb::ClientContext::PrepareInternal(duckdb::ClientContextLock&, duckdb::unique_ptr<duckdb::SQLStatement, std::__1::default_delete<duckdb::SQLStatement>, true>)::$_0, std::__1::allocator<duckdb::ClientContext::PrepareInternal(duckdb::ClientContextLock&, duckdb::unique_ptr<duckdb::SQLStatement, std::__1::default_delete<duckdb::SQLStatement>, true>)::$_0>, void ()>::operator()() + 60
21       duckdb::ClientContext::RunFunctionInTransactionInternal(duckdb::ClientContextLock&, std::__1::function<void ()> const&, bool) + 132
22       duckdb::ClientContext::PrepareInternal(duckdb::ClientContextLock&, duckdb::unique_ptr<duckdb::SQLStatement, std::__1::default_delete<duckdb::SQLStatement>, true>) + 212
23       duckdb::ClientContext::Prepare(duckdb::unique_ptr<duckdb::SQLStatement, std::__1::default_delete<duckdb::SQLStatement>, true>) + 196
24       duckdb::Connection::Prepare(duckdb::unique_ptr<duckdb::SQLStatement, std::__1::default_delete<duckdb::SQLStatement>, true>) + 52
25       duckdb_prepare_extracted_statement + 148
26       _cgo_529715d91f62_Cfunc_duckdb_prepare_extracted_statement + 40
27       runtime.asmcgocall.abi0 + 124

This error signals an assertion failure within DuckDB. This usually occurs due to unexpected conditions or errors in the program's logic.
For more information, see https://duckdb.org/docs/stable/dev/internal_errors%

Is the problem that I'm trying to use a 1.3.0 duckdb build?

Original comments

@taniabogatsch on 2025-05-07T10:53:19Z

I guess that a35d8b6d5ba121f2601b917e1a7732207c944ae5 is a nightly build, and the extensions you're installing correspond to that build?

In general, an InternalError (and the stack trace) don't look like a go-duckdb error.

The build succeeds, and I can run the built duckdb binary just fine.

Can you reproduce this InternalError in the CLI (of your binary, i.e., without go-duckdb)? If so, you likely want to head over to duckdb/duckdb to file this issue there.

@EthanBlackburn on 2025-05-07T18:41:14Z

Yes, its a nightly build and the extensions were built for that commit.

I haven't been able to reproduce it in the CLI :/ It seems like an issue with prepared statements but Im not totally sure. I'll set some break points in duckdb to see if I can get any further on the investigation

@taniabogatsch on 2025-05-08T10:28:37Z

Sounds good. There's also a DuckDB release coming up, and go-duckdb will move to that release with the extensions build for it. That might make it easier to reproduce this (including a reproducible code snippet), without the need for a custom build.

@taniabogatsch on 2025-05-14T13:03:29Z

I've tagged this as requires latest duckdb for now. 👍

@taniabogatsch on 2025-09-18T16:34:21Z

Hi @EthanBlackburn - any update on this with the latest release? :)