An error occurred when the daemon was connected to hdfs ha

[bright-zy]

(you don't have to strictly follow this form)

Bug Report

Briefly describe the bug

升级到0.4.0后 Daemon 开始报错，其他server worker 都读写正常

2024.05.17 11:54:09.007005 [ 36893 ] {} <Error> CnchRefreshMaterializedView: std::unordered_map<UUID, StorageID> DB::DaemonManager::getUUIDsFromCatalog(DB::DaemonManager::DaemonJobServerBGThread &): Code: 210, e.displayText() = DB::Exception: Unable to connect to HDFS: InvalidParameter: Cannot parse URI: hdfs://ns-fed, missing port or invalid HA configuration        Caused by: HdfsConfigNotFound: Config key: dfs.ha.namenodes.ns-fed not found SQLSTATE: 08000, Stack trace (when copying this message, always include the lines below):

0. Poco::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x26f45d32 in /data1/byconity/clickhouse
1. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, bool) @ 0x105d16e0 in /data1/byconity/clickhouse
2. DB::createHDFSFS(hdfsBuilder*) @ 0x20080bd2 in /data1/byconity/clickhouse
3. DB::HDFSFileSystem::HDFSFileSystem(DB::HDFSConnectionParams const&, int, int, int) @ 0x20072d4a in /data1/byconity/clickhouse
4. DB::DiskByteHDFS::DiskByteHDFS(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::HDFSConnectionParams const&) @ 0x20086882 in /data1/byconity/clickhouse
5. std::__1::shared_ptr<DB::DiskByteHDFS> std::__1::allocate_shared<DB::DiskByteHDFS, std::__1::allocator<DB::DiskByteHDFS>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, DB::HDFSConnectionParams&, void>(std::__1::allocator<DB::DiskByteHDFS> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, DB::HDFSConnectionParams&) @ 0x2008a2c7 in /data1/byconity/clickhouse
6. std::__1::shared_ptr<DB::IDisk> std::__1::__function::__policy_invoker<std::__1::shared_ptr<DB::IDisk> (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, Poco::Util::AbstractConfiguration const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::shared_ptr<DB::Context const>)>::__call_impl<std::__1::__function::__default_alloc_func<DB::registerDiskByteHDFS(DB::DiskFactory&)::$_0, std::__1::shared_ptr<DB::IDisk> (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, Poco::Util::AbstractConfiguration const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::shared_ptr<DB::Context const>)> >(std::__1::__function::__policy_storage const*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, Poco::Util::AbstractConfiguration const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::shared_ptr<DB::Context const>&&) @ 0x200889e7 in /data1/byconity/clickhouse
7. DB::DiskFactory::create(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, Poco::Util::AbstractConfiguration const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::shared_ptr<DB::Context const>) const @ 0x206fdf3f in /data1/byconity/clickhouse
8. DB::DiskSelector::DiskSelector(Poco::Util::AbstractConfiguration const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::shared_ptr<DB::Context const>) @ 0x20702bf2 in /data1/byconity/clickhouse
9. void std::__1::allocator<DB::DiskSelector>::construct<DB::DiskSelector, Poco::Util::AbstractConfiguration const&, char const* const&, std::__1::shared_ptr<DB::Context const> >(DB::DiskSelector*, Poco::Util::AbstractConfiguration const&, char const* const&, std::__1::shared_ptr<DB::Context const>&&) @ 0x208a0cd9 in /data1/byconity/clickhouse
10. DB::Context::getDiskSelector(std::__1::lock_guard<std::__1::mutex>&) const @ 0x20853b95 in /data1/byconity/clickhouse
11. DB::Context::getStoragePolicySelector(std::__1::lock_guard<std::__1::mutex>&) const @ 0x2083a21e in /data1/byconity/clickhouse
12. DB::Context::getStoragePolicy(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) const @ 0x20847f8e in /data1/byconity/clickhouse
13. DB::StorageCnchMergeTree::getStoragePolicy(DB::IStorage::StorageLocation) const @ 0x217521ee in /data1/byconity/clickhouse
14. DB::StorageCnchMergeTree::StorageCnchMergeTree(DB::StorageID const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::StorageInMemoryMetadata const&, bool, std::__1::shared_ptr<DB::Context>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::MergeTreeMetaBase::MergingParams const&, std::__1::unique_ptr<DB::MergeTreeSettings, std::__1::default_delete<DB::MergeTreeSettings> >) @ 0x21751824 in /data1/byconity/clickhouse
15. DB::create(DB::StorageFactory::Arguments const&) @ 0x21e8a0ac in /data1/byconity/clickhouse
16. DB::StorageFactory::get(DB::ASTCreateQuery const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::shared_ptr<DB::Context>, std::__1::shared_ptr<DB::Context>, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, DB::ForeignKeysDescription const&, DB::UniqueNotEnforcedDescription const&, bool, std::__1::unique_ptr<DB::StorageFactory::HiveParams, std::__1::default_delete<DB::StorageFactory::HiveParams> >, bool) const @ 0x217aaead in /data1/byconity/clickhouse
17. DB::createStorageFromQuery(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::shared_ptr<DB::Context const> const&) @ 0x1fe2fe7d in /data1/byconity/clickhouse
18. DB::Catalog::CatalogFactory::getTableByDefinition(std::__1::shared_ptr<DB::Context const> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) @ 0x1fdab059 in /data1/byconity/clickhouse
19. DB::DaemonManager::constructStorageTrait(std::__1::shared_ptr<DB::Context>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) @ 0x22c409be in /data1/byconity/clickhouse
20. DB::DaemonManager::getUUIDsFromCatalog(DB::DaemonManager::DaemonJobServerBGThread&) @ 0x22c3f420 in /data1/byconity/clickhouse
21. DB::DaemonManager::DaemonJobServerBGThread::executeImpl() @ 0x22c494c6 in /data1/byconity/clickhouse
22. DB::DaemonManager::DaemonJob::execute() @ 0x22c29103 in /data1/byconity/clickhouse
23. DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x20229ffe in /data1/byconity/clickhouse
24. DB::BackgroundSchedulePool::threadFunction() @ 0x2022c3a7 in /data1/byconity/clickhouse
25. void std::__1::__function::__policy_invoker<void ()>::__call_impl<std::__1::__function::__default_alloc_func<ThreadFromGlobalPool::ThreadFromGlobalPool<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, unsigned long, char const*, std::__1::shared_ptr<DB::CpuSet>)::$_1>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, unsigned long, char const*, std::__1::shared_ptr<DB::CpuSet>)::$_1&&)::'lambda'(), void ()> >(std::__1::__function::__policy_storage const*) @ 0x2022cbb7 in /data1/byconity/clickhouse
26. ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>) @ 0x1060dc00 in /data1/byconity/clickhouse
27. void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda0'()> >(void*) @ 0x1061207a in /data1/byconity/clickhouse
28. start_thread @ 0x7e25 in /usr/lib64/libpthread-2.17.so
29. __clone @ 0xfebad in /usr/lib64/libc-2.17.so

Version

0.4.0

[bright-zy]

先执行 export LIBHDFS3_CONF='path to hdfs site' 在启动 dm

[kevinthfang]

现在还有这个问题么？

[bright-zy]

@kevinthfang 0.4.0 版本还存在

[frankye1982]

Regarding this issue, I have already submitted a PR.