I think you need to add dns_server_stop() inside wifi_manager_destroy() to properly stop and destroy the wifi_manager along with dns_server.