[FIX] Fix unchecked fopen() and alloc_demuxer_data() return values in process_hex()

[FIX] Fix unchecked fopen() and alloc_demuxer_data() return values in process_hex() #2304