[FIX] Add NULL checks for fopen() and alloc_demuxer_data() in process_hex()

[FIX] Add NULL checks for fopen() and alloc_demuxer_data() in process_hex() #2283