The following demo illustrates a comparison of our method for GTA-V → Cityscapes with the inital GTA-V rendered appearance.
demo.mp4
- 09/06/2026: Added code for exporting the models into ONNX format. Added sample code for inference through ONNX Runtime. Added instructions for integrading the models into Unreal Engine 5.
Generative models are widely employed to enhance the photorealism of synthetic data for training computer vision algorithms. However, they often introduce visual artifacts that degrade the accuracy of these algorithms and require high computational resources, limiting their applicability in real-time training or evaluation scenarios. In this letter, we propose Hybrid Patch Enhanced Realism Generative Adversarial Network (HyPER-GAN), a lightweight image-to-image translation framework based on a U-Net–style generator designed for real-time inference. The model is trained using paired synthetic and photorealism-enhanced images, complemented by a hybrid training strategy that incorporates matched patches from real-world data to improve visual realism and semantic consistency. Experimental results demonstrate that HyPER-GAN outperforms state-of-the-art paired image-to-image translation methods in terms of inference latency, visual realism, and semantic robustness. Moreover, it is illustrated that the proposed hybrid training strategy indeed improves visual quality and semantic consistency compared to training the model solely with paired synthetic and photorealism-enhanced images.
If you used HyPER-GAN or any of the pretrained models from this repository in a scientific publication, we would appreciate using the following citation:
@misc{pasios2026hyperganhybridpatchbasedimagetoimage,
title={HyPER-GAN: Hybrid Patch-Based Image-to-Image Translation for Real-Time Photorealism Enhancement},
author={Stefanos Pasios and Nikos Nikolaidis},
year={2026},
eprint={2603.10604},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2603.10604},
}
pip install numpy torch torchvision pillow tqdm pygame
pip install onnxruntime-gpu
conda install -c conda-forge faiss-gpu
To train the model, it is required to have access to a synthetic dataset generated by a game or simulator, the corresponding images that were photorealism-enhanced by a robust unpaired image-to-image translation method such as Enhancing Photorealism Enhancement (EPE), as well as the real-world images employed during the robust unpaired image-to-image translation phase.
To train a model that enhances the photorealism of the CARLA simulator towards the characteristics of real-world datasets (Mapillary Vistas, Cityscapes, and KITTI), we already provide both the original rendered frames and the results of EPE here.
To train a model that enhances the photorealism of GTA-V towards the characteristics of real-world datasets (Mapillary Vistas and Cityscapes), the results of EPE are already provided by the authors at the official repository. The initial rendered GTA-V frames can be found in the Playing for Data dataset, which can be downloaded here.
The real-world datasets can be acquired from public benchmarks such as Cityscapes, KITTI, and Mapillary Vistas.
After collecting the required datasets, the following command can be executed to start the training process:
python main.py --mode train --rendered <path-to-rendered-dir> --enhanced <path-to-enhanced-dir> --real <path-to-real-dir>where <path-to-rendered-dir> contains the rendered images, <path-to-enhanced-dir> the corresponding photorealism-enhanced images produced by the robust unpaired image-to-image translation model (with the same names), and <path-to-real-dir> the real-world images.
To test HyPER-GAN, we provide pretrained models for GTA-V → Cityscapes and GTA-V → Mapillary Vistas inside the ./pretrained_models. To execute the testing process on a directory with images, use the following command:
python main.py --mode infer --input ./test_images --ckpt ./pretrained_models/gta2cs.pth --out ./outputTo execute the testing process on a video, use the following command:
python video_test.py --input_video ./test_videos/001.mp4 --output_dir ./ --ckpt ./pretrained_models/gta2cs.pthWe provide the carla_hypergan.py file, which can be employed to test the HyPER-GAN models on the output of the CARLA Unreal Engine 5 Simulator. First, install the CARLA 0.10.0 executable from here. Then, install the carla 0.10.0 python package located in PythonAPI/carla/dist. Finally, open the CARLA executable and subsequently execute the following command:
python carla_hypergan.py --ckpt ./pretrained_models/gta2cs.pth --width 1280 --height 720📝 Note: The vehicle can be moved using the WSAD keys.
In order to easily integrate the models into your own pipelines, we provide code for transforming the models into the widely used for deployment ONNX format. To export a model into ONNX, run the following command:
%example command
python hypergan_onnx_eport.py --input <path-to>\gta2cs.pth --output <path-to>\gta2cs.onnx --height 1080 --width 1920A sample script is also provided in onnx_utils/test_onnx.py to understand the preprocessing as well as the postprocessing steps that are required for inference with ONNX Runtime. To test the exported ONNX model on an image, run the following command:
%example command
python test_onnx.py --onnx <path-to>/gta2cs.onnx --image <path-to>/image.jpg --height 1080 --width 1920 --output <path-to>/output.jpgWith the release of the Unreal Engine 5 version 5.4 and above the engine now supports the real-time integration of neural rendering models through ONNX runtime. The integration requires no more than 7-8 minutes following the video tutorial: see the tutorial here. Below, we provide the exact preprocessing and postprocessing steps that should be applied to the post-processing material:
📝 Note: In Unreal Engine, the GPU will have to render both the engine's synthetic environment and run the model. At a resolution of 1920x1080, an RTX 4090 can achieve above 20 FPS when integrating HyPER-GAN into Unreal Engine 5.