Noise in RGB Images During Imitation Learning

[yvfengZhong]

Hello,

I am currently using your code for imitation learning and have encountered an issue where the RGB images occasionally contain noise. This occurs while I am saving the RGB images captured by the robot.

Environment Configuration:

STRETCH_ENV_ARGS = dict(
gridSize=AGENT_MOVEMENT_CONSTANT
* 0.75, # Intentionally make this smaller than AGENT_MOVEMENT_CONSTANT to improve fidelity
width=INTEL_CAMERA_WIDTH,
height=INTEL_CAMERA_HEIGHT,
visibilityDistance=MAXIMUM_DISTANCE_ARM_FROM_AGENT_CENTER,
visibilityScheme="Distance",
fieldOfView=INTEL_VERTICAL_FOV,
server_class=ai2thor.fifo_server.FifoServer,
useMassThreshold=True,
massThreshold=10,
autoSimulation=False,
autoSyncTransforms=True,
renderInstanceSegmentation= True,
agentMode="stretch",
renderDepthImage=SAVE_DEPTH,
cameraNearPlane=0.01, # VERY VERY IMPORTANT
branch=None, # IMPORTANT do not use branch
commit_id=STRETCH_COMMIT_ID,
server_timeout=MAXIMUM_SERVER_TIMEOUT,
snapToGrid=False,
fastActionEmit=True,
action_hook_runner=_ACTION_HOOK_RUNNER,
# antiAliasing="smaa", # We can get nicer looking videos if we turn on antiAliasing and change the quality
# quality="Ultra",
)

controller_args={
**STRETCH_ENV_ARGS,
"platform": (
ai2thor.platform.OSXIntel64
if sys.platform.lower() == "darwin"
else ai2thor.platform.CloudRendering
),
}
controller = Controller(**controller_args)

Command to Read RGB Images:

event = controller.step("ToggleMagnetVisibility", visible=False, raise_for_failure=True, renderImageSynthesis=True)
static_rgb = event.frame # 224 396 3
gripper_rgb = event.third_party_camera_frames[0][:, :, :3] # 224 396 3

Could you please provide guidance on why this might be happening and how to resolve it? Any insights or suggestions would be greatly appreciated.

Thank you for your help!