Optical Flow Definition

[milmario]

For me, the definition of the optical flow output of CARLA is not clear.

1. Flow Type: at time-step t, am I getting the forward optical flow from t to t+1 or am I getting the backward optical flow from t to t-1? (For reference, see Figure 2 in this paper)
2. Scaling: From your documentation, it's not really clear to me how to correctly scale the flow output to get pixel values. Do I multiply the flow in x-direction with the sensor width and the flow in y-direction with the sensor height?

[danqu130]

I have the same question and look forward to answering it.

[milmario]

To save you some time, here's what i found out so far:

1. Flow Type: Flow[t] is either the forward flow from t -> t+1 (you have to multiply the flow in y-direction by -1, to be in the standard computer vision coordinate system), OR it's the flow from t -> t-1 (if you multiply the flow in x-direction by -1). The second option seems less convincing from a logical point of view, but numerical errors after warping an image are better for this option.
2. Scaling: It seems like the correct scaling is sensor_width/2 for the flow in x-direction, and -1*sensor_height/2 in y_direction.

To summarize, my visual & numerical tests show the following:

• Flow[i] is the optical flow from i->i-1 (so backward optical flow).
• To match the standard computer vision definition, you have to scale the x-direction by (-1)/sensor_width, and the y-direction by 1/sensor_height.

I'm looking forward to your input.

[danqu130]

@milmario
Thanks for your help, I have tested the assumption of using the simulated optical flow as the forward flow t->t+1 using the backward warping function provide by mmcv (https://github.com/open-mmlab/mmcv/blob/master/mmcv/video/optflow.py#L143).

I first normalize the raw flow map (captured at time t) with the shape of [frame_height, frame_width, 2], to the pixel displacements.

flow_data[:, :, 0] *= frame_width * 0.5
flow_data[:, :, 1] *= frame_height * -0.5

Then use the backward warping function on the second frame (at t+1).

warped_image = flow_warp(rgb2_image, flow_data)

The warped result seems match to the first frame (at t).

flow

sampled flow direction on the first frame

the first frame

the second frame

the warped result

I'm not sure whether this assumption is correct. I am working on test as the backward flow from t to t-1.
Besides, I hope we can get further official clarification from the Carla team.

[danqu130]

To summarize, my visual & numerical tests show the following:
Flow[i] is the optical flow from i->i-1 (so backward optical flow).

I think this assumption are right. I have tested the assumption of using the simulated optical flow as the forward flow (t->t+1) or the backward flow (t->t-1) using the backward warping function provide by mmcv (https://github.com/open-mmlab/mmcv/blob/master/mmcv/video/optflow.py#L143).

For given three rgb image rgb0_image, rgb1_image, rgb2_image (at time t-1, t, t+1, with the shape of [frame_height, frame_width, 3]) and one flow data flow_data (at time t, with the shape of [frame_height, frame_width, 2]) obtained by carla.

rgb0_image

rgb1_image

rgb2_image

I first normalize the raw flow data to the pixel displacements.

flow_10 = flow_data.copy()
flow_10[:, :, 0] *= frame_width * -0.5
flow_10[:, :, 1] *= frame_height * 0.5
flow_12 = flow_data.copy()
flow_12[:, :, 0] *= frame_width * 0.5
flow_12[:, :, 1] *= frame_height * -0.5

flow_10

sampled backward flow vector on the first frame

flow_12

sampled forward flow vector on the first frame

Then use the backward warping function. For forward flow, warp on the next image. For backward flow, warp on the pervious image.

warped_image0 = flow_warp(rgb0_image, flow_10)
warped_image2 = flow_warp(rgb2_image, flow_12)

warped_image0

warped_image2

The warped result using backward flow seems more matched to the frame rgb1_image at t.

In addition, the flow at time t and the image of at time t are simulated at the same time. At this time, the image of frame t+1 has not been simulated, and the future motion is uncertain. Therefore, I think it is impossible to obtain the forward optical flow from t to t+1 during real-time simulation, which is also consistent with the simple experimental results above.

I'm not sure whether this conclusion is correct. I hope we can get further official clarification from the Carla team.

[jonasdieker]

Hey, @danqu130 thanks for the great visualizations! I have a question about how you obtained the flow_data from the raw data. I read in the Carla docs that the raw data is 64-bit but if I use np.frombuffer and dtype float64 to convert from raw to a numpy array, I then receive only one value per pixel location, instead of the expected two values. Any insight would be appreciated. Thanks in advance!

[danqu130]

Hey, @danqu130 thanks for the great visualizations! I have a question about how you obtained the flow_data from the raw data. I read in the Carla docs that the raw data is 64-bit but if I use np.frombuffer and dtype float64 to convert from raw to a numpy array, I then receive only one value per pixel location, instead of the expected two values. Any insight would be appreciated. Thanks in advance!

The code I use to convert carla.Image data into numpy array flow_data is as follows. It may be slow, but it works.

flow_data = np.array([(pixel.x, pixel.y) for pixel in data], dtype=np.float32)
flow_data = flow_data.reshape((data.height, data.width, 2))

[danqu130]

I was confused when I observed the simulated optical flow. It has obvious layered optical flow in some scenes, but it seems unreasonable.
Let me show an example of a vehicle moving forward.

This is the simulated flow and the corresponding first frame.

And this is the optical flow computed by RAFT (https://github.com/princeton-vl/RAFT-3D) using the first and the sceond frame.

I calculated the EPE error of these two flow, and marked the position where the error is greater than 1 pixel as white.

The layering problem in the CARLA flow does not match the real motion, which is also confirmed by the results of RAFT. I have tried a variety of motion speeds and simulation frame rates, which also have this problem. In addition, I have checked my optical flow visualization steps. Even if I directly use the method provided by Carla (carla.OpticalFlowImage.get_color_coded_flow()), such layering still exists.

Hey, @milmario @jonasdieker, did you also encounter this? Do you think it's normal?

[knelk]

@danqu130, under which simulated conditions (light, speed, camera intrinsic,...) do you get these artefacts? Can you please share your overall config for the above result? It seems that other carla-flow samples look quite good, so I am wondering where it comes from.

Just to make sure: Is this the (very) first frame you are showing? If this is the case, it would make sense to have such artefacts, since OF is backward.

[danqu130]

CARLA version is 0.9.13, 4.26.2-0+++UE4+Release-4.26
I can see this artifact by directly using the official manual_control.py python example, with autopilot mode and change to show the optical flow sensor.

I think it is not normal for such artifacts to appear in this autonomous driving scene, whether forward flow or backward flow. In addition, has the CARLA team confirmed that the simulated optical flow is backward flow?

[ilaygol]

@danqu130 I've encountered the same issue of this staircase effect. Did anyone solve it?

[danqu130]

@danqu130 I've encountered the same issue of this staircase effect. Did anyone solve it?

Unfortunately, I don't have any more updates.