About acquiring training data image focal length

[Alex000821]

Hi, I'm curious how do you segment images with focal length and other parameters from panoramic images? The method you provided seems to estimate the focal length by the maximum distortion radius and image size. Is there any similar literature reference？

[AlanSavio25]

Hi, here is the function that does the perspective image sampling from panorama images. We use inverse mapping—we start from the image coordinates and map each image pixel to a pixel in the panorama.

1. Given a desired image height and width, we can obtain the pixel coordinates for which we want the colour values.
2. Using the focal length, f, and distortion parameters, k1 and k2, we unproject the pixel coordinates to the z=1 plane (normalized coordinates).
3. You can then rotate your camera (i.e. rotate the normalized coordinates) using your desired roll, pitch, and yaw angles.
4. Since the panorama is a 360-degree image, we can roll it to a sphere and then interpolate the pixel values at our rotated normalized coordinates to give us the final image.

Regarding how the model learns the focal length, we predict the field of view (fov) instead. This is easier to predict by looking at the scale of objects in the image. You can then retrieve the focal length by solving this equation: horizontal_fov = 2*arctan2(width/(2*focal_length)). You can find more details in Section 3.2 in my project report and also in this paper. Hope this helps

[Alex000821]

Hi, here is the function that does the perspective image sampling from panorama images. We use inverse mapping—we start from the image coordinates and map each image pixel to a pixel in the panorama.

1. Given a desired image height and width, we can obtain the pixel coordinates for which we want the colour values.
2. Using the focal length, f, and distortion parameters, k1 and k2, we unproject the pixel coordinates to the z=1 plane (normalized coordinates).
3. You can then rotate your camera (i.e. rotate the normalized coordinates) using your desired roll, pitch, and yaw angles.
4. Since the panorama is a 360-degree image, we can roll it to a sphere and then interpolate the pixel values at our rotated normalized coordinates to give us the final image.

Regarding how the model learns the focal length, we predict the field of view (fov) instead. This is easier to predict by looking at the scale of objects in the image. You can then retrieve the focal length by solving this equation: horizontal_fov = 2*arctan2(width/(2*focal_length)). You can find more details in Section 3.2 in my project report and also in this paper. Hope this helps

Thanks for your answer, I now have a better understanding of how to get the data set, but I still have a question, why not use the existing image data set with various real camera parameter labels?

[AlanSavio25]

Existing datasets are typically unsuitable because they often do not contain enough data covering a wide and continuous range of parameters and are usually highly biased towards specific parameters. Training a model on such data will likely hurt generalisability. With panoramas, we can generate a training dataset with any choice of parameter distribution.

We could also transform images in existing datasets to have different rolls, pitches, etc., but this would lead to significant artefacts or invalid pixels.

[Alex000821]

Existing datasets are typically unsuitable because they often do not contain enough data covering a wide and continuous range of parameters and are usually highly biased towards specific parameters. Training a model on such data will likely hurt generalisability. With panoramas, we can generate a training dataset with any choice of parameter distribution.

We could also transform images in existing datasets to have different rolls, pitches, etc., but this would lead to significant artefacts or invalid pixels.

I want to try to use higher focus images for training, but I always fail, it gives me the prompt CUDA ERROR: device-side assert triggered at, do I need to modify some values in the network?

[Alex000821]

Existing datasets are typically unsuitable because they often do not contain enough data covering a wide and continuous range of parameters and are usually highly biased towards specific parameters. Training a model on such data will likely hurt generalisability. With panoramas, we can generate a training dataset with any choice of parameter distribution.

We could also transform images in existing datasets to have different rolls, pitches, etc., but this would lead to significant artefacts or invalid pixels.

The specific error message is: ../aten/src/ATen/native/cuda/Loss.cu:250: nll_loss_forward_reduce_cuda_kernel_2d: block: [0,0,0], thread: [3,0,0] Assertion t > = 0 && t < n_classes failed.
../aten/src/ATen/native/cuda/Loss.cu:250: nll_loss_forward_reduce_cuda_kernel_2d: block: [0,0,0], thread: [7,0,0] Assertion t >= 0 && t < n_classes failed.
terminate called after throwing an instance of 'c10::Error'
what(): CUDA error: device-side assert triggered
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.