ios – How you can relate intrinsics and distortion from AVDepthData to the present video stream?

Spread the love


I’m writing a small check app within the context of Pc Imaginative and prescient. The ultimate utility would require utilizing the digicam calibration, so for now my check app creates a seize session, allows the supply of the intrinsics matrix and logs it.

For example, this code:

let calibrationPayload = CMGetAttachment(sampleBuffer, key: kCMSampleBufferAttachmentKey_CameraIntrinsicMatrix, attachmentModeOut: nil)
    if let knowledge = calibrationPayload as? Information {
        let matrix: matrix_float3x3 = knowledge.withUnsafeBytes { $0.pointee }
        print(matrix)
}

operating on an iPhone 13 Professional again digicam provides me:

simd_float3x3([[4220.394, 0.0, 0.0], [0.0, 4220.394, 0.0], [941.9231, 533.648, 1.0]])

// Corresponding matrix (which is sensible for an 1920x1080 digicam):
// [4220.394,    0.0,   941.9231]
// [   0.0,   4220.394, 533.648]
// [   0.0,      0.0,      1.0]

Nonetheless, I might like now to additionally get the distortion related to that lens. To take action I’ve modified my app to request a tool of sort .builtInDualCamera and enabled the depth knowledge stream as solely AVDepthData buffers have some companion distortion knowledge (of their cameraCalibrationData property).

Within the depth seize delegate name I am logging the distortion middle, lookup desk and in addition the digicam intrinsics:

guard let calibrationData = depthData.cameraCalibrationData else {
    return
}
print("Intrinsics = (calibrationData.intrinsicMatrix)")

let distoCenter = calibrationData.lensDistortionCenter
print("Distortion middle: (distoCenter)")

// Extra code to log the LUT knowledge too

Nonetheless on this case the intrinsics are extensively totally different and truly make no sense for an 1920×1080 digicam (there appears to be a scale issue of 2.20 between the two intrinsic matrices):

Intrinsics = simd_float3x3([[9284.896, 0.0, 0.0], [0.0, 9284.896, 0.0], [2072.8423, 1174.5812, 1.0]])

// Corresponding matrix:
// [9284.896, 0.0, 2072.8423]
// [0.0, 9284.896,1174.5812]
// [0.0, 0.0, 1.0]

Distortion middle: (2072.839599609375, 1174.5499267578125)
  1. Can somebody please clarify to me the place this 2.20 ratio come from?
  2. Is it potential to precompute it primarily based on some queries to the seize session, or does it should be estimated from the focal lengths within the intrinsics matrices?
  3. Making use of accurately the LUT to rectify the picture requires computing distances from the distortion middle, I assume this has to bear in mind the additional rescaling?
// Instance code in AVCalibrationData.h to rectify a degree:
// Decide the utmost radius.
float delta_ocx_max = MAX( opticalCenter.x, imageSize.width  - opticalCenter.x );
float delta_ocy_max = MAX( opticalCenter.y, imageSize.peak - opticalCenter.y );
float r_max = sqrtf( delta_ocx_max * delta_ocx_max + delta_ocy_max * delta_ocy_max );
 
// Decide the vector from the optical middle to the given level.
float v_point_x = level.x - opticalCenter.x;
float v_point_y = level.y - opticalCenter.y;
 
// Decide the radius of the given level.
float r_point = sqrtf( v_point_x * v_point_x + v_point_y * v_point_y );
 
// Lookup the relative radial magnification to use within the offered lookup desk
float magnification;
const float *lookupTableValues = lookupTable.bytes;
NSUInteger lookupTableCount = lookupTable.size / sizeof(float);
 
if ( r_point < r_max ) {
  ...
}

Leave a Reply

Your email address will not be published. Required fields are marked *