# Algorithms file format specification
Algorithms spec is the file format used by videostitch-cmd. It uses full [JSON](http://www.json.org/) syntax. Algorithms are simply defined by populating the root object with a few named variables.
## Objects
VideoStitch will interpret the following objects. An object is of course free to have other members in addition to these.
### Root
videostitch-cmd will interpret the following optional members of the root object:
<table>
<tr><th>Member</th><th>Type</th><th>Default value</th><th></th></tr>
<tr><th>outputPtv</th><td>list</td><td>Optional</td><td>If provided, this value specifies the output ptv file that will contain the results of the "algorithms" and the settings from the input ptv.</td></tr>
</table>
The tables below show the members of the "outputPtv"
<table>
<tr><th>Member</th><th>Type</th><th>Default value</th><th></th></tr>
<tr><th> name</th><td>string</td><td>No Default</td><td>Names of the output ptv.</td></tr>
<tr><th> outputFile</th><td>list</td><td>Optional</td><td>An optional list to specify the format of the final output in the new ptv file.</td></tr>
</table>
#### outputFile
The "outputFile" contains two optional members.
<table>
<tr><th>Member</th><th>Type</th><th>Default value</th><th></th></tr>
<tr><th> name</th><td>string</td><td>Optional</td><td>An optional string to specify the name of the output file.</td></tr>
<tr><th> type</th><td>string</td><td>Optional</td><td>An optional string to specify the output extension.</td></tr>
</table>
An example setting
"outputPtv": [
{
"name" : "outputFinal.ptv",
"outputFile": {
"name" : "out-vs",
"type" : "jpg"
}
}
]
### Algorithms
Algorithms specify which algorithms to be called from command-line
In the tables below, mandatory members are shown in bold. If not set, optional members take a default value specified in the "Default value" column.
<table>
<tr><th>Member</th><th>Type</th><th>Default value</th><th></th></tr>
<tr><th> name</th><td>string</td><td>mask</td><td>Type of algorithm. One of the types below.</td></tr>
</table>
#### "mask" algorithm:
The "mask" algorithm optimizes for the blending mask and blending order.
<table>
<tr><th>Member</th><th>Type</th><th>Default value</th><th></th></tr>
<tr><th> list_frames</th><td>list</td><td>[0]</td><td>The list of frames used for the optimization.</td></tr>
<tr><th> blending_order</th><td>bool</td><td>true</td><td>To specify whether the blending order is consider or not. If not, the order of inputs will be used as the blending order.</td></tr>
<tr><th> seam</th><td>bool</th><td>true</td><td>To specify whether the optimal seams are computed.</td></tr>
</table>
An example setting for the "mask" algorithm is:
{
"algorithms" : [
{
"name": "mask",
"config" : {
"list_frames" : [0],
"blending_order" : true,
"seam" : true
}
}
]
}
#### "autocrop" algorithm:
The "autocrop" algorithm detects the crop circle for the fish-eye images.
The first frame of the inputs is used for detection.
<table>
<tr><th>Member</th><th>Type</th><th>Default value</th><th></th></tr>
<tr><th> neighborThreshold</th><td>int</td><td>40</td><td>A threshold used for the image binarization step. Indicates the similarity of neighboring pixels to be considered as a connected component.</td></tr>
<tr><th> differenceThreshold</th><td>int</th><td>500</td><td>A threshold used for the binarization step. Indicates the similarity of a pixel to the seed pixel to be considered as a connected component.</td></tr>
<tr><th> fineTuneMarginSize</th><td>int</th><td>100</td><td>A pre-defined range around the coarse circle's samples to look for the fine scale samples, e.g. 0 indicates that the fine scale samples are the coarse samples, while 100 indicates that the fine scale samples will be searched in the range [-100..100] px around the coarse samples.</td></tr>
<tr><th> scaleRadius</th><td>double</th><td>0.98</td><td>A parameter used to scale the fine scale circle's radius.</td></tr>
<tr><th> circleImage</th><td>bool</th><td>false</td><td>Whether to dump an image with the crop circle overlays on top of the original image.</td></tr>
</table>
An example setting for the "autocrop" algorithm is:
{
"algorithms" : [
{
"name": "autocrop",
"config" : {
"neighborThreshold" : 40,
"differenceThreshold" : 500,
"fineTuneMarginSize" : 100,
"circleImage" : false,
"scaleRadius" : 0.98
}
}
]
}
#### "calibration" algorithm:
The "calibration" algorithm runs a calibration with presets for the rig.
<table>
<tr><th>Member</th><th>Type</th><th>Default value</th><th></th></tr>
<tr><th>config</th><td>list</td><td>Optional</td><td>The list of calibration configuration parameters.</td></tr>
<tr><th>apply_presets_only</th><td>bool</td><td>Optional</td><td>If true, applies the rig presets to the PanoDefinition without actually calibrating. Meant to obtain a template project definition out of inputs and presets.</td></tr>
<tr><th>improve_mode</th><td>bool</td><td>Optional</td><td>If true, calibration will reused past control points found in the JSON tree.</td></tr>
<tr><th>auto_iterate_fov</th><td>bool</td><td>Optional</td><td>If true, calibration will try to estimate the FOV of the lens.</td></tr>
<tr><th>single_focal</th><td>bool</td><td>Optional</td><td>If true, the optimizer will estimate a single (fu,fv) pair of parameters for all lenses in the rig.</td></tr>
<tr><th>dump_calibration_snapshots</th><td>bool</td><td>Optional</td><td>If true, calibration will dump pictures with control points at various stages of the procedure.</td></tr>
<tr><th>deshuffle_mode</th><td>bool</td><td>Optional</td><td>If true, calibration will extract key points and reorder the video inputs using the rig presets, before running the full calibration.</td></tr>
<tr><th>deshuffle_mode_only</th><td>bool</td><td>Optional</td><td>If true, calibration will extract key points and reorder the video inputs using the rig presets, without running the full calibration.</td></tr>
<tr><th>deshuffle_mode_preserve_readers_order</th><td>bool</td><td>Optional</td><td>If true, deshuffling will preserve the order of input readers in the returned PanoDefinition and reorder the geometries. Otherwise, the deshuffling will keep the order of geometries and reorder the input readers.</td></tr>
<tr><th>use_synthetic_keypoints</th><td>bool</td><td>false</td><td>If true, the calibration algorithm will generate artificial keypoints from the PanoDefinition geometry, to cover the input areas where no real keypoint was extracted and preserve the PanoDefinition geometry.</td></tr>
<tr><th>synthetic_keypoints_grid_width</th><td>int</td><td>5</td><td>Grid width to generate artificial keypoints in each input picture.</td></tr>
<tr><th>synthetic_keypoints_grid_height</th><td>int</td><td>5</td><td>Grid height to generate artificial keypoints in each input picture.</td></tr>
<tr><th>cp_extractor</th><td>list</td><td>Optional</td><td>The list of parameters for the control point detector and matcher.</td></tr>
<tr><th>extractor</th><td>string</td><td>Optional</td><td>The name of the control point detector.</td></tr>
<tr><th>matcher_norm</th><td>string</td><td>Optional</td><td>The name of the control point matcher.</td></tr>
<tr><th>octaves</th><td>int</td><td>Optional</td><td>The number of octaves used for the detection.</td></tr>
<tr><th>sublevels</th><td>integer</td><td>Optional</td><td>The number of sublevels used for the detection.</td></tr>
<tr><th>threshold</th><td>double</td><td>Optional</td><td>Detection threshold.</td></tr>
<tr><th>nndr_ratio</th><td>double</td><td>Optional</td><td>Ratio between the first and second best score to claim a right match.</td></tr>
<tr><th>cp_filter</th><td>list</td><td>Optional</td><td>Parameters for the control point filter, which uses RANSAC.</td></tr>
<tr><th>angle_threshold</th><td>double</td><td>Optional</td><td>Angle threshold to validate a control point rotated from one lens to the other.</td></tr>
<tr><th>min_ratio_inliers</th><td>double</td><td>Optional</td><td>Minimum ratio of inliers, before RANSAC.</td></tr>
<tr><th>min_samples_for_fit</th><td>integer</td><td>Optional</td><td>Minimum number of control points to run RANSAC.</td></tr>
<tr><th>proba_draw_outlier_free_samples</th><td>double</td><td>Optional</td><td>Target probability to achieve for RANSAC.</td></tr>
<tr><th>decimating_grid_size</th><td>double</td><td>Optional</td><td>Grid size for decimation (w.r.t. picture size).</td></tr>
<tr><th><a name="rig-preset"> rig </a></th><td>list</td><td>Optional</td><td>The list of rig preset parameters.</td></tr>
<tr><th>name</th><td>string</td><td>Mandatory</td><td>The name of the rig presets.</td></tr>
<tr><th>rigcameras</th><td>list</td><td>Mandatory</td><td>The list of camera presets.</td></tr>
<tr><th>yaw_mean</th><td>double</td><td>Mandatory</td><td>Mean value of the yaw rotation angle for the optimizer.</td></tr>
<tr><th>yaw_variance</th><td>double</td><td>Mandatory</td><td>Variance value of the yaw rotation angle for the optimizer.</td></tr>
<tr><th>pitch_mean</th><td>double</td><td>Mandatory</td><td>Mean value of the pitch rotation angle for the optimizer.</td></tr>
<tr><th>pitch_variance</th><td>double</td><td>Mandatory</td><td>Variance value of the pitch rotation angle for the optimizer.</td></tr>
<tr><th>roll_mean</th><td>double</td><td>Mandatory</td><td>Mean value of the roll rotation angle for the optimizer.</td></tr>
<tr><th>roll_variance</th><td>double</td><td>Mandatory</td><td>Variance value of the roll rotation angle for the optimizer.</td></tr>
<tr><th>camera</th><td>string</td><td>Mandatory</td><td>Camera preset name (from the list of cameras).</td></tr>
<tr><th><a name="cameras-presets">cameras</a></th><td>list</td><td>Mandatory</td><td>The list of camera(s) presets.</td></tr>
<tr><th>name</th><td>string</td><td>Mandatory</td><td>The name of the camera presets.</td></tr>
<tr><th>format</th><td>string</td><td>Mandatory</td><td>The format of the lens projection.</td></tr>
<tr><th>width</th><td>integer</td><td>Mandatory</td><td>The width of the camera pictures.</td></tr>
<tr><th>height</th><td>integer</td><td>Mandatory</td><td>The height of the camera pictures.</td></tr>
<tr><th>fu_mean</th><td>double</td><td>Mandatory</td><td>Mean value of the horizontal focal for the optimizer.</td></tr>
<tr><th>fu_variance</th><td>double</td><td>Mandatory</td><td>Variance value of the horizontal focal for the optimizer.</td></tr>
<tr><th>fv_mean</th><td>double</td><td>Mandatory</td><td>Mean value of the vertical focal for the optimizer.</td></tr>
<tr><th>fv_variance</th><td>double</td><td>Mandatory</td><td>Variance value of the vertical focal for the optimizer.</td></tr>
<tr><th>cu_mean</th><td>double</td><td>Mandatory</td><td>Mean value of the horizontal center of projection for the optimizer.</td></tr>
<tr><th>cu_variance</th><td>double</td><td>Mandatory</td><td>Variance value of the horizontal center of projection for the optimizer.</td></tr>
<tr><th>cv_mean</th><td>double</td><td>Mandatory</td><td>Mean value of the vertical center of projection for the optimizer.</td></tr>
<tr><th>cv_variance</th><td>double</td><td>Mandatory</td><td>Variance value of the vertical center of projection for the optimizer.</td></tr>
<tr><th>distorta_mean</th><td>double</td><td>Mandatory</td><td>Mean value of the "a" distortion parameter for the optimizer.</td></tr>
<tr><th>distorta_variance</th><td>double</td><td>Mandatory</td><td>Variance value of "a" distortion parameter for the optimizer.</td></tr>
<tr><th>distortb_mean</th><td>double</td><td>Mandatory</td><td>Mean value of the "b" distortion parameter for the optimizer.</td></tr>
<tr><th>distortb_variance</th><td>double</td><td>Mandatory</td><td>Variance value of "b" distortion parameter for the optimizer.</td></tr>
<tr><th>distortc_mean</th><td>double</td><td>Mandatory</td><td>Mean value of the "c" distortion parameter for the optimizer.</td></tr>
<tr><th>distortc_variance</th><td>double</td><td>Mandatory</td><td>Variance value of "c" distortion parameter for the optimizer.</td></tr>
<tr><th>list_frames</th><td>list</td><td>Optional</td><td>The list of frames used for the optimization.</td></tr>
</table>
### "calibration\_presets\_maker" algorithm:
The "calibration\_presets\_maker" algorithm takes an input PanoDefinition, and creates calibration presets (i.e. <a href="#rig-preset">"rig"</a> and <a href="#cameras-presets">"cameras"</a> JSON objects) from it, to be used by the "calibration" algorithm.
The returned PanoDefinition also contains the calibration presets, and its control point list is cleared.
<table>
<tr><th>Member</th><th>Type</th><th>Default value</th><th></th></tr>
<tr><th>output</th><td>string</td><td>none</td><td>The name of the JSON file receiving the results of the algorithm.</td></tr>
<tr><th>name</th><td>string</td><td>Mandatory</td><td>The name of the rig presets.</td></tr>
<tr><th>focal_std_dev_value_percentage</th><td>double</td><td>5.0</td><td>Standard Deviation of focal parameters, expressed in terms of percentage of the input values.</td></tr>
<tr><th>center_std_dev_width_percentage</th><td>double</td><td>10.0</td><td>Standard Deviation of center parameters, expressed in terms of percentage of the width of the input.</td></tr>
<tr><th>distort_std_dev_value_percentage</th><td>double</td><td>50.0</td><td>Standard Deviation of distortion parameters, expressed in terms of percentage of the input values.</td></tr>
<tr><th>yaw_std_dev</th><td>double</td><td>5.0</td><td>Standard Deviation of yaw angles, in degrees.</td></tr>
<tr><th>pitch_std_dev</th><td>double</td><td>5.0</td><td>Standard Deviation of pitch angles, in degrees.</td></tr>
<tr><th>roll_std_dev</th><td>double</td><td>5.0</td><td>Standard Deviation of roll angles, in degrees.</td></tr>
<tr><th>translation_x_std_dev</th><td>double</td><td>0.0</td><td>Standard Deviation of X translations, in meters.</td></tr>
<tr><th>translation_y_std_dev</th><td>double</td><td>0.0</td><td>Standard Deviation of Y translations, in meters.</td></tr>
<tr><th>translation_z_std_dev</th><td>double</td><td>0.0</td><td>Standard Deviation of Z translations, in meters.</td></tr>
</table>
An example setting for the "calibration\_presets\_maker" algorithm is:
{
"algorithms" : [
{
"name": "calibration_presets_maker",
"output": "output_presets.json",
"config" : {
"name" : "Orah Tetra 4i",
"focal_std_dev_value_percentage" : 15.0,
"center_std_dev_width_percentage" : 10.0,
"distort_std_dev_value_percentage" : 50,
"yaw_std_dev" : 1.0,
"pitch_std_dev" : 1.0,
"roll_std_dev" : 1.0,
"translation_x_std_dev" : 0.07,
"translation_y_std_dev" : 0.07,
"translation_z_std_dev" : 0.
}
}
]
}
An example output of the "calibration\_presets\_maker" algorithm is:
{
"rig" : {
"name" : "Orah Tetra",
"rigcameras" : [
{
"angle_unit" : "degrees",
"yaw_mean" : 0,
"pitch_mean" : 0,
"roll_mean" : 0,
"yaw_variance" : 16414,
"pitch_variance" : 16414,
"roll_variance" : 16414,
"camera" : "camera_0"
},
...
{
"angle_unit" : "degrees",
"yaw_mean" : 93.068,
"pitch_mean" : -67.2862,
"roll_mean" : -68.2285,
"yaw_variance" : 16414,
"pitch_variance" : 16414,
"roll_variance" : 16414,
"camera" : "camera_3"
}
]
},
"cameras" : [
{
"name" : "camera_0",
"format" : "circular_fisheye_opt",
"width" : 1920,
"height" : 1440,
"fu_mean" : 1035.6,
"fu_variance" : 24130.5,
"fv_mean" : 1046.15,
"fv_variance" : 24624.7,
"cu_mean" : 885.214,
"cu_variance" : 36864,
"cv_mean" : 814.362,
"cv_variance" : 36864,
"distorta_mean" : 0,
"distorta_variance" : 0,
"distortb_mean" : -0.401758,
"distortb_variance" : 0.0403524,
"distortc_mean" : 0,
"distortc_variance" : 0
},
...
{
"name" : "camera_3",
"format" : "circular_fisheye_opt",
"width" : 1920,
"height" : 1440,
"fu_mean" : 1035.6,
"fu_variance" : 24130.5,
"fv_mean" : 1046.15,
"fv_variance" : 24624.7,
"cu_mean" : 974.67,
"cu_variance" : 36864,
"cv_mean" : 690.774,
"cv_variance" : 36864,
"distorta_mean" : 0,
"distorta_variance" : 0,
"distortb_mean" : -0.311981,
"distortb_variance" : 0.024333,
"distortc_mean" : 0,
"distortc_variance" : 0
}
]
}
#### "epipolar" algorithm (experimental, used for debugging purposes):
The "epipolar" algorithm takes a calibrated pano definition and a list of frames, of input points and/or matched input points to produce pictures showing:
* a spherical grid of points (optional)
* the location of points or point pairs
* the epipolar curves corresponding to the points or point pairs
* the estimated depth of point pairs
and outputs on the "Info" logger level the estimated error-free stitching distance.
The point pairs can be automatically detected and matched.
It also computes the rig minimum stitching distance (i.e. sphere scale) where points seen by at least 2 cameras become visible by only one as a floating metric point value in the output JSON file, and saves a equirectangular gray-level 8 bits picture "output\_min\_stitching\_distance.png" (which size is given by the output panorama size of the input panorama definition) where one intensity value is 1 cm. This scaling can be controlled by the **image\_max\_output\_depth** parameter below.
<table>
<tr><th>Member</th><th>Type</th><th>Default value</th><th></th></tr>
<tr><th> list_frames</th><td>list</td><td>[0]</td><td>The list of frames used for the optimization.</td></tr>
<tr><th> spherical_grid_radius</th><td>double</th><td>0</td><td>If non-zero, generates a spherical grid of points with this radius.</td></tr>
<tr><th> auto_point_matching</th><td>bool</th><td>true</td><td>If true, automatically matching points, in addition to the ones provided.</td></tr>
<tr><th>decimating_grid_size</th><td>double</td><td>0.04</td><td>Grid size for decimation of matched points (w.r.t. picture size).</td></tr>
<tr><th> single_points</th><td>list</th><td>empty</td><td>List of single points.</td></tr>
<tr><th> input_index</th><td>int</th><td>mandatory</td><td>Element of single_points list: camera input index of a single point.</td></tr>
<tr><th> x</th><td>float</th><td>mandatory</td><td>Element of single_points list: "x" coordinate of a single point in its input.</td></tr>
<tr><th> y</th><td>float</th><td>mandatory</td><td>Element of single_points list: "y" coordinate of a single point in its input.</td></tr>
<tr><th> matched_points</th><td>list</th><td>empty</td><td>List of matched pairs of points.</td></tr>
<tr><th> input_index0</th><td>int</th><td>mandatory</td><td>Element of matched_points list: camera input index of the first point in pair.</td></tr>
<tr><th> x0</th><td>float</th><td>mandatory</td><td>Element of matched_points list: "x" coordinate of the first point in a pair, in its input.</td></tr>
<tr><th> y0</th><td>float</th><td>mandatory</td><td>Element of matched_points list: "y" coordinate of the first point in a pair, in its input.</td></tr>
<tr><th> input_index1</th><td>int</th><td>mandatory</td><td>Element of matched_points list: camera input index of the second point in a pair.</td></tr>
<tr><th> x1</th><td>float</th><td>mandatory</td><td>Element of matched_points list: "x" coordinate of the second point in a pair, in its input.</td></tr>
<tr><th> y1</th><td>float</th><td>mandatory</td><td>Element of matched_points list: "y" coordinate of the second point in a pair, in its input.</td></tr>
<tr><th> image_max_output_depth</th><td>float</th><td>2.55</td><td>Depth value mapped to 255 in the output depth image. Leaving it to 2.55 means that 255 will be mapped to 2.55 meters, i.e. one gray-level is 1 cm. </td></tr>
</table>
An example setting for the "epipolar" algorithm is:
{
"algorithms" : [
{
"name": "epipolar",
"config" : {
"auto_point_selection" : false,
"single_points" : [
{
"input_index" : 0,
"x" : 1000,
"y" : 720
},
{
"input_index" : 1,
"x" : 600,
"y" : 600
}
],
"matched_points" : [
{
"input_index0" : 3,
"x0" : 144,
"y0" : 445,
"input_index1" : 2,
"x1" : 76,
"y1" : 931
}
],
"list_frames" : [
1,
200
]
}
}
]
}
An example JSON output is:
{
"minStitchingDistance" : 0.073859989643096924
}
#### "scoring" algorithm:
The "scoring" algorithm takes a calibrated pano definition and produces a list of scores per frame: a score between 0 and 1, based on the normalized cross correlation of input pairs in overlap in the equirectangular projection, and the ratio of uncovered pixels (i.e. holes in the projection).
<table>
<tr><th>Member</th><th>Type</th><th>Default value</th><th></th></tr>
<tr><th>output</th><td>string</td><td>none</td><td>The name of the JSON file receiving the results of the algorithm.</td></tr>
<tr><th> first_frame</th><td>int</th><td>0</td><td>First frame number for the scoring.</td></tr>
<tr><th> last_frame</th><td>int</th><td>0</td><td>Last frame number for the scoring.</td></tr>
</table>
An example setting for the "scoring" algorithm is:
{
"algorithms" : [
{
"name": "scoring",
"output": "output_scoring.ptv",
"config" : {
"first_frame" : 0,
"last_frame" : 0
}
}
]
}
An example output of the "scoring" algorithm is:
[
{
"frame_number" : 0,
"score" : 0.609535,
"uncovered_ratio" : 0.0705967
},
{
"frame_number" : 1,
"score" : 0.609598,
"uncovered_ratio" : 0.0705967
},
{
"frame_number" : 2,
"score" : 0.609391,
"uncovered_ratio" : 0.0705967
}
]