sdfstudio
sdfstudio copied to clipboard
autonomousvision
transform.json with several cameras???
I have been preprocessing datasets from distinct captures (moments in time and camera sizes). I could only acquire this by using hloc, obtaining the images.bin and cameras.bin. However there is the problem that I cannot export the transforms.json in which each frame is from a distinct camera (f.e. 18 frames for 18 cameras).
This error arises https://github.com/nerfstudio-project/nerfstudio/blob/main/nerfstudio/process_data/colmap_utils.py#L461
I commented this assertion and the code works, however, it is considering only the first camera and so the output transforms.json is incorrect.
Here is the cam_id_to_camera:
{
"1":Camera(id=1,
"model=""SIMPLE_RADIAL",
width=1521,
height=1073,
"params=array("[
3.1692782e+03,
7.6050000e+02,
5.3650000e+02,
-3.2129741e-01
]"))",
"2":Camera(id=2,
"model=""SIMPLE_RADIAL",
width=1062,
height=1523,
"params=array("[
2.01058080e+03,
5.31000000e+02,
7.61500000e+02,
7.41873012e-02
]"))",
"3":Camera(id=3,
"model=""SIMPLE_RADIAL",
width=1629,
height=1157,
"params=array("[
1.62281426e+03,
8.14500000e+02,
5.78500000e+02,
2.30765979e-02
]"))",
"4":Camera(id=4,
"model=""SIMPLE_RADIAL",
width=1601,
height=1119,
"params=array("[
2.09333469e+03,
8.00500000e+02,
5.59500000e+02,
1.16253708e-01
]"))",
"5":Camera(id=5,
"model=""SIMPLE_RADIAL",
width=1615,
height=1156,
"params=array("[
2.48802583e+03,
8.07500000e+02,
5.78000000e+02,
6.09272343e-02
]"))",
"6":Camera(id=6,
"model=""SIMPLE_RADIAL",
width=1624,
height=1162,
"params=array("[
1.96078379e+03,
8.12000000e+02,
5.81000000e+02,
7.16309553e-02
]"))",
"7":Camera(id=7,
"model=""SIMPLE_RADIAL",
width=1155,
height=1642,
"params=array("[
1.60673918e+03,
5.77500000e+02,
8.21000000e+02,
8.55971204e-03
]"))",
"8":Camera(id=8,
"model=""SIMPLE_RADIAL",
width=1613,
height=1114,
"params=array("[
1.32070856e+03,
8.06500000e+02,
5.57000000e+02,
3.21522170e-02
]"))",
"9":Camera(id=9,
"model=""SIMPLE_RADIAL",
width=1578,
height=1092,
"params=array("[
2.60890528e+03,
7.89000000e+02,
5.46000000e+02,
8.12536186e-02
]"))",
"10":Camera(id=10,
"model=""SIMPLE_RADIAL",
width=1580,
height=1003,
"params=array("[
2.56096207e+03,
7.90000000e+02,
5.01500000e+02,
1.39310730e-01
]"))",
"11":Camera(id=11,
"model=""SIMPLE_RADIAL",
width=1592,
height=1116,
"params=array("[
3.03610690e+03,
7.96000000e+02,
5.58000000e+02,
5.80613135e-02
]"))",
"12":Camera(id=12,
"model=""SIMPLE_RADIAL",
width=1593,
height=1107,
"params=array("[
2.52260764e+03,
7.96500000e+02,
5.53500000e+02,
7.04535419e-02
]"))",
"13":Camera(id=13,
"model=""SIMPLE_RADIAL",
width=1611,
height=1103,
"params=array("[
2.88055835e+03,
8.05500000e+02,
5.51500000e+02,
2.10120324e-01
]"))",
"14":Camera(id=14,
"model=""SIMPLE_RADIAL",
width=1646,
height=995,
"params=array("[
2.4982031e+03,
8.2300000e+02,
4.9750000e+02,
-1.3854031e-01
]"))",
"15":Camera(id=15,
"model=""SIMPLE_RADIAL",
width=1597,
height=1135,
"params=array("[
2.73717713e+03,
7.98500000e+02,
5.67500000e+02,
1.97624105e-01
]"))",
"16":Camera(id=16,
"model=""SIMPLE_RADIAL",
width=1611,
height=1106,
"params=array("[
4.84974167e+03,
8.05500000e+02,
5.53000000e+02,
1.22514371e+00
]"))",
"17":Camera(id=17,
"model=""SIMPLE_RADIAL",
width=1610,
height=1111,
"params=array("[
2.59722838e+03,
8.05000000e+02,
5.55500000e+02,
6.36294581e-02
]"))",
"18":Camera(id=18,
"model=""SIMPLE_RADIAL",
width=1609,
height=1063,
"params=array("[
1.50583974e+03,
8.04500000e+02,
5.31500000e+02,
1.06968565e-02
]"))"
}
and here the frames:
[
{
"file_path":"images/1126.jpg",
"transform_matrix":[
[
0.921260037730486,
0.06288893809857878,
0.3838292906301658,
2.105211533209113
],
[
0.37958703050345155,
0.06982038797403659,
-0.9225176419433542,
-0.49627269588663403
],
[
-0.08481526486661992,
0.9955752582743246,
0.0404509079949632,
-0.20863332076687988
],
[
0.0,
0.0,
0.0,
1.0
]
],
"colmap_im_id":1
},
{
"file_path":"images/1134.jpg",
"transform_matrix":[
[
0.8583242977307534,
0.11998327997713212,
0.49888216289113607,
1.770437228111344
],
[
0.4977941717173231,
0.04104329885648988,
-0.8663235020610032,
3.0586422844543204
],
[
-0.12442010500426791,
0.9919271445750971,
-0.024498516808594096,
-0.7686162062318393
],
[
0.0,
0.0,
0.0,
1.0
]
],
"colmap_im_id":2
},
{
"file_path":"images/2.jpg",
"transform_matrix":[
[
0.9987673374281286,
0.023247725771476515,
-0.043856002248050946,
-0.3619371437571082
],
[
-0.04332284445077449,
-0.022961861788500397,
-0.9987972186845036,
4.184698731103925
],
[
-0.024226779303506703,
0.9994660054997667,
-0.021926399953461295,
-0.8347879189623232
],
[
0.0,
0.0,
0.0,
1.0
]
],
"colmap_im_id":3
},
{
"file_path":"images/2326.jpg",
"transform_matrix":[
[
0.9370367611769083,
0.06775503444844672,
0.3425950430318276,
1.728675369817075
],
[
0.33861503269095566,
0.06376540098335143,
-0.9387618618548238,
1.2761133084455978
],
[
-0.08545155258269459,
0.9956622062448298,
0.03680765160784886,
-0.45186142155216497
],
[
0.0,
0.0,
0.0,
1.0
]
],
"colmap_im_id":4
},
{
"file_path":"images/2338.jpg",
"transform_matrix":[
[
0.8813299488127686,
0.10930712419638257,
0.4596841023198423,
1.841601872364886
],
[
0.4602579735036135,
0.021432087078036188,
-0.8875264860666562,
2.203777714907431
],
[
-0.10686495754937475,
0.9937769459206609,
-0.031420735264698414,
-0.6513425540316853
],
[
0.0,
0.0,
0.0,
1.0
]
],
"colmap_im_id":5
},
{
"file_path":"images/2340.jpg",
"transform_matrix":[
[
0.9206981141371043,
0.10766983005107991,
0.37512956471218234,
1.7572824124729307
],
[
0.3758289148940189,
0.014497557771394089,
-0.9265756566779905,
2.1902888554735798
],
[
-0.1052027060201598,
0.9940809969592199,
-0.027117561295204536,
-0.6448570997422394
],
[
0.0,
0.0,
0.0,
1.0
]
],
"colmap_im_id":6
},
{
"file_path":"images/2456.jpg",
"transform_matrix":[
[
0.8573272033900071,
0.05573010625622554,
0.5117462472594589,
1.2127377194929818
],
[
0.504346668458593,
0.10812185718237283,
-0.8567053764358857,
3.589515987762743
],
[
-0.10307523631885408,
0.9925743394104568,
0.0645885160203267,
-0.7812858964244921
],
[
0.0,
0.0,
0.0,
1.0
]
],
"colmap_im_id":7
},
{
"file_path":"images/2463.jpg",
"transform_matrix":[
[
0.9954484676003469,
0.03313166916561513,
-0.08935681759340801,
-1.6523912197371058
],
[
-0.09407310453685454,
0.19155985422161045,
-0.9769621657226002,
-0.1398620968511394
],
[
-0.015251208310143776,
0.9809215640146237,
0.1938047623155453,
1.0450828310962592
],
[
0.0,
0.0,
0.0,
1.0
]
],
"colmap_im_id":8
},
{
"file_path":"images/2485.jpg",
"transform_matrix":[
[
0.9891916330813568,
0.04931210562895927,
-0.13808775934265996,
-1.5941377427275276
],
[
-0.14527684385887527,
0.20202848644940175,
-0.9685448514660371,
-0.5301334614190419
],
[
-0.01986332500469107,
0.9781374171870753,
0.2070087955096492,
0.7207519715670379
],
[
0.0,
0.0,
0.0,
1.0
]
],
"colmap_im_id":9
},
{
"file_path":"images/2567.jpg",
"transform_matrix":[
[
0.8925616769222634,
0.06041009953577875,
0.44686046229644716,
2.161356105579826
],
[
0.43273160719096765,
0.16390283363283475,
-0.8864982894879434,
-1.3635388897736749
],
[
-0.12679514591513044,
0.9846250458937172,
0.12015202857694987,
0.5136719645620728
],
[
0.0,
0.0,
0.0,
1.0
]
],
"colmap_im_id":10
},
{
"file_path":"images/2573.jpg",
"transform_matrix":[
[
0.9991551080868791,
0.036873966540006284,
0.018148845018437103,
-0.6931337791903103
],
[
0.016013139447350416,
0.05741096296918211,
-0.9982222000616853,
-0.9374166026593576
],
[
-0.037850354673852955,
0.9976694301834439,
0.056771988942662915,
-0.09372331565158667
],
[
0.0,
0.0,
0.0,
1.0
]
],
"colmap_im_id":11
},
{
"file_path":"images/2601.jpg",
"transform_matrix":[
[
0.9557191580717495,
0.08063304345979128,
0.2830180262757794,
1.307514461284852
],
[
0.27937732346070393,
0.053565911278007576,
-0.9586860822421956,
1.0340873188077349
],
[
-0.09246189501929722,
0.9953034740476736,
0.028666923763676317,
-0.4118952305167651
],
[
0.0,
0.0,
0.0,
1.0
]
],
"colmap_im_id":12
},
{
"file_path":"images/2610.jpg",
"transform_matrix":[
[
0.998453318691547,
0.0305494432024383,
0.0464510701045464,
-0.2646977676315645
],
[
0.04352520702189186,
0.09031714805785866,
-0.9949614912751128,
-1.3977587790913204
],
[
-0.03459084774240067,
0.9954443953766156,
0.08884778537333513,
0.05542609888591876
],
[
0.0,
0.0,
0.0,
1.0
]
],
"colmap_im_id":13
},
{
"file_path":"images/2615.jpg",
"transform_matrix":[
[
0.9714382605592887,
0.04191056517656841,
-0.23356200557480283,
-1.7504395113195594
],
[
-0.23666696635240758,
0.24254180979536413,
-0.9408305998099434,
-15.926176434914343
],
[
0.01721780935810302,
0.9692352526748963,
0.24553324013896363,
7.8069388140706595
],
[
0.0,
0.0,
0.0,
1.0
]
],
"colmap_im_id":14
},
{
"file_path":"images/2616.jpg",
"transform_matrix":[
[
0.9367652645482465,
0.06555439110847552,
0.34376367018963866,
1.752089086403888
],
[
0.33802595877024866,
0.0848274703909408,
-0.9373061140654786,
-0.18522699767214082
],
[
-0.09060513414430646,
0.9942368541114692,
0.05730434183568592,
-0.2446859384790211
],
[
0.0,
0.0,
0.0,
1.0
]
],
"colmap_im_id":15
},
{
"file_path":"images/2917.jpg",
"transform_matrix":[
[
0.5109149170601622,
0.008185497969738711,
-0.8595923133371974,
-19.047031369140104
],
[
-0.8480028237921291,
0.1687171748883521,
-0.5024198699676082,
-4.9440612706440765
],
[
0.14091542983641886,
0.9856305152138304,
0.09314144680719047,
1.4995949329568972
],
[
0.0,
0.0,
0.0,
1.0
]
],
"colmap_im_id":16
},
{
"file_path":"images/3064.jpg",
"transform_matrix":[
[
0.9273008922603315,
0.013600882385027945,
-0.37406960744163836,
-3.7092495637182714
],
[
-0.3735407178236307,
0.09794186215691611,
-0.922428709313106,
-0.9985153665915756
],
[
0.024091229545199995,
0.9950991948723424,
0.09590205953688483,
-0.009991324859579633
],
[
0.0,
0.0,
0.0,
1.0
]
],
"colmap_im_id":17
},
{
"file_path":"images/5.jpg",
"transform_matrix":[
[
0.98894126695783,
0.0010595069134651477,
-0.14830390403810095,
-1.3443514171466973
],
[
-0.14827754096055015,
0.027225227695202324,
-0.9885709675201052,
1.2298767028500044
],
[
0.0029902097809862524,
0.9996287633026797,
0.0270812522498681,
-0.34892139763097163
],
[
0.0,
0.0,
0.0,
1.0
]
],
"colmap_im_id":18
}
]
In https://github.com/autonomousvision/sdfstudio/blob/master/scripts/heritage_to_nerfstudio.py#L87 there is a workaround for this, however, it requires a config file with these parameters: radius, min_track_length, voxel_size, origin.
In https://github.com/InternLandMark/LandMark?tab=readme-ov-file#prepare-dataset define this format of transforms.json also for multifocal sfm, see:
### single focal example ###
{
"camera_model": "SIMPLE_PINHOLE",
"fl_x": 427,
"fl_y": 427,
"w": 547,
"h": 365,
"frames": [
{
"file_path": "./images/image_0.png",
"transform_matrix": []
}
]
}
### multi focal example ###
{
"camera_model": "SIMPLE_PINHOLE",
"frames": [
{
"fl_x": 1116,
"fl_y": 1116,
"w": 1420,
"h": 1065,
"file_path": "./images/image_0.png",
"transform_matrix": []
}
]
}
Already opened an issue in nerfstudio https://github.com/nerfstudio-project/nerfstudio/issues/3057
Merged in nerfstudio in https://github.com/nerfstudio-project/nerfstudio/commit/db93476b6a3d7ae0114f3c45f0ce70a3a15853be pending to make the pull-request in sdfstudio