2D to 3D model / video generation leads

[Ross Scott]

These are some leads I've had on 2D image / video & model generation.  This is not the dream software, but it is an interesting halfway point I'm seeing how far I can take.  Some stuff worth checking out:

 

Best 2D to 3D depth map generator I've found (Midas v3):
https://github.com/isl-org/MiDaS

 

Instructions + code for a way to fix the flickering from Midas, however, I believe it is only for v2 of Midas + the instructions are only for Linux, but the code is apparently cross-platform:
https://colab.research.google.com/drive/1YOLXsb4JUOD1wt5TXB2ln78koocm-bu8?usp=sharing#scrollTo=zUYrfiNRfTUv
https://github.com/facebookresearch/robust_cvd

 

Method to generate model from a depth map + contains links to open source code on how this was made:
https://depthplayer.ugocapeto.com/

 

Sample content I made containing footage + different versions of depth maps to experiment with (about 330MB).  I generated it with Midas v3, and I used 2 different temporal blurring scripts on the depth maps in order to balance out the flickering:

https://mega.nz/file/H6BERRqZ#QWuZlHimMmq83zy9AZpMPnVsui7WrRCNZiXGNBGKG0M

 

There's also the program StereoPhoto Maker + Depthgate, but I found that while they do an okay job, the depth can't be pushed without skewing the convergence too much, which can hurt your eyes.  I think converting to actual models that can be manipulated will have much better results.

STUFF TO SOLVE:

1. Figure out how to get the anti-flicker code working (preferably for Windows 10, but Linux would be good also).
2. Figure out how to have a dynamically changing model based on the individual footage frames + depth maps.  Maybe modify the code from the model generator?
3. Figure out how to export above models to VR + and / or batch convert it to frames for stereoscopic video.
4. Integrate the Midas v3 depth map generation code with the anti-flicker code (this one is a longshot and may not be doable).

 

Feel free to reply here or email me ([email protected]).  I'll more reliably notice the emails.  Work on this could go towards a future video!