The software of my dreams

I've got multiple dream softwares... a lot of these tie into "The GUI should be better", so...

1: A discord client with proper customization. I don't like how *flat* discord is and the only other discord clients I know of are Ripcord (which I'm using right now) and Better Discord (which is a pain in the ass!)

2: Something to make Windows 10 MUCH more customizable. Once again, flatness blows. Give me Aero or give me death.

Hey Ross I came up with an idea/suggestion for the next Freeman’s mind episode!! How about including the all knowing vortagaunt after the part where you destroy the chopper. If you don’t know what I’m talking about you can search it on YouTube. Of course it’s  completely up to you. Just an idea 

Ross: I've put a big reply up on my site, with some detailed thoughts on technical direction, unexpected outcomes and ethical issues relating to 3D gameplay capture and 3D world reconstruction:

 

https://halestrom.net/darksleep/blog/044_3dto3d/

 

 

Apologies to everyone who has been commenting here, I have not read through things yet.  I spent quite a bit longer on this reply than I anticipated.  I'll get there.

 

I only found out about Ross' video a few days ago; Ross' RSS feed is being blocked by his site's Cloudflare setup, so my (and everyone elses'?) feed reader can't access it.

On 1/23/2021 at 9:35 PM, Veyrdite said:

Ross' RSS feed is being blocked by his site's Cloudflare setup, so my (and everyone elses'?) feed reader can't access it.

It's probably because someone is trying to DDoS the site, or trying to bruteforce passwords.

If anyone is interested, I can do some work on prototyping an application for this, what I would need is somebody to generate some training data.

 

First, to deal with the "shadow problem:" If someone can generate screenshot pairs of game screens with lighting turned on, and lighting turned off, then we can use that to train a model to de-shadow an image, producing a texture layer and a shadow layer.  This should be done programatically so the screenshots match up exactly - so you would record a demo of walking around a map, and then on playback get the engine to screencap the original screen once and then again with lighting disabled.  This can also be done to the spectral highlights as well, but it's best to start off basic first.

 

I'm not familiar with photogrammetry algorithms, be we can give that a shot.  Off the top of my head, we can create another model to add depth information to a screenshot (and perhaps even better if it's from a video, but for now it's better to focus on just the screenshot case first), with the dataset again being generated the same way as in the above, just with a depth-based shader being used instead.  Stitching together the depth-based images into 3d space should be easier.  Otherwise, there are tonnes of other algorithms to look into as well (eg: https://github.com/natowi/3D-Reconstruction-with-Deep-Learning-Methods ).

 

So if anyone wants to take the dataset task on, or if anyone has old gaming hard to devote to this, post here I guess.  For anyone interested in the deep learning side of things, trying to implement some of these algorithms could be a good way to get into AI stuff.

 

 

Finally got around to watching this.

 

This got SO CLOSE to my own dream software: real-time world creation.

 

Imagine a game that created itself AS YOU EXPLORED IT. It would take data from real places, other games, etc, and throw it at a super-AI that would then generate more world on the fly. We're talking machine learning based on real places, not just random procedural stuff.

 

I imagine an MMO game where there are literally no limits to where you can go. The game could take place in a city, initially, but someone could hop in a car and drive past the city limits and the game will start filling in the surrounding country-side, and even generating a neighboring city. Then the players take it from there and breathe life into them by, say, moving into the houses and apartments, taking up jobs in the commercial buildings, etc. You'll feel like you're exploring a real place, which also makes it perfect for VR! All you have to do to determine what content is generated is to pick what initial data is thrown at the AI (i.e. sample images, 3D data, etc).

 

I actually think this is within our reach with machine learning in its current state. The issue is server space. Whatever content is generated would need to be stored and preserved for consistency, and that data is potentially infinite. We're talking Google-size server space here. But the end result would be a video game completely unbounded in where, and how far, you can go.

On 2/15/2021 at 12:49 AM, Ganymede105 said:

The issue is server space. Whatever content is generated would need to be stored and preserved for consistency, and that data is potentially infinite. We're talking Google-size server space here. But the end result would be a video game completely unbounded in where, and how far, you can go.

Easy fix: Make it a limited size sphere of a world, rather than an endless flat expanse. Assign different structure parameters to smaller indicative symbols, and it can recreate the world on the fly, without actually storing the entire world shape-by-shape.

I sent an email to Ross a while back asking for his thoughts, but I haven't gotten a response. So I guess I'll just do a public post here.

 

The short version is: Have we considered using voxels?

 

If anyone here isn't familiar with them, voxels are basically just pixels in 3D space. You've probably seen them in some retro-esque 3D games that are going for a visual style similar to something like Minecraft. Cube World is probably the most obvious example I can think of.

 

However voxels have much, MUCH more versatility than just blocky retro-throwback stuff. Computers now can actually render really detailed models using voxel geometry instead of polygons. And for a while, some lesser-known titles were using voxel based engines as opposed to polygonal ones to try and get more detailed environments. Nowadays because Polygons have become the standard and additional levels of detail aren't really that big of a deal anymore, Voxels have kind of fallen into a niche. Usually they're used for simulation games where updating polygonal 3D models on the fly aren't really an option. I'm sure a lot of you are familiar with Teardown, and in my search I also found this: 

 

However both of these games seem to be low-balling what voxels are capable of.

Here's a tech demo I found of a tank, and this is from over five years ago: 

 

 

And here's an example of Voxels being used to create realistic terrain on the Nintendo DS: 

 

 

I bring all this up because I feel like this is the easiest compromise for reading a 3D world off of a video file. For one, the AI wouldn't have to carefully construct polygonal models for everything it finds. It just needs to replicate what it sees: Pixels. And two, we don't have the special engines with advanced optimization tricks needed to generate those huge, detailed open world games... but voxels are capable of having a very crude level of detail system. As the object gets further away, just merge the voxels into fewer larger ones. 

 

The main thing I'm worried about would be Ross's obsession with Anti-Aliasing, because (and correct me if I'm wrong) I think most of the proper Anti-Aliasing systems in games these days rely on the model and texture data to get the best result. FXAA could probably be decent general substitute, but if you're in VR and you mash your face into something it's just gonna look weird.

Relevant:

 

I think that it's great video!

I've found a channel that does these videogame walks https://www.youtube.com/channel/UCBX_wRb6MXqG3WKIfseM4oA

Now I know, it's not the same as walking in VR yourself, but it's a nice background video that could be a substitute if you have no access to a real Dream software.

this is solved now Luma AI - Video to 3D API (lumalabs.ai)

"This is solved now"

 

bruh, no it's not

 

the advances in neural net tech have been impressive (and make a lot of the ideas in Ross's video seem achievable), but they clearly aren't at "solution" level

Yeah, maybe not all parts of dream software are done, but "capture gameplay video -> 3d static scene" seems to be possible with things like F2-NeRF today.

This is what Luma AI was able to create from about 20 seconds from this moment (from 0:02:41 to 0:03:15) out of a random videogame footage.

 

 

And here's attached zip file of a low-poly (because of the forum 8Mb upload file size limitation, full hi-poly geometry is about 70Mb in this example) 3d OBJ file that came out of it. With proper game settings (like first-person view and intent on actually capturing all angles of the level) the results would be better.

 

This is what it looks like in 3D view (note, this was obtained from a YouTube video, not from GPU video memory locally):

 

83ac1cf7fd3e0ff01696cde41dd81cc06e5c8c3970489109a65bde6c94c16d22.zip

lo and behold. Luma did exactly what Ross was asking for, easy import of captured 3D data into Unreal

Quote

Luma AI on Twitter: "Introducing the LumaUnreal Engine alpha! Fully volumetric Luma NeRFs running realtime on Windows in UE 5 for incredible cinematic shots and experiences, starting today! Try now: https://t.co/mdyGTWhhgN https://t.co/Lj1Lm3F2P4" / Twitter

On 3/30/2023 at 5:59 PM, kpcg said:

Yeah, maybe not all parts of dream software are done, but "capture gameplay video -> 3d static scene" seems to be possible with things like F2-NeRF today.

Sorry! I was definitely being a bit aggressive before. It IS cool work,  the language of "solved" just put me off a bit, because this is a thing I think is both important and interesting, and sometimes seeing a work-in-progress sold as a solution gets under my skin, makes me worry that it's not gonna continue improving.

 

You see it in reverse, too! People talk about Y2K like it was a fake problem people were wrong to ever worry about, instead of an issue that a lot of people worked very hard to prevent from being too destructive. We're probably gonna see the same mentality brought out whenever we get around to climate mitigation.

On 4/4/2023 at 11:27 AM, Shaddy said:

Sorry! I was definitely being a bit aggressive before.

I miss angry Shaddy

heres more in this vein https://www.worldlabs.ai/blog