I went for AIO because at the time I had a Hyper 212-like on it that couldn't handle the heatload, and nothing else that could was in my price range at the time. (Thermalright stuff wasn't available anywhere for less than the AIO back then) That's the main reason. I also am a fan of Arctic's fans.
Nowadays, I recommend Thermalright to anyone without hesitation as they are inexpensive, available, and performant. (and most look pretty neat too)
There are full ATX motherboard capable cases in the mATX desktop size. They just shift the layout a bit, and are generally a little harder to find, but usually around the same price. It's almost always cheaper to get one of those and an ATX board than go full mATX with everything if you want/need the desk space. (I got an incredible deal on both my motherboard and case, or I would've gone full ATX for both)
Now, about the RAM...
Honestly, if you can get G.Skill RAM, it's going to be more stable at better timings than any other RAM that will work in your system, especially if you can get 3600 with 16-16-16 timings. (if it's available, it shouldn't be more than $20 more than the 18's, though it's been a few years so prices may have changed) When it comes to RAM, stability is infinitely more important than speed. Also, the 5800X's memory controller won't do over 3600 on the RAM unless you have a golden sample chip, and you'd've had to gotten really unlucky for it to not be able to do 3600. (basically, always look exclusively for 3600 with the tightest timings possible for the 3000 and 5000 series AM4 chips) In addition, don't think that you can take higher speed RAM and turn it down to the 3600 sweet spot, unless you're willing to sit there and manually tune it for several weeks. (not exaggerating, that's what it took to find a slightly better tune to the timings of my RAM that's already at 3600)
If you're not aware, the timings and the MT/S speed determine the latency, which is the actual speed you need to make as low as possible. Going to a higher MT/S can look impressive, but generally the timings are significantly worse, and result in significantly worse RAM latency, which means worse performance. A good source for learning more details about this is here:
https://hardwaresecrets.com/understanding-ram-timings
Don't mix RAM kits if at all possible. I got incredibly lucky that my two 2x8GB kits were stable after tweaking a few timings, and that's with identical kits that ended up being from the same production lot despite buying them over a year apart. Your best bet is to get a 2x16GB kit, as that'll save you a LOT of headache in the upgrade if you're not a silicon lottery winner.
As for the RGBness of the RAM, I've personally never seen the appeal. I mean, sure it looks interesting for the 10-15 seconds you're looking at it instead of your monitor, but is that really worth the extra money you're spending on it? It's not adding any performance, and frequently increases the heat generated. (which can in some cases either reduce the lifespan or performance of the RAM modules) How much time overall are you going to spend looking at the inside of your case VS how much time you're going to spend using the system for doing things... Only you can decide those.
