I have an AMD Ryzen 5 5600X which was the best CPU I could find when I was building it (outside of something in the 4-digit range for just the chip). I messed around with it more and I didn't try the downsampling yet, but changing it from Quality to Balanced made a huge difference. However, I remembered another reason I turned RTX off - no matter what the other settings, I'll just randomly get massive frame drops for 10-20 seconds (even in menus) and just have to pause and wait it out. Happens semi-randomly every 8-10 minutes. I had already forgotten it did that, according to the internet it just is a thing some people have to deal with I guess. I'm tempted to try on my other PC too and see if it is hardware dependent.I just checked and all graphical settings are basically maxed out, including all the ray-tracing options, and DLSS is on quality mode. I'm downsampling with DLDSR, which I think is supposed to be less demanding, yeah. Maybe give that a shot to see if it makes any difference for you? Other than that, I'm on a 3080 with a i9-12900K. What CPU do you have? Perhaps the game is CPU bound somehow?
edit: I should mention I'm not getting a constant ~100fps. Most of the time it's closer to 90 and it regularly drops into the 80s and 70s depending on what's happening. I have a g-sync monitor, though, so as long as it stays above 60fps I don't really notice the drops.
edit2: Actually, I've been playing some more and I started getting serious drops after setting level streaming distance to 4 (I had it at 1 before).
It does suck because the RTX adds a nice level of depth to the game - it looks great without it but it looks ever better with it.
Edit: Odd, most optimizations showed the streaming distance having next to no impact, so I set it to max and didn't see any FPS loss myself. Maybe it doesn't play well with RTX