I feel similarly about those series as well.
With RDR (the first one at least), the world was so well realized that I didn't mind not having tons of side stuff to do. Although I absolutely loved finding the hidden treasures. Those were done perfectly for me.
But to put it in perspective. I'd rank Oblivion (my favorite ES title) quite a bit above RDR, so I'm pretty much on the same page. (But I love both of them very much.)