This was a great question! I hadn't looked into this in a while.
By default, there isn't a saved setting in a new instance of DNN. Once you save the settings in the view you're referring to, a record is created for the "Page Output Caching Provider" in the HostSettings table, but only is a value other than "Select" is chosen.
As you've noted and found out on your own, there's no clear way to simply revert this setting to the default. You can still do this though.
I'll be directing you to use the SQL Console feature in DNN.
To clear this setting, use the query below. Just copy it, paste it into the SQL Console, and execute it. After you do, you must also clear the cache of the website.
DELETE FROM {databaseOwner}[{objectQualifier}HostSettings] WHERE [SettingName] = N'PageCaching';