Personally - If you are not having problems with the current driver you only need to uninstall via Programs and Features, reboot and install the new drivers. If you are having problems and have done several re-installs then after rebooting it sometimes helps to run Driver Fusion to clean out leftover files/registry entries.
The AMD/NVIDIA/INTEL video drivers can normally be uninstalled from the Windows Control panel, this driver uninstaller program was designed to be used in cases where the standard driver uninstall fails, or when you need to thoroughly delete NVIDIA and ATI video card drivers.
When you installed your current driver, if you first uninstalled the previous driver you won’t be able to roll back. In this case, follow the same steps that you previously used to uninstall your driver.
I also tried to install and uninstall the expected version (NVIDIA-Linux-x86_64-470.86.run), but this one ofcourse does not support my graphics card, so the installation does not complete and i cant uninstall whatever is causing this version missmatch.
Based on my own experiences, I'd say that there is generally no need to uninstall old drivers when installing new ones for the same card as the installer takes care of everything automatically.
It is recommended to uninstall current driver before moving backwards in driver versions however, because there is no guarantee that old (un)installer routines are compatible with newer drivers. For the most part though, it's not a big issue.