If you have good eye sight a traditional muzzleloader is not a big disadvantage unless you like to shoot 200 yards and need multiple shots.
I like to muzzleloader hunt too but old eyes keep my traditional muzzleloader shots to 50 yards or less. I can shoot twice that with a scoped inline but I like traditional better.
I've done the "only muzzleloader" thing many times over the years. You just have to commit yourself to it, for good or bad. If you just have to kill a deer every year then it's probably not for you.
As it is I hunt with a flintlock probably 85% or more of the season anyway, if I don't have it with me I miss checking my prime.
There are just sometimes you need to shoot a scoped rifle to be effective. I no longer have a problem doing that. That's when my .308 Tikka w/Zeiss scope gets the nod.
Keep the wind in your face and you will do well.