Since vermillion pheonix sword if for all classes (I think it is) does it matter what the class is on the Avalon for increase damage?
Nope, you can be any class you wish. Only Class-Exclusive Exocores will only work with their respective class's Avalon.
Upgrade Skill of All Class Exocore & Enhanced Skill Attack will work. However, if it's Upgrade Skill of "Shaman", it won't work on the Vermilion Phoenix Sword.