That's hard to say since it's so dependent upon each individual tastes/producers/style/wine and even bottle variation.
The only generic "rule" I can offer is based on the ABV.
The higher the ABV, if served for immediate consumption, I personally feel should be colder. Like below room temperature (18C) and letting the wine warm up in the glass. If serving at room temperature or warmer, then i'd have the bottle opened and decanted for a longer period of time to let the alcohol blow off.
From each individual location:
Alot of sherries are typically served warm because they've already been so heavily oxidized that serving t hem chilled doesnt make them taste that much better, but again that's my personal tastes.
Vintage ports are served coming out at cellar temperatures, then decanted for a while afterwords where they're basically at room temperature.
White ports are typically chilled because they're relatively cheap, but the nice white ports with some wood age to them will be served near room temperature.
Madeira has been drunk when it's insanely hot out and doesnt make a difference when youre sweaty and hot. I personally let my madeira colder so that more of the acidity shows through.
Ruby ports are served usualyl chilled too but some places that have had the bottle open for weeks will just serve it on the warmer side.
