Another aspect of this risk at higher altitudes is your ability to get down
quick enough in an emergency.
I know most of us don't like to go idle power and point the nose down
looking for VNE.
Like other emergencies, we need to be mentally prepared when the need
arrises.
I did it once when an O2 tube came off at a connection when at 19,000 ft.
In about two breaths I felt pretty loopy.
I tried to reconnect the tube and found myself fumbling with it.  Then I did
the screaming dive.
It takes a LOT longer to loose 10,000+ ft than you might think.

An emergency O2 bottle becomes a good idea at higher altitudes.  In the Air
Force, we carried a small tank as part of the chute when operating above
25,000 ft.

Mark Ravinski
N360KB 1390 hrs