Bill, yes I think it would. If the shaft "moves" at all in the coupling because of the misalignment the "torque" against the "key" would oscillate from end to end ever be it so slightly. The brass is far softer and much more malleable than steel or stainless steel and would eventually break down, mostly because the brass is not strong enough in the first place. A set screw will not aid much in holding because it is at a single point. A split coupling would reduce this quite a bit but because of clamping pressure however it may indeed still break down. The "key" takes the torque and needs to be strong enough to resist.
Look at older OB lower units. The gears were protected by the "key way" or aptly named "shear pin" incase of an over revving gear shift or a prop strike.
Dave Neptune