A bass guitar signal can most certainly damage the speakers in a guitar amp. Especially if its an open backed cabinet. The cone movement is greater at low frequencies, and there's not much restraining the cone, so the voice coil can actually pop out from between the magnets or the cone tear. A bass speaker will typically be ported or sealed, which provides some protection for the driver.
I have seen an explanation of how a bass signal with its extra energy could (IIRC) somehow overwhelm the output transformer of a valve amplifier, but I didn't understand it and can't remember it so your guess is as good as mine how much truth there is in it. My utterly uninformed guess is you'd need to be thrashing the amp though.