A couple down sides not already mentioned:
Cost. Braid costs more.
Knots. Good braid knots are harder to tie and take more practice in the beginning. Because braid will slip and or cut itself.
Abrasion. Braid will wear easier than mono (it will look fuzzy) You have to watch it closely around cranks and the releases. The loss of only a couple of the spectra fibers will drastically reduce it's strength.

That said we prefer braid for the reason you first mentioned, the lack of stretch. We use releases with all our inline board and with braid a quick little pop will release the boards as long as you twist or wrap the release as Bob mentioned.

To each their own, both work well. Try them both and use what works best for you.