when my car idles it puts out about 13.8 volts (normal) but when i turn the stereo right up it sits at 11.5 volts constantly. could this a problem for the car down the track?

Yes it is a problem.
Need a bit more info, what amps? how many?

What size cable to the boot? 0 gauge? 4 gauge?

i've got 2 400w clarion amps, one runs my speakers, one my sub. the power cable is preety thick but i'm not sure what guage. does this help?

First of all where are you taking the reading at the boot or at the battery? if there is a large drop at the boot its either bad connections or you need a lower guage cable to carry the amps (less voltage drop)
If the drop is at the battery it self i would be looking into a new or larger altenator, also renewing the battery depending on its condition.