Personally, I think installing the belts was a good idea regardless.

But can you be a bit more specific than "the brakes SUCK"? How so?
I'll grant you, there are relatively easy upgrades, all of which bolt right in. For example, you could use the larger master cylinder from an early GT6. You could upgrade the fronts to either the Mk3 and later Spitfire calipers, or upgrade to a complete caliper/rotor/hub, etc., assembly from a GT6. You can upgrade the rears to 8" drums from an early GT6. There is a wide variety of pad and shoe material available.
That said, I've never found early Spitfire brakes to be lacking in any way. (Heck, I've got an all-drum-brake Herald, and I don't find those brakes lacking in any way!)