well, whenever I can get away with it I use a simulation.
I made one brick and made a duplicated it till I had a wall, then (as you can see in the wireframe) I have an attractor near the base of the wall that pushes outwards, and that gave a nice scattering effect. for the other wall I just let it fall apart.

I used a very low substepping value for the simulation and the only some of the wall, and the middle castle are being used as obstacle so I could move the forces around in real time and didn't have to worry about pre-simulating anything.
And there you go, scattered bricks

Also, I think we've found (from the other speed modeling challenges) that extending deadlines never really produces any extra results for some reason.