But if you implement all of that perfectly, doesn't that defeat the whole purpose of having an open office? (I.e. to stimulate more spontaneous collaboration)
Depends on the kind of work you do. If you're in sales or marketing the open office plan may indeed stimulate creativity or whatever.
But for people who have to concentrate (like software developers) it's a productivity killer. Joel Spolsky used to touch on this occasionally - he claims all sorts of research shows the most productive arrangement is to put everyone in his own office with a door.
Where I work that will never happen, because there are corporate-wide rules about who gets an office and who doesn't. You have to be a director to get your own office, which is two levels above non-managers. They're so anal about it one time I worked on a floor with no directors and they left all the offices unoccupied. They crowded everyone into cubes and even doubled one up when they ran out of (cubicle) space.
Partitions can still be installed between the work pods to provide a measure of privacy. Many of the open-plan offices I've seen have no partitions, the only thing between you and everyone else is a computer monitor. It's insane.
With partitions, at least I don't have to watch the office gumbies milling around flapping their arms and smacking their pieholes. Half of my monkey brain is tracking their movements for threats, and the other half is struggling to prevent me flinging handfuls of my faeces at them out of annoyance.
It depends on the partitions and how other surfaces are finished.
With sufficiently high walls (about 6'), carpeting, and acoustic ceilings, you'll actually cut noise quite a bit. Lower and sound travels in straight lines.
One of my gigs had a "pod" style cubical layout with 6' high walls, glass in the upper 2' or so, arranged in groups of four (see diagrams below). It's still one of the best workspace designs I've seen, period.