Drag based refactoring is unexpectedly powerful. One must be able to cautiously explore this power without getting inexplicably stuck.
Small moves shouldn't count as drags. One safeguard would be to require dragging items some distance in order to break them free.
A drag on the surface of an item could bubble up to the usual text selection logic until the cursor leaves the bounding box. Drag animation begins with the larger move.
Touch interfaces will change all of this.
It would be sweet to have some sort of global-undo that could reverse an inadvertent refactoring such as dragging an item onto some random page and causing multiple forks and updates in the course of the change. People wouldn't trust an undo unless it always worked. That's the hard part.