For me it accompanies the downbeat feeling of the film, much like Dead Man, which works even better with the fades.

I think the only reason it's distracting is because most people are used to standard directing with simple cuts or wipes.