No don't worry about difficulty. All you need to worry about is making sure each spawning cell is the correct distance from the others so they don't merge into a single village. Also, if you build a very large farm, you will need to use minecart rails that keep you close enough to the spawn cells for the golems to move.
