There's probably some kind of complicated mathematical equation/algorithm to explain the spawn rates, but I'll use Zera language. As far as I've seen, the spawning rates are directly related to the amount of lag the server is experiencing. I may be (probably am) wrong about this, but that's been my experience. There is a mob cap and a mob culling plug-in to prevent mass lag due to too many mobs (hostile or non) grouping into one spot.
Mob cap: Mobs will stop spawning in an area once that area is occupied by a certain number of mobs.
Mob culling: If too many mobs (such as villagers) are contained in one area, they will be slaughtered mercilessly, never to be seen again. (At least a few of them so they're below the mob cap).
Again, this is my experience, and what I've gleaned from visually evesdropping on other people's conversations
To those who can, feel free to correct me and modify my info!
I encourage it. o_o