Fixing this problem is very simple all you need is just go to "Downloads" Then --> "Speed Limiter" Then-->Turn off". That's it, Speed limiter feature in IDM limits the download speed to some value assigned.
How to Fix Very Slow Download Problem in IDM?
Other things Which cause this problem:
1. A site/blog you are downloading from.
2. The time of the day you are downloading at.
3. Other Internet applications that download are downloaded from same time.
4. Your Internet connection.
5. Your Local Internet service provider (ISP).