Stop all of your torrents, turn off anything using network bandwidth.
Go to http://speedtest.net/ and post your results here.
Who is your ISP, and figure out your line speed while you're at it.
180's a respectable speed. Now, I imagine you've got at least a 10mb NIC - this is an ancient technology - so - you must have it. This allows for 5/5 throughput. Full duplex, 10/10.
Things to consider:
A- know your max line speed - find that out. Turn off ALL connections when you measure. Call your ISP if you need to.
B- What is the throughput of your DSL router/Cable router? ( This can be a bottleneck. Say it sucks = and ask your ISP to swap it for you, if it's leased. )
C- Do you have a hardware firewall? Consumer-grade firewalls usually have
VERY POOR capacity (Anyone - just try and prove me wrong. You already failed. If you can buy it at Staples/Officemax/BestBuy/etc - I don't care how much you paid for it - it's
consumer grade. It sucks.) Very often, they are bottlenecks. The more connections going through your firewall, the slower it is. A good way to solve this issue - is to get an old enterprise grade router used. Like a snapgear (my personal fave), sonicwall, juniper, watchguard, etc. You can get one for a decent price. Even a 5 year old unit - is often BETTER than the shit you can pick up at the store.
D - Your network is as good as your hardware. Are you using wireless? If so, get wired. Are you using cheap hubs? Get a good switch. Don't skimp.
I got good hardware here - and I can seed at 100% of my upstream speed - AND - download fast as hell on top of it. I'm dealing with a 20/6 connection here too.

So - that's a shitload of data moving. My internal network - It's damn close to as fast as drive access. It's killer.
A lot of basic isp's give you a high download and low upload cap. mines 16mbps/2mbps (speed boost can make it a bit higher sometimes) with comcast and I have the same 180kb/s upload cap for torrents.
Also - make sure your client is set up to randomize the download ports and/or encrypt your data so the ISP can't detect torrent traffic. If you're using an out of date client, get a new one that supports it.