For all three of my regular readers, sorry for the slow updates. I've decided to make the site a little less newsy and more about what I'm learning on a day-to day basis.
And I've been learning quite a bit, because about three weeks ago, our internet usage spiked incredibly - saturating our T1s out to the internet and slowing traffic down to a crawl.
The chart below shows the traffic flowing
into our district from our ISP. Note the sharp increase on Weds in week 15 - that's when the trouble started.
I first became aware of the problem when our AP and payroll clerks came to ask why their terminal screens were taking 3-5 seconds to display
every letter typed they typed. I checked the normal things - CPU utilization on my router and ATM switch (they would be high in a virus outbreak), Checked my
Fluke One Touch to check for broadcast storms or excessive errors, and checked my mail server for signs of spamming. Nothing seemed out of the ordinary - low CPU utilization, no network problems, no spam.
I called the Lead Tech at my ISP to see if he saw anything going on out of the ordinary. He checked for PTP traffic, spy/adware traffic and streaming media and found just standard web traffic going on - his traffic shaping box didn't see anything out of the ordinary. I decided to give it a day or two and see if it settled down on it's own - while figuring out what my options were if it didn't.
Monday found the traffic still heavy and me scrambling for solutions. I decided to set up
Squid on the
SuSE 9 Enterprise Server I had recently set up for testing the possibility of migrating my web server to Linux. The install went fairly smoothly with
YaST, and I had Squid up and running in a matter of minutes. The only problem is, that it comes programmed to disallow all connections by default. I just needed to edit the config files and allow the machines in my network to talk to it.
At this point in time, I decided to install
Webmin, as it simplifies this type of task tremendously. Unfortunately, it isn't included in the base SuSE packages, but I was able to download a
RPM off of the site and get it up and running in no time. After a bit of ACL tweaking, I was able to point my workstation to the proxy and hit the web.
I used Windows group policies to set a couple of grades to use the proxy Monday afternoon, and let them run Tuesday to make sure the server didn't choke on a fair-sized load. Tuesday also kept me busy setting up the Fluke
OptiView on loan from my ISP. Initial tests with the OptiView showed nothing out of the ordinary. I set the proxy as on and active in all Windows Group Policies later Tuesday afternoon.
I was rewarded with a greatly reduced traffic load on Wednesday (week 16). Telnet traffic was smooth again, and the web lost the slow as molasses feeling it had developed. I wasn't quite back to pre-spike levels, but I had seen a lot of streaming video form ESPN, and hoped that blocking it would bring me where I wanted to be. I patted myself on the back and decided all I needed to do was decide on if I should use the linux/squid combo going forward, or if I should pick up a commercial appliance that would do the same.
How wrong I was....... the gory details are forthcoming in my next installment in which we see bandwidth continue to rise, and the real culprit unmasked.
From Om Malik's weblog:
