By Cutter Kilgore and James Hearne
Network outages that caused difficulties for students and staff at Whatcom Community College for the first two weeks of fall quarter have been resolved.
The culprit was a faulty piece of software used in five of the labs on campus, Ward Naf, head of Whatcom’s IT department, explained in an email.
The software is used to track student computer usage, and has been used by the college for years. However, it was modified over the summer.
“This modification seems to have caused some very bad behavior in the way the software communicates that was not noticed over the summer due to decreased traffic,” wrote Naf.
But on the first day of fall quarter, with all labs in use and all computers on, the malfunctioning software caused enough corrupted traffic to cause the system to stop responding.
The phones were on and off, the debit card machines at the student bookstore and cafeteria were down, and email was largely unavailable. On day one, Doug McKeever’s geology class lasted only 15 minutes because of intermittent network availability.
“This is the worst nightmare the IT department has ever experienced,” said Linda Compton-Smith, the library’s lead of circulation.
On Wednesday, Oct. 3, the situation baffled many. “IT really has no idea what is wrong with our server,” Kimberly Reeves, a Whatcom faculty member, wrote in an email to faculty that day. She wrote that she’d had a long conversation with Scott Carter, of the IT department.
“Even more alarming is that the company that makes and services the servers has no idea what is wrong with our system,” she wrote.
The issues being dealt with after the initial two weeks were mostly isolated and not as widespread, said Nathan Langstraat, the interim vice president for administrative services. “IT’s been working very hard,” he said. “They knew what the issues were; they just needed to isolate them.”
On October 4, Langstraat sent an email regarding the issue:
“[…] through the great work of our IT folks, we believe that the source has been identified. It appears (although further testing and analysis is needed) that software used in five computer labs on campus (in four different buildings) caused “network storms” to occur. The software application is being further reviewed today to confirm that this is indeed the source of our issues.
The network is stable at the moment. There continue to be some technical issues, but most are cleanup from the original network problems. IT is working hard to resolve them one by one.”
Langstraat said he feels bad about the impact that the network issues have had on students and faculty. “Everyone’s been extremely patient and understanding,” he said.
Piles of anecdotes about frustrating experiences can finally be put to rest. Experiences like those of JT Taylor, an instructor, who said he had trouble checking his email on campus for the first week of the quarter.
Compton-Smith said Tuesday, October 4, that the Internet connection was intermittent all day and that network issues were campus wide, but each building had been affected differently and certain areas were worse than others. She said the team thought that they had fixed the problems, only to have them come back. “Usually, they can overcome these sorts of things quickly,” she added.
Koichi Hirata, who works at the Student Life desk, said that due to network outages on Monday, Oct. 1, they were intermittently unable to make student identification cards, which was an obstacle for many students. “We were kind of stressing people out, and we don’t like that,” he said. By Wednesday, student identification cards were available again.
Jon Spores, the manager of the student bookstore, said he and his staff found ways around the limitations, such as using cell phones to place necessary book orders. At one point when the bookstore couldn’t accept debit cards, the staff had to send students off campus to the Food Co-op, as it had the closest working ATM.
Whatcom’s IT department consists of seven full-time employees and several part-time ones. Naf wrote that they “worked to resolve the network issues constantly, putting in many 12 to 14 hour days in an effort to alleviate the problem.” They also used outside technical experts as a resource to get things back on track.
“IT is currently working with the software manufacturer to identify the reasons the software had this behavior and to fix it,” wrote Naf. “They are also working with the network equipment provider to identify ways to keep this from happening again and to verify the configurations and make any changes that are necessary.”
