BY JONAS BLANK
 |
For Harvard Law students, January is the time for winter term classes (or 1L exams), miserable weather and soon-to-be-broken New Year’s resolutions. For the last two years, it has also been time for massive e-mail problems. Unfortunately, this year’s irrecovable loss of an entire day of mail made last year’s service interruptions look minor.
On Friday, January 24, Information Technology Services Director Jane Sulkin sent an e-mail to all HLS users informing them that they had just lost an entire day of mail, beginning Thursday, January 23 at 1 a.m. and extending until Friday at 2:30 a.m. Due to a glitch in Harvard University Information Services’ (UIS) CAMail system — which replaced the aging OpenMail system that caused last year’s trouble — the system deleted every HLS e-mail account.
“It was a costly error but a very common error,” Sulkin said of the problem, which involved a failure of the software to provide adequate data integrity checking.
While a robust system of data backups ensured that all but the January 23 mail could be restored,
We think it’s probably something to do with the security settings on the server side,” Sulkin said. One problem ITS faces that typical corporate users do not is the diversity of e-mail programs and software operating systems used by students. Because there are so many different software combinations being used by HLS students, Sulkin said, individual problems are difficult to pinpoint.
What is most frustrating for ITS in this situation is that the switch to CAMail, which caused the failure, should not have happened in January at all. OpenMail, the e-mail program used by HLS since 1996, was already scheduled to be replaced at the end of this year. The aging program was about to lose the support of its manufacturer, Hewlett-Packard, and could not handle the exponentially increasing volume of HLS mail, which now numbers about 40,000 messages a day. According to Sulkin, ITS may have decided to go with CAMail anyway, but the department would have had much more time to make the transition than it ended up having.
Instead, ITS was forced to make a quick switch on January 7, after a massive failure of the OpenMail system. The failure was caused by the addition of a new virus scanning server late this past fall. Despite testing over winter break, the virus scanner interfered with OpenMail’s operations, causing the system to crash.
Panicked, Sulkin and ITS decided that rather than restore the failed OpenMail system, it would transition to the new system immediately. Sulkin turned to UIS, which handles e-mail for all of the University’s central administrative offices.
“UIS was in a better position to respond more quickly,” Sulkin said. “They made the transition for us overnight. They were heroic.” The transition appeared to have worked. Within 16 hours of OpenMail’s failure, the CAMail system was online, and HLS e-mail services were restored. The new UIS system had many advantages that OpenMail did not–the system had worked superbly for the 2500 administration users it already served, and UIS had an excellent reputation for its fast service, low cost and reliable data backups.
CAMail seemed to be delivering all those things. Even after the crash, UIS backups were extremely effective–the only reason the January 23 mail was lost was that the system began backing up accounts as the mail was being deleted, a problem that has since been solved. The system was installed quickly, before discussions of payment even began.
And since the crash, Sulkin said UIS has been doing “everything within their power” to resolve the system’s problems. One of the most serious difficulties faced by all users is UIS’ weak hardware. Although the system capably handled UIS’ original 2500 users with room to spare, the Law School’s 3200 users send and receive far more mail than administrative users within the University. To rectify the problem, UIS plans to have a new server in operation by the beginning of March.
In retrospect, it appears that ITS made two critical errors, neither without justification. First, Sulkin admits that the decision to ditch OpenMail came too late. ITS planned, and the administration budgeted, for the spring 2003 system upgrade based on the advice of a team of outside computer consultants who said the system would last until then.
Sulkin hesitates to blame the consultants. “We concurred with their advice,” she said with a shrug.
Second, and perhaps more fatally, the decision to install a virus-scanning server last fall placed far too much demand on an already overstressed system. Again, the decision was not without reason. Not only were students frustrated by a constant stream of viruses, such as the notorious “MyParty” virus that flooded inboxes last year, but the additional e-mail generated by such viruses taxed the system even further.
Students still frustrated by the new CAMail system will be pleased to know that it may not last beyond the spring term anyway. At that time, ITS will decide whether to adopt the system on a permanent basis or whether to migrate to yet another system. In the meantime, UIS is vastly expanding its infrastructure, with a new data center being built at the corner of Everett and Oxford Streets that will house University servers and backup systems.
Sulkin also notes that students should consider using e-mail programs such as Microsoft Outlook or Eudora, which can download copies of mail directly onto students’ computers while also leaving them on the ITS servers.