[tex-live] Another service interruption at Utah

Nelson H. F. Beebe beebe at math.utah.edu
Wed Jul 6 03:07:22 CEST 2016


I'm sorry to report that we have had another service interruption at
Utah, this time a fairly serious one.  In the wee hours of Sunday
3-Jul-2016, smack in the middle of the most popular summer long
weekend in the USA, the motherboard on our tertiary fileserver, a
summer-2005-vintage Sun Fire E2900, failed.

That system hosted our FTP site, ftp.math.utah.edu, as well as
providing storage for backup savesets, numerous virtual machines, and
a software license manager.  Sadly, the tight budgets that we always
suffer from in academia have made it impossible to keep that machine
on a hardware maintenance contract, so we are in the process of moving
those services to other systems, and retiring the Sun hardware that
has otherwise performed reliably for 11 years, and in that time,
served files to more than 100,000 student accounts.

FTP services were restored on another system that came up at 16:09
Monday 4-Jul-2016 and has been stable since then.  No FTP-able files
were lost, because the motherboard failure happened a couple of hours
after the failing machine's nightly filesystem snapshot completed
around midnight, and was propagated to another system.  That snapshot
was used to populate the replacement server.

We are in the process of bringing up a remote snapshot storage system
at a datacenter several km from our main campus.  That should provide
us with much-needed storage for a longer backup cycle, as well as
providing off-site copies of all system and user data that are updated
nightly, to guard against building disasters (earthquake, fire, flood,
sabotage, ....).  We are also looking into the possibility of that
remote system providing an emergency fallback server for our normal
FTP and HTTP servers, allowing much quicker fail-over than the 36
hours that it took us this time.

The failure did not affect most HTTP services, but the FTP tree, which
has always been visible from

	http://www.math.utah.edu/pub

was inaccessible via HTTP until FTP services came back.

-------------------------------------------------------------------------------
- Nelson H. F. Beebe                    Tel: +1 801 581 5254                  -
- University of Utah                    FAX: +1 801 581 4148                  -
- Department of Mathematics, 110 LCB    Internet e-mail: beebe at math.utah.edu  -
- 155 S 1400 E RM 233                       beebe at acm.org  beebe at computer.org -
- Salt Lake City, UT 84112-0090, USA    URL: http://www.math.utah.edu/~beebe/ -
-------------------------------------------------------------------------------


More information about the tex-live mailing list