Washington, DC – 17 September 2019
Symplicity Government Division is providing the following service update regarding the Online System for Clerkship Application and Review Program (OSCAR). The Symplicity Government Division hosts the OSCAR system on Amazon Web Services (AWS), a service that suffered a power outage on Aug. 31 affecting availability to OSCAR users and related issues which are summarized below.
At approximately 9:15 a.m. EDT on Saturday, Aug. 31, the AWS facility where the OSCAR system is hosted experienced a complete loss of utility power affecting all AWS clients in this facility. According to AWS, the facility subsequently suffered the failure of the fuel delivery system which supplied the facility’s backup power that then caused the deactivation of HVAC systems. The loss of facility cooling from the HVAC resulted in severe damage to a number of the AWS network switches, networking and data storage systems, impacting hundreds of AWS clients in the facility.
Throughout the AWS outage, the Symplicity Government Division team worked closely with the AWS support team through the resolution of this core issue. After Symplicity’s access to the AWS services was restored at approximately 7:00 p.m. EDT on Saturday, August 31, the OSCAR system was restored by approximately 10 p.m. EDT allowing all users to log in and resume access to the OSCAR system. At no time during this event was user or system data access compromised or breached.
However, due to the HVAC outage, a significant number of AWS hardware devices suffered failure rendering data for hundreds of AWS clients unrecoverable, including a portion of the OSCAR files. While the OSCAR system was backed up, a significant portion of the AWS hardware storing certain OSCAR files suffered irreparable damage, requiring Symplicity to restore a portion of the OSCAR data from backups at a different facility.
While the system and all database data were immediately restored on Aug. 31, including all job opportunity listings and all application history logs, the restoration of a portion of file attachments in the system such as cover letters, grade sheets and letters of recommendations required additional recovery procedures over the course of several business days. All recovered documents are available on OSCAR and do not require additional action from users. For the portion of the documents that were not restored through the recovery procedures, Symplicity is working closely with the Administrative Office of the United States Courts to notify impacted applicants with the procedure to resubmit any such documents.
Throughout this process, the Symplicity Government Division team worked directly with the Administrative Office of the United States Courts staff to identify file impairments and provide remediation solutions and updates of the status of restoration.
