Incident Report Number: 2017-002

Moodle (e-Learning)

Ticket Number: INC0081715

What happened?

Monitoring tools detected servers hosting the Learning Management Consortium cloud environment (Moodle (e-Learning)), the university's online learning platform, became unavailable.

Who was affected?

Moodle (e-Learning) users located at the University of Alberta (eClass), NorQuest College (myclass.norquest.ca) and the Northern Alberta Institute of Technology (NAIT) (moodle.nait.ca) were potentially affected by this outage.

What was the impact?

The affected users were not able to access the Moodle (e-Learning) services at their respective institutions during the service outage.

What was the timeline of the incident?

Start: 2017/05/29 10:50 – Monitoring tools began alerting Moodle was down for all institutions (UofA , Norquest, NAIT). Support analysts started investigation and notification procedures.
2017/05/29 11:05 – Investigation revealed all the servers hosting the Moodle environment were available, but the Moodle service was still unavailable.
2017/05/29 11:20 – Support analysts determined the issue was the result of network communication ports being disabled due to inappropriate activity in the Moodle environment.  
2017/05/29 11:45 – Further investigation determined the source of the inappropriate activity was the result of unexpected behavior of the Moodle environment to a new device being added to the network. When this device was added to the network, a change to the communication protocol was detected by the Moodle environment resulting in the automatic shutdown of network communication ports which effectively made the Moodle service unavailable. Work began to make configuration changes to restore service.
2017/05/29 12:00 – Service was restored to UofA Moodle and NAIT Moodle.     
End: 2017/05/29 12:45 – Service was restored to NorQuest Moodle.

 

What was the root cause of the incident?

When a new device was added to the network infrastructure, the Moodle environment detected this change and began to behave incorrectly causing the network to shut down communication ports to restrict this behavior by design.

What was the work around and resolution for the incident?
Work Around

Configuration changes were made to the environment to restore service.



Resolution

Permanent configuration changes are scheduled for Moodle environment on June 3, 2017.

What are any recommendations to prevent this incident from occurring again?

Permanent configuration changes are scheduled for Moodle environment on June 3, 2017.

Updates

None.