Report Date/Time: 3rd October 2014 12.13pm
Confidentiality
Distribution of this document is limited to DataTools Customers. Access should only be granted to
those with a business relationship to DataTools Customers on a need-to-know. If you have any questions pertaining to the distribution of this document, please contact DataTools.
Summary/Technical Issue
This incident occurred shortly after the Security Group maintenance performed at 17.00 on 2nd October 2014. This issue only effected https calls made to 12% of DataTools Kleber and Correct Address + cluster controllers. This issue was caused by a change of the firewall setting, although considered the adding of a firewall rule to be low risk, it caused an unexpected outage.
Notification
The issue was identified by DataTools Automated Monitoring Service at 17.36, 2nd October 2014.
Fix Action
During the application of the new rules for the security group a system error caused some of the existing rules to drop off the list and not re-applied. Once the rules were reinstated the service went back to normal operation.
Duration of the Incident
The incident occurred between the hours of 17:36 to 19:42 and was resolved in just over 2 hours.
Prevention Measures
Proposals to prevent this type of issue occurring in the future include raising the after maintenance monitoring of security group changes to also include secondary systems monitoring and additional checks to existing firewall rules before and after changes.
Proposals have also been made to change the scheduled time of low risk maintenance to when extra support and monitoring resources are available so in the unlikely event a fault occurs it can be rectified in a shorter period.
We sincerely apologise for any inconvenience this outage may have caused as we endeavour to improve our systems.