Below is the Incident report for the resolved issue regarding access to the Phone Verification Data that occurred on November 6th 2017.
Remember – to monitor any issue in real time – always check our Status page at status.datatools.com.au. Use the Subscribe to Updates button to receive updates via email.
Incident Overview
Severity: Minor
Status: CLOSED
Summary/Technical Issue
Call to DataTools’ support team regarding an issue with Phone validation returning no information when called. A check of the system confirmed the issue.
Whilst the Kleber service was fully maintained – access to the phone validation data was unavailable.
Upon talking with the 3rd Party Provider it was discovered that they had made a change to the access portal on Sunday but had not communicated the change to DataTools.
Our developers instigated an emergency change to the Kleber servers which was tested and rolled out to the cluster. This fixed the issue.
Notification
The issue was notified to DataTools Support at 11am on 6th Nov 2017.
Fix Action
Our developers changed the call to use the updated details for access which was tested and rolled out to the cluster.
Duration of the Incident
Access to the data was unavailable from roughly 4.30pm on Sunday 5th November 2017 to 6.30pm Monday 6th November when the servers were updated.
From the time of notification at 11am DataTools took 7.5 hours to liase with the 3rd Party provider to work out the issue; devise the solution; code the fix; test and then roll it out to the server cluster.
Incident Postmortem and Expected Prevention Measures
We have had discussions with the provider of the data and have requested advanced notice of any changes to their system so we can make the necessary changes within Kleber.
Whilst we have extensive checks to ensure the Kleber servers are healthy and available – DataTools will also instigate additional checks to assist in detecting any issue with access to 3rd Party data so that we can communicate earlier to our clients that there is an issue and to minimise the data access downtime where possible