Norwegian BankID largely unavailable

Incident Report for Criipto

Postmortem

A post mortem was received from Nets on July 10. Part of the text has been extracted from the provided PDF - in English following the Norwegian summary. Please send email to incident@criipto.com to receive the full report from Nets.

BankID specific incident report related to network outage major incident INC0253086

Norwegian summary/Norsk sammendrag

Grunnet en regelendring på en intern brannmur ble DNS trafikk for en rekke tjenester blokkert på morgenen 09.07.2019. Dette medførte at alle brukersteder som benytter internettkommunikasjon opp i mot BankID FOI fikk nedetid på BankID på mobil. Brukersteder som benytter Ekstranett kommunikasjon opp i mot BankID FOI opplevde ikke problemer med BankID på mobil. De som var berørt opplevde ustabilitet fra 10:05 – 10:22 og nedetid fra 10:22 – 13:03. Dette berørte 50% av total BankID på mobil-trafikk.

Signering med BankID 2.1 ble også berørt av hendelsen og var utilgjengelig 09:45 – 14:30.

En utsteders HA2 tjeneste var også berørt og nede 09:57 – 14:43.

Brukersteder som benytter E-ident og E-signing tjenestene opplevde også nedetid under denne incidenten.

Hendelsen ble løst ved å rulle tilbake regelendringen.

Executive Summary

‌What has happened: A rule was implemented on the network firewall which resulted in some network traffic (DNS requests) being blocked on services that used the affected servers

When did the incident occur_:_ 2019-07-09 09:44

When did the incident end: 2019-07-09 14:35

Who was affected: _
BankID on mobile:_
instability from 10:05 - 10:22
50% unavailability from 10:22 - 13:03
All merchants communicating with BankID FOI at an internet line was affected. The merchants communicating at Ekstranett was not affected.

BankID Netcentric:
Signing 2.1 unavailability from 09:45 - 14:30
One HA2 service for one BankID issuer was unavailable from 09:57 - 14:43

What was the consequence of the incident: Outages on BankID as detailed above

Action/workaround to stabilise operation: After troubleshooting the network the following actions were taken to alleviate/fix the error:
Rule was added to allow DNS traffic - this partially fixed the issue
Erroneous rule was deleted - This completely resolved the incident

Duration of the incident:
0 days 04:51:00 hours

Posted Jul 11, 2019 - 07:34 CEST

Resolved

As communicated earlier via email, the BankID service in Norway experienced a more or less complete outage for part of the day. The issues have now been resolved and operations are back to normal.

The issue was caused by an as of yet undisclosed problem at Nets, the operator of Norwegian BankID.

If we receive a post mortem from Nets we will share it here.

Posted Jul 09, 2019 - 16:43 CEST