Like comedy, timing is the essence of a good data centre.
“The issue is related to time synchronisation systems within Telstra data centre servers, but the root cause remains unknown, Telstra CFO Michael Ackland said at a press conference five-and-a-half hours after the telco detected the issue at 4.30am.”
Some more details from The Guardian yesterday.
Looks like it was a GPS rollover bug. I ran into this with a commercial appliance relatively recently while continuing on the Self Hosted NTP path. I’ve been sparing the group the ongoing updates 
, but back in April my [second hand “brand name”] commercial NTP appliance hosted at home unexpectedly rocketed back to September 2006. It was suddenly seeing the first week rollover in 1999 but not the second one from 2019, so April 2026 was now September 2006 - 1024 weeks behind.
I reached out to the company for a bug fix, which they happily provided (delightfully helpful considering that I bought the discontinued model off eBay and wasn’t actually a customer), and it was low consequence for me as I am primarily using it as a crosscheck while building and testing several other ESP32-S3 and ESP32-P4 based GPS+PPS devices to go alongside the Linux+chrony one mentioned in the NTP thread. I really hope Telstra release a full post incident review at some point, as I’m curious to see whether the device(s) in question are multi-constellation (e.g., Galileo, BeiDou, GLONASS, QZSS, etc.) or GPS only. Given that the ~1024 week rollover is GPS specific, I suspect the latter.
If anyone knew about this I knew it would be you, @Belfry.
One thing I don’t understand is why now? If the roll-overs starting in 1979 are every 19.6 years how come this hit Testra now and you a few months ago?
I resisted the bait as long as I could, but you got me hook, line, and sinker !
Excellent question. I’d still call this a “rollover bug” even though this wasn’t during an actual rollover period, but I’m probably not using the term strictly correctly. “Off by 1024 weeks bug” doesn’t have the same ring to it 
.
GPS epoch is Jan 6, 1980, and number of weeks since epoch is reported as a 10-bit binary field (i.e., up to 1024). That value overflows and reverts back to 0 once 1023+1 is reached. Receivers (as in the modules/ICs, rather than a full appliance or server) kind of “infer” the date rather than receiving it directly from GPS. This doc from u-blox discusses the issue, provides examples, and covers the common way of resolving the issue (adding a sensible offset based on the firmware build date) to kick the can down the road ~20 years.
As an example:
1006 which implies April 19991024 (one rollover) + 1006 = week 20301006 is actually week 2030 in December 2018I’ve only loosely been following Telstra’s outage coverage over the last few days and hope they release a proper post incident report with the full detail. My absolute baseless speculation based on my own [very] amateur homelab metrology is:
week = 2426 (i.e., mid-July 2026) value was lost on the reboot378 (April 1987) started coming in to a freshly rebooted box+1024), a firmware regression, or a latent bug in the firmware “corrected” this to 378+1024 weeks (see above - e.g., box manufactured in 2018 automatically makes the assumption that 1999 has already occurred) = week 1,402Galileo, BeiDou, GLONASS, and more modern receiver modules internally use values > 10-bit, so this is now less of an issue within the lifetime of any appliance the modules are built into. This is the main reason why I speculate Telstra had/have “GPS only” timing devices rather than multi-constellation GNSS ones - that type of device is likely to be the only type susceptible to this type of issue.
In my case (April 2026 → September 2006), I don’t have any visibility into the specifics as it’s not my firmware, but my commercial GPS (single constellation) locked time server was constantly being poked and prodded while I was configuring, testing, and benchmarking against it, so I wonder whether there was a value that overflowed and somehow corrupted the GPS week multiplier. Similar sort of circumstances, but without the reboot in the middle of the process. It was easily fixed with a firmware update.