Some systems are experiencing issues

Stickied Incidents

Monday 6th May 2024

Site and database backup restore issue

A backup and restore script has corrupted some data, we're working to restore this urgently.

  • We've completely recovered, keeping up this notice for a few days for context for those who may not have seen this.

  • Registrations are open again, backfill for missing releases is about half way done.

  • We've made good progress with user restoration, if you're still having an issue please use the forgot password link. Any issues after that please reach out to us via help.althub.co.za/contact.

    A backfill is underway for missed context, this may slow down new releases.

  • We're bringing services back online slowly, the site, API and indexing may be slow.

  • We've restored the corrupted data, however some users will need to reset their passwords in order to sign in.

    A detailed review of the incident can be read here > https://help.althub.co.za/data-loss-april-may-2024/

  • A restoration is underway, we'll mail affected users.

  • Users who signed up over the last 4-6 weeks may not be able to sign in. We're working to restore from backup, however this may take some time. Please be patient while we work on resolving this.

  • Past Incidents

    Friday 12th April 2024

    Website Search not returning results

    We're investigating an issue with search, updates to follow.

  • This has been fully resolved.

  • The fix is out, we're monitoring.

  • The issue has been identified, a fix is underway.

  • Thursday 11th April 2024

    API API load testing

    We're testing some changes against the API - it may be slow or unavailable over the next hour.

  • Testing is complete

  • Wednesday 10th April 2024

    No incidents reported

    Tuesday 9th April 2024

    No incidents reported

    Monday 8th April 2024

    No incidents reported

    Sunday 7th April 2024

    No incidents reported

    Saturday 6th April 2024

    Indexing Infrastructure maintenance updates

    Per the scheduled maintenance, updates will be posted here to track progress.

  • Maintenance is complete, site performance is looking good. We're still ironing out a few bugs, feel free to contact us if you spot any issues.

  • The API has been re-enabled, we're monitoring performance and any errors closely.

  • The frontend site has been re-enabled, we've still got lots of implementations to do across the cluster so things may be slow.

    The API will remain offline for now.

  • All altHUB services are down while we do the core work, we'll be back soon!

  • The API has been disabled. We expect this to come back last due to it's size, traffic volumes and complexity.

  • The maintenance that didn't require downtime is complete, we're moving to more disruptive changes now - starting with indexing which has been disabled while we complete the work.

    Any backfills will be complete once the new hardware has been integrated.