Six days from deletion to fix
Crimson: the labs down, from the deletion to the fix. Gold: our response. Dark: normal activity.- 1: the labs homepage is last seen working.
- 2: a contractor, whose hosting account has full administrator rights, creates an API token for a billing review done with an AI assistant.
- 3: CAPIEExamServer, which hosts the labs, is shut down and deleted a minute later through that account, unintentionally. Its IP address goes back to the provider, but labs.hackxpert.com still points at it.
- 4: the contractor creates a second token for the same review; one of the two is read-only.
- 5: we notice the spam site at 20:54, rebuild the labs on another server and point labs.hackxpert.com at it by 21:13, and revoke all non-owner access.
Full event log
All times are Europe/Brussels (CET, UTC+1).
| When | Event | Source |
|---|---|---|
| 13 Oct | Last confirmed normal operation of a lab endpoint, /RXSS/POST/10.php | Work records |
| 24 Oct | Labs homepage last seen working | Owner |
| 24 Oct, 13:02 | Contractor logs in to the hosting console, verified with a code sent to their email | Linode login history and notification |
| 28 Oct | Contractor creates an API token named "test" for a billing review done with an AI assistant | Linode event log, contractor |
| 29 Oct, 10:39 | CAPIEExamServer, which hosted the labs, shut down and deleted about a minute later through the contractor account, with no new login that day; its hosting backups are deleted with it | Linode event log |
| 29 Oct – 4 Nov | The deleted server's IP address is reassigned by the provider. labs.hackxpert.com still points at it, so visitors reach a third-party spam site | DNS record, observation |
| 1 Nov | Contractor creates a second "test" token for the same review; one of the two is read-only | Linode event log, contractor |
| 4 Nov, 20:02 | Owner logs in to the hosting console | Linode login history |
| 4 Nov, 20:54 | Owner notices labs.hackxpert.com serving a spam blog | Team chat |
| 4 Nov, 21:03 | Owner finds the server deleted | Team chat |
| 4 Nov, 21:13 | Labs being rebuilt on another server; labs.hackxpert.com already pointed at its address (172.104.232.237) | Team chat, Linode event log |
| 4 Nov, 21:20 | Contractor's hosting access disabled as a precaution | Team chat |
| 4 Nov, 21:25 | Kubernetes credentials reset on both clusters | Linode console |
| 4 Nov, 21:32 | Labs confirmed back online | Team chat |
| 4 Nov, 21:34 | All hosting users other than the owner's deleted, which revokes their API tokens; this report started | Linode event log |
| 4 Nov, 21:35 | External check: one resolver still returns the spam site from its cached copy of the old record (30-minute TTL) | Investigation |
| 4 Nov, 21:35–21:39 | Every direct request reaches the new server and genuine lab content; the spam site's pages return 404 there | Investigation |
| 4 Nov, evening | Certificate server (RatCerts) reviewed as a precaution: no sign of unauthorised access; its application secrets rotated; GitHub tokens, keys, collaborators and apps audited | Owner |
| 4 Nov, evening | Linode's login-verification email confirms the 24 Oct login was the contractor's own | Linode notification |
| 4 Nov, evening | Deletion confirmed accidental. Cloudflare's audit log shows no change to the labs record between the deletion and the 21:13 fix | Owner, Cloudflare audit log |
The deletion was confirmed accidental on 4 November. We handled it as a security incident until then.
Summary
On 29 October CAPIEExamServer was deleted by mistake through a contractor's hosting account with full administrator rights. The server hosted HackXpert Labs and, from time to time, exam code; no personal data was stored on it. The account's last login was verifiably the contractor's own, and the deletion has been confirmed as accidental.
Our DNS record for labs.hackxpert.com still pointed at the deleted server's IP address. The provider reassigned that address, and whoever received it runs a site that answers any hostname with a generated spam blog, so visitors to the labs saw "labs.hackxpert.com - PDF Documents" instead. The spam was never on our servers. On 4 November we rebuilt the labs on another server and pointed the record at it, nineteen minutes after noticing.
Until the cause was clear we handled this as a security incident. We removed every hosting user except the owner's, revoked their API tokens, reset the Kubernetes credentials on both clusters and hardened related systems. Those measures stay in place.
Incident record
- Incident ID
- HXL-2026-001
- Status
- Resolved: root cause confirmed and all corrective actions in place; monitoring until 4 December 2026
- Category
- Unintended deletion of a server through a privileged contractor account, followed by a dangling DNS record whose recycled IP address served third-party spam
- Functional impact
- Medium: labs unavailable on labs.hackxpert.com for six days and replaced by a third-party spam site; one server lost
- Information impact
- None: neither affected server stored personal data
- Recoverability
- Supplemented: the deleted server and its hosting backups are gone; labs rebuilt from source on another server; exam code redeployed from source when needed
- Affected
- CAPIEExamServer (deleted), which hosted HackXpert Labs; the labs.hackxpert.com DNS record; Kubernetes clusters ToolsAndInfra and Hackxpert-labs (credentials reset as a precaution)
- Not affected
- RatCerts exam bookings and certificates; RatCTF; the course platform
- Organisation
- The XSS Rat (Wesley Thijs), Maasmechelen, Belgium
- Incident handler
- Wesley Thijs
- Report version
- 1, published 4 November 2026
Impact ratings use the functional, information and recoverability categories from NIST SP 800-61 Rev. 2, which Rev. 3 leaves to each organisation's own scheme.
What we found
The spam came from a server that is not ours. labs.hackxpert.com pointed at an IP address we no longer held, and that address now belongs to a site that serves templated spam for any hostname it receives.
DNS
- hackxpert.com is hosted on Cloudflare DNS.
- From the deletion on 29 October until 21:13 on 4 November, the labs record still pointed at the deleted server's address. Cloudflare's audit log shows no change in between.
- At 21:35 the record pointed at the new server,
172.104.232.237, with a TTL of 1800 seconds. There is no CNAME, no AAAA record and no wildcard record.
HTTP behaviour, checked 4 Nov from 21:35
- One request still returned the spam site: a fake "latest news" blog of scraped book and document titles with invented authors and backdated posts, set to be indexed by search engines. That resolver was serving its cached copy of the old record, which a 30-minute TTL allows.
- Every direct request reached the new server and the genuine 17,482-byte labs index, across nine variants: desktop, mobile, Googlebot and Bingbot user agents, a Google referrer, Dutch and English language headers, and HTTP to HTTPS.
- The spam site's own article links returned our server's 404 page, so none of its pages exist on our systems.
Assessment
This is a dangling DNS record: a hostname that outlived the server it pointed to. Operators who pick up recycled cloud IP addresses often run a catch-all site that turns any hostname still pointing at them into SEO spam. Nobody accessed our systems to put it there.
Indicators
Use these to check whether you saw the same page. The site belongs to whoever now holds our old IP address; its tracker IDs are the most useful pivots for linking it to other hijacked hostnames.
| Type | Indicator | Context |
|---|---|---|
| Page title | labs.hackxpert.com - PDF Documents | Spam homepage; the hostname is templated into the title |
| Tracker | Histats counter 5061625 | Footer image on the spam page |
| Tracker | Yandex Metrica counter 51689945 | Tracking pixel on the spam page |
| URL pattern | /pdf/, /report/, /document/, /archive/, /paper/, /journal/, /file/, /ebook/, /digest/ + a 12-character ID starting with u + a slug | Spam article links |
| URL pattern | /dmca, /contact, /privacy, /category/<genre> | Spam site navigation |
| Content | Invented authors (e.g. "Douglas Kemmer", "Clara Bosco") with posts dated Aug–Oct 2025 | Generated filler |
| Meta | robots index, follow, canonical https://labs.hackxpert.com/ | Aimed at search ranking |
Impact
For six days labs.hackxpert.com showed a third-party spam site instead of the labs, and one server was lost. No personal data was involved.
- Visitors: anyone opening labs.hackxpert.com in that window saw a spam blog. It had no downloads, login forms or payment fields, but it did load third-party trackers (Histats, Yandex Metrica) that record a visitor's IP address and browser details.
- Lab users: the labs were unavailable on their usual address for those six days. HackXpert Labs needs no account, so no lab credentials were at risk.
- Search results: the spam page asked to be indexed under labs.hackxpert.com, so search results may show book or document titles until the site is recrawled.
- Exams: CAPIEExamServer and its hosting backups were deleted on 29 October. Besides the labs it only hosted exam code, which is kept in source and redeployed when needed. Exam bookings and certificates on RatCerts were not affected: that server was not deleted, its root password was unchanged and it shows no sign of unauthorised access. We hardened it on 4 November as a precaution.
- Other services: RatCTF, the course platform and thexssrat.com show no symptoms and are being checked as part of scoping.
What we've done, and what's next
Until the cause was known, the priority was to get the labs back on a server we control and remove all non-owner access.
DoneIn progress or planned
Containment
- Delete every hosting user other than the owner's, which revokes the API tokens they created 4 Nov
- Reset the Kubernetes credentials on both clusters 4 Nov
- Audit GitHub access to the certificate platform's repository: tokens, SSH keys, deploy keys, collaborators and authorised apps 4 Nov
- Rotate the certificate platform's application secrets and stop tracking secret files in its private repository 4 Nov
- Check both clusters for service accounts, tokens or role bindings nobody can account for 4 Nov
- Restrict network access to the Kubernetes control planes 4 Nov
- Rotate root passwords, SSH keys and shared password-manager entries the contractor account could reach, as standard offboarding 4 Nov
- Reset every other account the contractor holds for The XSS Rat and sign out all their sessions 4 Nov
- Confirm with the contractor whether they created the two "test" tokens: they did, for a billing review 4 Nov
- Report the deletion to Linode support and request API-level logs for both "test" tokens 4 Nov
- Review Cloudflare's DNS audit log: no change to the labs record between the deletion and the fix 4 Nov
Eradication
- Rebuild the labs from source on another server and point labs.hackxpert.com at it 4 Nov
- Confirm none of the spam site's pages exist on our new server 4 Nov
- Harden the certificate server's network exposure and run its application with least privilege 4 Nov
Recovery
- Bring the labs back on the rebuilt host 4 Nov
- Re-check the labs homepage from several networks and user agents, since the spam was cloaked 4 Nov
- Ask Google and Bing to recrawl the site and drop the spam URLs from their indexes 4 Nov
- Redeploy the exam code from source when the next exam needs it
- Watch the homepage and the certificate server for 30 days for changes or logins by removed accounts running since 4 Nov
The labs were back on their usual address nineteen minutes after we noticed the spam site.
Root cause
Both parts are explained: an accidental deletion, and a DNS record that outlived the server.
The server deletion
The deletion was made through the contractor's account during the contractor's working hours, with no new login that day. The account's last login, on 24 October, was the contractor's own: it passed Linode's emailed verification code. The day before, the contractor had created an API token for a billing review carried out with an AI assistant, and the deletion most likely went through that token. It has been confirmed as accidental.
The spam on labs.hackxpert.com
When the server was deleted, its IP address went back to the provider's pool, but labs.hackxpert.com kept pointing at it. The address was reassigned to a third party whose server answers any hostname with a generated spam blog, so our hostname showed their content.
We also considered whether a visitor had escaped one of the deliberately vulnerable labs and replaced the homepage. That is ruled out: the record pointed away from our servers for the whole window, and the spam site's pages do not exist on our server.
What turned a mistake into an incident
One account could delete a production server with no approval step and no alert. The hosting backups were deleted along with the server. And nothing removed the DNS record that pointed at it. Corrective actions 1, 3, 5, 6 and 7 address each of those.
Corrective actions
No single account should be able to delete production unnoticed, and a DNS record should never outlive the server it points to. All fourteen actions were in place by 4 November.
| # | Action | Why | Status |
|---|---|---|---|
| 1 | Give contractors restricted hosting users with only the permissions their work needs | One account could delete any server, by mistake or otherwise | Done |
| 2 | Require 2FA on every hosting, GitHub and password-manager account, and alert on logins from new addresses | Unusual logins should be seen when they happen | Done |
| 3 | Require approval and a written record before any server is deleted or rebuilt | Destructive actions should never be a surprise | Done |
| 4 | Allow API tokens only with a named purpose, minimum scopes and an expiry, and never give AI assistants or other automation delete rights; review monthly | Tokens act silently and do not appear in login history | Done |
| 5 | Keep backups outside the hosting account | Hosting backups are deleted along with their server | Done |
| 6 | Enable notifications for deletions, rebuilds and user changes | The deletion went unnoticed for six days | Done |
| 7 | Remove or repoint DNS records before deleting or rebuilding the hosts they point to | The labs record outlived its server and its recycled address served spam | Done, standing rule |
| 8 | Run each vulnerable lab in its own container with a read-only filesystem and no write access to the public homepage | Keeps a lab exploit inside that lab | Done |
| 9 | Monitor key pages from several regions and user agents | Cloaked spam is invisible from our own browsers | Done |
| 10 | Proxy public hosts through Cloudflare and restrict access to the Kubernetes control planes | Hides origin IPs and limits who can reach cluster APIs | Done |
| 11 | Keep secrets out of git, with secret scanning on every push | Credentials in a repository stay readable in its history | Done |
| 12 | Default-deny firewalls on every production host | Only the ports a service needs should be reachable | Done |
| 13 | Run every production app as an unprivileged user | An app bug should never mean root on the server | Done |
| 14 | One offboarding and compromise checklist covering hosting, GitHub, Kubernetes, password manager, email and social accounts | Removing one account does not remove copied credentials | Done |
NIST mapping
NIST SP 800-61 Rev. 3 organises incident response around the six functions of the Cybersecurity Framework (CSF) 2.0 rather than a fixed lifecycle.
| CSF 2.0 function | Where this report covers it |
|---|---|
| Govern (GV) | Contractor access, approval for destructive actions and offboarding (actions 1, 3, 14) |
| Identify (ID) | Affected and unaffected assets in the incident record |
| Protect (PR) | Least privilege, token hygiene, off-account backups, lab isolation, secrets handling and host hardening (actions 1, 2, 4, 5, 8, 10–13) |
| Detect (DE) | What we found; monitoring and notifications (actions 2, 6, 9) |
| Respond (RS) | Incident record, timeline, indicators, containment and eradication |
| Recover (RC) | Recovery steps, and this page as recovery communication |
Sources: NIST SP 800-61 Rev. 3; NIST announcement, April 2025.
For lab users and exam candidates
HackXpert Labs is a deliberately vulnerable practice environment, so it will occasionally go down and be rebuilt. That is expected, and the labs are back.
If you saw the "PDF Documents" page, you don't need to do anything: it was someone else's site on an address we had given up, with no downloads or login forms. Ignore search results for labs.hackxpert.com that show book or document titles until they disappear.
Exam bookings and certificates on RatCerts were not affected, and no personal data was held on any affected server. Questions go to wesley@thexssrat.com.