Cyberattack on ZEIT.IOs Email Reputation

On May 1st, there was an attack on ZEIT.IO's email reputation. An attacker sent an invoice to tens of thousands of recipients.

ZEIT.IO is a platform for the automated billing of project time. Companies can create projects and assign them to employees or freelancers. Once project time entries have been approved, an outgoing invoice is automatically generated and sent to the client. The client themselves can also be added as an approver for the project. Ideally, the client approves the timesheet and receives the corresponding invoice just two minutes later. If the work was performed by a freelancer, a credit note can also be issued automatically, thereby eliminating the need for an incoming invoice from the freelancer. Invoice documents can also be automatically transferred to DATEV, ensuring that the tax advisor has access to them in real time.

Freelancers can create up to two e-invoices free of charge within their account. These e-invoices can then be sent directly to the client from within ZEIT.IO. A dedicated form is provided for this purpose, allowing users to edit the recipient, subject line, and email message.

The Attack

On May 1st, an unknown user created a free freelancer account on ZEIT.IO. Two invoices were subsequently generated within this account. One of these invoices was then sent to over 29,000 recipients over a period of several hours. The form used to send the invoice was accessed thousands of times, each time with a different recipient.

Here, you can see the surge in server traffic on May 1st.

On the page where invoices are sent, the most recent audit logs are displayed below the form. This allows users to see when an invoice was last sent to a recipient. Typically, this table contains between zero and two entries.

The attack eventually ceased because the audit log table grew so large that the page could no longer load. Consequently, the attacker began consistently encountering timeouts. This also resulted in a brief downtime of two minutes, which triggered a monitoring alert—prompting the dispatch of emails and SMS notifications to the responsible personnel.

The logs within SigNoz clearly demonstrate that all HTTP requests associated with this attack originated from a single IP address. This IP address traces back to a data center in Germany. Criminal charges are being filed.

Consequences

Incidentally, the recipients listed on the invoices were all fake addresses as well. The likely objective of the attack was to damage ZEIT.IO's email reputation. Typically, ZEIT.IO's bounce rate sits at 0.01%. As a result of this attack, the bounce rate surged to 20%, and AWS SES temporarily suspended our SMTP access—leaving ZEIT.IO completely unable to send any emails. This was, of course, far from ideal—especially for ZEIT.IO's auto-invoicing feature. Some customers rely on this feature to automatically send time-scheduled invoices to their own clients—typically on the first or second day of the month. Due to the attack, this functionality was temporarily rendered inoperable.

ZEIT.IO has since temporarily switched to a different SMTP server, and all invoices that had been stalled by the auto-invoicing system have now been successfully sent.

GDPR & Data Security

No customer data was compromised. This attack was isolated entirely to the attacker's own account.

Countermeasures

To prevent similar attacks in the future, the following measures have already been implemented:

• Invoice Sending Limit: An individual invoice can now be sent a maximum of 5 times per day.
• Traffic Throttling: Middleware has been installed to implement general traffic throttling. If an excessive number of requests originate from the same IP address within a short timeframe, those requests are now rejected.

With these measures in place, any similar attack attempted in the future will be unsuccessful.

Conclusion

An attack of this nature—occurring on May 1st, during a long holiday weekend—is particularly malicious. This was not our first cybersecurity incident, and it certainly won't be our last. Cybersecurity will continue to grow in importance in the years to come. The best thing to do in the face of such attacks is to learn from them and continue to improve one's systems. ZEIT.IO becomes more resilient with every attack.