A datacenter switchboard has gone out and is being replaced, which is causing the outage.

The site went down last night because I added a new database to the cluster which I was hoping to set up quietly without any downtime. Unfortunately, it caused other databases to crash, which I'm having trouble recovering, because when multiple nodes go down restarting everything is a massive pain in the ass.

The new node is going to replace this more complicated setup. Over the last month, I've found more sure footing for our physical hardware I have not been able to put into use for over 6 months now. This transfer is to put our hardware back in use so I can stop juggling servers on multiple continents.

We will still retain redundancy so that if the Kiwi Farms is ever disconnected from the Internet again, there will not be extended downtime. However, the overall topology will be simplified and (hopefully) random crashes like this will stop happening so frequently.

TL;DR: We're basically right back to where we started in August and the last thing I have to do is get Ol' Faithful running again, which unfortunately means some downtime.

Edit at 4PM EDT: After a grueling process, I am close to having the site back up.

I have been working on this continuously since 8am my time (it is now 11pm). The sheer size of the database is making working with it unwieldy and every time I (re)attempt something to try and get proper replication, it takes hours.

My initial gambit (which started last night at this time) involved just plugging a new, empty database into the cluster and letting it synchronize itself unattended overnight so in the morning, with fresh eyes, I could set it up. This scenario would have created zero downtime.

The reality was that this somehow caused the other instances to crash and bring down the site. With the new database not instantiated and the entire site down, I tried what I thought were clever ideas to quickly bring the new server up to speed and get the site back up.

I gave up on clever ideas about 5 hours ago. I am abandoning the clustering entirely and setting up a very vanilla database topology that requires - once again - a complete replication of the many gigabytes of data between two servers, which will take more hours to complete.

Meanwhile, the database replication is fighting for bandwidth to replicate the 8 terabytes and many millions of files that make up our decade of obsessive archiving. That's actually still going and I started the download yesterday morning before I streamed.

I'm really sorry for the downtime. I know that some prostitute and her pet manchild were going to have a bunch of e-celebrities beat each other today and I wanted the site up in time for that.

The forum is back up. Please report any issues to the Technical Grievances thread. Thank you for your patience.

Sorry, back up. The new environment still needs some tweaking to stop crashes. This of course happens at 2am my time because I don't think the site has ever crashed at a time other than 2am when I'm fucking asleep.

Threads are viewable again.

>literally set alarm clock to check the site every few hours while sleeping
>do that
>site looks like it's working
>threads actually don't load

ffs

A DDoS attack is degrading performance. I'm looking into it.

12:40am - Database restart in progress.
12:46am - Restart complete, observing performance...
1:10am - Site seems stable. KiwiFlare is now more aggressive. Please report issues.
1:23am - Looking into issues with non-EU connections.
1:44am - Site stability has improved for global users.
1:59am - The tranny is tweaking the attack constantly, but I feel we're pretty close to snuffing him out.
2:03am - Mutated again.

I've made the determination that the inbound traffic is too much for the one server we have to handle. Even with KiwiFlare and aggressive rate limiting, it's too much and the CPUs are pegged just trying to handle SSL handshaking - the first step of an HTTPS connection. I will be ordering an identical server for that datacenter.

Less DDoS traffic is flowing through the EU datacenter (try proxying to London or Amsterdam), and I've been told Tor is unaffected at all.

Sorry again for the downtime.

I have mitigated the attack from last night but many users will have issues connecting because my filters are overzealous. I am working on* a change to KiwiFlare right now so that legitimate traffic can pass through again.

As of 7:12pm, it is up but filtering attack traffic still. 3000 IPs are currently banned and as that attack continues it will run out of attack IPs to work with.

As of 7:52pm, I am pretty sure the troon has adjusted the attack vector. There are thousands and thousands of IPs at play but he's gotten them working very passively so it's difficult to cull them without hurting legitimate traffic.

As of 8:17pm, a new IP is being banned every few seconds and it's still not unpegging the server. While waiting for the new server, I am going to see if I can make the L7 protection more efficient so it's not as easy to peg. Right now it's capable of handling tens of thousands of requests a second but that is not enough.

April 23rd
As of 3:21pm, the troon figured out how to get around the cheeky fix and I'm working on something else while waiting for the new server.

As of 7:19pm, the new server has been delegated and I am setting it up. The attack is really strong again, but I'm working on doubling up our processing power.

As of 8:40pm, the device is provisioned and being configured for production.

The second server is up and the DDoS attack does not appear to be causing any downtime. If you're unable to access the Kiwi Farms by clearnet via your ordinary connection, please write me an email at <josh@kiwifarms.net> with information.

Authoritative DNS outage. The site is up. If you are adventurous, you may manually set your hosts file.

103.114.191.1 kiwifarms.net uploads.kiwifarms.net no-cookie.kiwifarms.net

If you're not adventurous,
1) Wait a bit, or
2) http://kiwifarmsaaf4t2h7gc3dfc5ojhmqruw2nit3uejrpiagrxeuxiyxcyd.onion/

Site is accessible from some EU networks and Tor.

3:44pm - Tier 1 outage confirmed.
3:45pm - Servers at Canadian location are down.
4:47pm - Resolved

The file server which stores attachments was compromised and all live version of files have been replaced with a 3kb file of some kid on Twitter taking credit for it. I'm closing the site to audit the attack. I'll update this message as I learn more.

12:43pm - The impact is much smaller than I expected. I am now revaluating what has happened. It does not appear the fileserver itself is compromised.

1:00pm - I've only found two video files that have been changed to the corrupted file. I'm not sure how the trick is being done yet.

1:45pm - I believe I have an idea of how it was done and I'm attempted to recreate it.

I have confirmed the attack vector, have applied patches to secure the attack, verified the patches individually, and am now working on undoing the damage.

The attack did not have direct access to the file server but did find a way to replace existing files. No accounts were compromised and the actual damage is relatively small. It was video files in 2 threads, my avatar, and one other video that's used on the registration page.

I reached out to the attacker and he's not politically motivated, it's just something he did for fun. I opted to give him 5 XMR for confirming my theories of what the vector was.

Back online ETA 1 or 2 hours.

Edit: It's literally just some kid, calm down.

Edit 2: To clarify, I did not pay for information. I had figured it out on my own. He just confirmed it for me outright. I compensated him for not destroying anything and to encourage him to pursue legitimate bug bounties, considering he was young and morally structured. Again - chill out. I would not pay blackmail or pay for information upfront.