Server Crash Investigation *DOINK DOINK*

[Jimer Lins]

OK, so I'm trying to narrow down what's going on when the server crashes. It seems that it only happens when there are players connected, at least that's what it seems like from what I've reviewed so far.

So if you've been online when it crashes, what are the circumstances around it?

* Were you doing anything specific that seems to be common?
* Did you just log in?
* Did someone else just log out?
* Etc...

If we can identify any commonality, maybe we can zero in on what's happening, because the logs show exactly as much as ze goggles.

[Jimer Lins]

My first investigations have revealed that the OS has been killing java for using too much memory. Basically it's running out of space and identifying processes to kill- because the Minecraft server takes up most of the RAM, it's first to go.

We had no swap space configured, so I've added a 1 GB swap file and enabled it. We'll see what happens now and if that allows the server to get past those points where the MC server takes too much RAM. It should only happen infrequently and in spikes. When it does occur, there may be lag- or there may not be. We'll have to see how it goes.

Worst case, I can add more RAM to the server, but that costs more, so I'm trying to keep us at 2GB usage for now.

Thanks everyone!

[Bulwark]

I wonder if there is a memory leak. Maybe a server restart at 3am isn't a bad idea.

[Jimer Lins]

So far, so good (knocks wood). We're stable for a while now and looking at the server, it has dinged the swap space for 10 MB, which if it hadn't been there would have caused the OS to go hunting for things to take down.

[Maizan]

This looks like it fixed the issue.

Thank you.
MaizanSpintik

[Maizan]

I think I spoke too soon. Server is down.



MaizanSpintik

[Bulwark]

server fall down, go boom.

It started getting real laggy, then I got a timeout exception on the client. I can't imagine it legitimately filled 10 GB of swap space, but that's what it was behaving like.

[Jimer Lins]

Well, this time it was the watchdog shutting things down. It's a new "feature" added to Minecraft a while back that watches for the "Warning: Can't keep up" messages that happen when the server gets laggy and after a certain number within a time frame, it terminates the server.

I'm going to bring it back up with the current watchdog settings, but if it keeps happening, I'll consider setting up a cron monitor that will automatically restart the server if necessary.

[Jimer Lins]

OK, the server is back up. Sorry for the trouble!