This blog is part of our ongoing Essential Guide to Game Servers series. This is part one on provisioning – part two is coming soon.
If you’ve ever provisioned a number of servers for a game, you’ll know that there is a lot of ground to cover before you get to a place where you can sleep at night.
This post will be looking exclusively at provisioning so many servers that pure manual labor is infeasible (or, at least, inconvenient). If you’re just a hobbyist trying to run a game for your mates, I would recommend you simply read the relevant game’s server set-up documentation.
Obviously, what you need from your game server will depend largely on what you need in your game – if you’re making an action-packed, fast-paced first-person-shooter, you’ll have very different requirements compared to making a turn-based strategy game.
To tackle this, we spoke with Ben Alton, a DevOps Engineer at Multiplay. Ben works as part of a team responsible for provisioning the thousands of servers we maintain at Multiplay.
But before we dive in, here are some things that this blog won’t cover, but you should think about:
Who is giving you your machine? Are you going into the cloud or will you use physical machines (also known as ‘bare metal’)?
This is a really big topic all by itself, so let’s save that for another day.
Where in the world do you want your servers?
Again; big topic. We’ll cover this later in the series.
- Quality Assurance
How do you verify that your machines are actually up to snuff?
This can vary massively depending on what you care about. I.e. If cost-saving is your number one priority, then perhaps you’re willing to accept things that organizations with deeper pockets might not.
Without further ado, let’s pass over to Ben to look at some provisioning basics to consider.
Operating System – Windows vs Linux
Windows Game servers? They’re fine I guess, but have you heard about Linux?
You’ve probably heard that Linux is faster than Windows, but whilst there are a lot of ways you could prove that, it’s not the full story.
How well your application runs depends on how well it was optimized for a given environment. It’s perfectly reasonable to assume that your game might run faster on Windows because the people that developed it were only really thinking of Windows at the time.
Similarly – what about you? – do you have experience with the operating system? Would you be willing to diagnose and repair any little problems that might pop up?
The fact of the matter is that as of 2019, most people still run Windows on their home computer. As a result, many game developers will write game clients for Windows and, since there’s a lot of code to re-use, write the game server program for Windows as well.
You might be surprised to see a lot of game servers running Windows in the world and that’s not bad.
However, if you have the skills and the time, I would say look at Linux. Using Windows requires a license and those can get expensive. Many Linux distributions are free and the programs that they use are often free as well.
It might be worth checking what your provider is capable of. For example, some providers will just ignore Windows Servers because they can be relatively niche. Another thing to look out for is how your provider will be giving you this machine – will you be given an IPMI address and be expected to install the OS yourself or will you be given a username and password to log in to a minimalistic pre-installed machine?
Many providers might make Linux installs easier for you simply because the technologies to deploy a brand new Linux server are freely available (PXE, Kickstart, Anaconda and so on). Windows has its options too, but they are less popular in some places – be sure to check what your provider is capable of.
That said, even if your provider can’t automatically provision and configure your desired OS, it’s worth trying to add your own special blend (again – ask your provider – they might block or intercept traffic to prevent any issues with their provisioning system).
TLDR? Go with what you’re comfortable with. If you have a choice, Linux would typically be cheaper and, depending on a lot of factors, likely to perform better on the same hardware.
Basic Configuration Details
Here are a few things to be aware of while you provision your machine.
Swap Memory (Paging)
Without going into too much detail, the amount of memory (or RAM – Random Access Memory) a machine has is finite, if you run out and you need more, you’re toast. Paging is a trick to use a part of the disk (either SSD or HDD) as a buffer for extra memory. This ‘fake memory’ is often called “swap space” (or just ‘swap’) or “memory pages” (or just ‘pages’). You can have as much ‘fake memory’ as you have free disk space.
Sounds great, right?
Well, no – firstly, the disk is a lot slower than RAM, even the best NVMe SSDs that you can find (at early 2019) will be about a thousand times slower to get the data you asked for. Of course, you probably won’t even be using the super-hot-mega-awesome-hyper-disks, so it’ll likely be slower still. If your system is configured to use swap, your applications probably won’t be able to detect if their memory is on real RAM or in fake swap memory. As a result, memory access in your game can become unpredictable and slow when the real memory is running low.
Also, to be helpful, most operating systems will try to be nice and put infrequently accessed data onto disks and leave the RAM to hold all the data that you keep using. But choosing what to shuffle around (and actually doing the shuffling) can take system resources. So, even if your application doesn’t use swap itself, you might notice some slowness on occasion.
Sounds terrible, right?
Well, no – the above scenario is only going to be obvious if you run out of RAM and your OS is configured to be eager to move data between RAM and disk.
There’s another fun point to consider with swap; security. If you don’t want to think about physical theft of your game servers, you can ignore this paragraph, but for those that are super paranoid, read on. A lot of applications need to handle sensitive data (passwords, credit card details, player details), but they usually don’t write them to the disk (or, if they do, not in plain text). However, you still need to handle the information in its most obvious form for a little bit. If, by chance, the memory was to be placed on the disk and that disk was stolen… well, you get the idea.
That said, for Linux at least, you can add a capability to the binary to effectively keep its memory in memory (kind of). For more information, search the Internet for guidance around ‘cap_ipc_lock’.
Swap isn’t for everyone, but it’s often very helpful when you need it.
If you want it, I recommend you configure it as soon as possible. If you don’t have a dedicated swap partition, you can create a swap file instead. If you are going with a swap file, you want to create this early as you’ll want a contiguous block (no fragmentation in the file) and that’s far easier when the disk is mostly empty.
If you’re using swap, how much swap you want and how frequently you want to shuffle things around are all things you might need to think about before you start getting real players.
Computers need to know what time it is – and this is super critical to security. For example, web certificates have a time period for when they are valid, if your computer thinks it’s in the past (or even in the future), it will complain before connecting to a website. Several other applications could be similarly impacted. Needless to say; time is important.
Sadly, computers are not perfect at keeping track of time – you can tell your computer the exact time to the millisecond today, but by tomorrow it will have drifted a bit. How much drift you get will vary between hardware.
To get around this, we use NTP (Network Time Protocol), a lovely way for your machine to constantly check its clock with a defined ‘good clock server’ (NTP server) and adjust itself automatically.
You, oh-great-provisioner-of-the-machine, might need to define that NTP server (or those NTP servers) yourself.
Again, the value (or values) will depend on a lot of things. If your provider has an NTP server in the datacenter, consider using that – closer is generally better. However, keep an eye on the ‘stratum’ – NTP servers themselves can drift, so they also use NTP to readjust their time (and that trickles through the network). The NTP server stratum tells you how close to an original time source you are – like one of those super-accurate (and very expensive) quartz crystal clocks. The lower stratum will be better, but if it takes you too long to query it, you might find your machine’s clock is consistently askew.
Also – just for fun – some providers will intercept and block external NTP traffic to prevent potential Distributed-Denial-of-Service attacks. If that’s the case, ask your provider which NTP server to use.
If your machine is already configured with NTP and time seems to be working for you, you can probably ignore this.
If you’re not sure about a decent NTP server choice, ask your provider or I’d recommend time.google.com as a start.
It’s always nice to be able to type in “human words” into your address bar and for your computer to instantly know where it needs to go. That’s a feature of the Domain Name System (DNS) – simply put; a phone book for the web. Like choosing your NTP server, you might need to select this yourself when provisioning your machine.
Like NTP, you can select multiple DNS servers – and it is generally recommended that you do to avoid any issues during DNS server maintenance.
I won’t go into too much detail here in the interests of brevity, but you should consider the following points:
- Does your provider block or intercept DNS traffic or require you to use their DNS server? If so, use that.
- Does your company have a DNS server that you will need to be able to use to resolve names? If so, use that.
- Do you have a preferred DNS server? If so, use that.
- Use ‘22.214.171.124’ and ‘126.96.36.199’ (which are managed by Google and are, generally, very good).
In part two, we’ll take a deep dive into configuring your game server – coming soon!
This is part seven of our ongoing Essential Guide to Game Servers, which includes:
- Patching a live game server
- Game server player density: tips and tricks to keep costs down
- Monitoring game servers – part one
- Monitoring game servers – part two
- Matchmaking – part one
- Matchmaking – part two
- Provisioning – part one
- Provisioning – part two