May 2018 Developer Blog

Posted by Citan on Sunday, May 6, 2018

May 2018 Developer Blog

Hi, I'm Eric ("Citan") Heimburg, the lead developer of Project: Gorgon. It's been a while since I did a dev blog -- while we were crunching on the UI rewrite, and then the Steam integration, there wasn't a lot to say besides "we're still doing that thing we already talked about." But now that we have some space, it's time for a dev blog!

Before I start, though: What's a dev blog? It's a developer diary, a look at where we are right now. It's meant to give you some insight into how we're thinking and what we want to do. But the details aren't solidified yet. Dates and times, core details ... once or twice I've even mothballed entire systems that just don't work, right after having talked excitedly about them in the dev blog.

So I apologize in advance for any inadvertent misinformation. I'm an optimistic person -- you don't start making an MMO by yourself if you're a pessimist -- and I want to convey some of what I'm excited about. And your feedback is a valuable byproduct of that -- we often change direction based on what people say here.

Let's get started!

Client Optimizations

Framerate in Serbule is bad! We know. This isn't something that will get fixed overnight. There are some small optimizations coming, but nothing that will fundamentally fix the FPS issues. There's no magic code fix that will make everything good; it will take a bunch of steps to fix. But never fear -- those steps are in progress!

Animation Overhead

If you log into Serbule right after the server reboots, you will have a pretty good framerate. Not great framerate on lower-end machines -- see below for why -- but not abysmal, either. I get 60fps in Serbule on my personal test server... because it's a server with nobody else on it.

The problem comes when there's a hundred or so other people nearby. This creates bottlenecks in two areas: animations and UI.

The underlying animation bottleneck is a limitation of the graphics engine we use, Unity. Animations in Unity are all calculated in a single thread of the CPU, so when the game has to render 100 animating players, your performance will always be bad. And there's frankly nothing I can do to magically fix that.

In the longer term, this is a problem that Unity itself will fix. Unity added a feature called "graphics jobs" which fixes the animation bottleneck. Unfortunately, the feature is still labeled "Experimental", and the last time we tried to use it (late last year), it caused random crashes for random people. They've made bug-fixes since then, but it's still labeled Experimental, so I'm hesitant to inflict it on players again. But when this feature is finally ready for primetime, it will basically solve the animation performance problem.

In the meantime, what can we do? Well, we can reduce the number of characters that are animating. In fact, try this now: in the Settings window, under graphics options, click "Show Advanced Options" and then set the "Entity Animation Distance Multiplier" to its minimum value. This will cause players to stop animating when they're farther away from you, and you can see how it affects your framerate in busy areas.

One avenue I'm exploring is creating a fancy heuristic option that quietly disables animations on some players when the screen gets too busy. Getting that to work well is a hard problem, though, and it doesn't make sense to try to write this until after our upcoming Mecanim Update (see below).

UI Overhead

The UI system has to do a lot of calculations to tell if you can click on something. These calculations are not simple, and when there's 50+ selectable players on the screen, those calculations really slow things down. I've added some simple hacky workarounds to improve framerate already -- you might have noticed how sometimes the labels over the heads of people trail behind the people themselves? Yeah, that's one of the optimizations to keep it from costing too much CPU. But a "real" fix will require a more intensive rewrite. That rewrite is waiting on some other things, but eventually the UI system will handle the problem more effectively -- and then the labels will keep up with the people more reliably.

Serbule Assets

Not all of Serbule's problems are due to having a lot of players in one tiny spot, though. Serbule itself is still a messy place. Throughout our long alpha, Serbule was the testing ground for new technology and design ideas. It's suffered several cataclysmic rewrites that entirely changed the map... and I expect there will be one more cataclysmic rewrite in its future. Some of the expensive parts of Serbule include:

Sky: The sky may just look like a sky, but it's actually a dynamic weather system. With the flip of a switch we can activate rain, snow, sleet, lightning, fog, high winds, and more! But those features reduce performance, even when they're disabled. (And of course they have a bigger impact on performance when they're enabled, which is why they aren't.) The sky is based on a conglomeration of some 3rd-party Unity sky libraries that just haven't proven to be efficient enough. I've been making optimizations to them, but ultimately I suspect we'll need to replace the whole thing with a different system. (This sky system is also used in Serbule Hills, and is some of the reason for FPS slowness there. Other areas use an older sky system that's very cheap but very primitive.)

Ground: The ground itself is not very expensive, and the trees in Serbule are Speedtree trees -- reasonably fast -- but the grass, little rocks, and other small details are too expensive. There are 3rd-party graphical libraries we can use here to improve things, eventually.

Buildings: Serbule has a lot of buildings in it, and they ARE optimized... but unfortunately they're optimized for an earlier version of Unity. That's one of the problems with making a game for five years: there's lots of moving targets in terms of performance. The building geometry uses too many draw-calls and there's not much I can do to fix them in code -- the assets themselves will need adjusting or replacing. That's something we're starting to plan.

So Serbule will continue to be a place where we experiment -- but now we're experimenting with new engine components for weather, grass, and so on. When we're happy with those underlying components, we'll either fix or replace the actual art of Serbule.

Fixing Overcrowding by Spreading Players Out, a/k/a "Serbule is for newbies!"

There's a lot of stuff we'll be doing to improve framerate over the coming months, but there's also a more fundamental problem. Serbule is the newbie town. New players show up here and learn how to play. They kill pigs and stuff right outside the gates. And Serbule is also, unfortunately, the only major hub city until level 60. There are many other mini-hubs in the world with vendors, storage, and supplies, but players aren't flocking there. Nothing is as central, as accessible, or as full-featured as Serbule.

And that's bad content design. If there's 500 players online, 200 of them should not be in the same small area.

So I think we'll need to prioritize adding another hub town between Serbule and Rahu, a home base for players after they get past the Serbule levels. There are two upcoming areas that might work: the elven city past Sedgewick Forest, and the dwarven city hidden in the mountains. These areas are currently slated to be mini-hubs (not much more elaborate than Amulna, another mini-hub), but we can expand one of them into a full city, and start reducing the game's emphasis on Serbule.

To be clear, there's no new city in development yet! There's lots of planning to do first. But it'll probably need to happen before we implement Statehelm, the level 80 hub.

Server Optimizations

There are many kinds of lag -- in fact, the word "lag" is not a technical term, it's just a slang word that players use. (That's why, when you report that you're having "lag", we always have to ask what specific problem you mean. Framerate drops, monster "rubberbanding", and slow NPC response times are three completely unrelated problems which players just call "lag".) We've talked about framerate problems, but the server can cause delays and slowdowns too.

Master Server Bottlenecks

After the last content update, I spent the next few weeks looking into the causes of server bottlenecks, and I managed to fix a bunch of them. This was a high priority for me because the master server shouldn't be under any real strain. It can easily handle 500 players without breaking a sweat, but it was getting jammed up pretty often. Why?

Well, the reason is bugs in the code. They're subtle bugs, though, that only show up when enough players try to do certain things at once. For instance, there was a bug where if enough players were downloading the contents of a storage vault at once, other NPCs in the area would stop interacting with players until one of the storage vaults finished downloading. (In java parlance, the NPCs were synchronizing on the wrong object.)

I've already found and fixed the most glaring and damaging of these bugs. You may have noticed that the server was rebooted every night for about a week -- each day I would add more logging to track down these problems, then the next day I would retrieve the logs, find bugs, fix them, and repeat. There are still some more synchronization problems in there somewhere which I haven't found yet, because they happen too rarely. But I'll track them all down eventually, and the remaining bugs are rare -- they aren't causing a lot of trouble (or else I'd be able to find them).

Sub-Server Problems

There's another component to server "lag", though: the sub-servers. These are separate processes that connect to the master server and control the monsters, pets, and NPCs in an area, making them move around and do things that the master server requests. There's a bunch of sub-servers, and they can sometimes get disconnected -- either due to a network problem or because they crashed from a code bug. When that happens, players often don't notice. The sub-server automatically restarts and reconnects.

But sometimes players DO notice, because their performance gets really bad. Why? What's happening? Well, that's a funny story. You see, there's a very fancy redundant system: when a monster needs to move but the sub-server is gone, the main server says "no problem. I'll just deputize a random player to do the work." It randomly chooses a nearby player and makes their game client perform the calculations to move the monsters! This is a powerful tool that we use internally during development and testing. And I figured, why not use it as an emergency backup system in the live game? It's not really much of a security risk to have a random client drive a random monster for a few seconds at a random moment. So what's the down side?

If your machine is a beast, like our development machines are, there's no real down side -- the overhead is unnoticeable. But if your machine is running near its capacity, even a little bit of extra work can drop your performance into the pits. And when there's only one player in an area (like when you're the only person in a big dungeon), that player's client gets tasked with moving ALL the active monsters nearby, which can be way too much CPU load for lower-end machines. If there's lots of players around, the work is spread between them randomly, which results in less CPU cost for each player, but can cause latency problems. If you ever see a monster "rubberbanding" (zooming around between several locations) it's usually because some player with a terrible ping has been tasked with driving the monster.

Overall, using this system as a backup seems to be causing a lot more problems than it's solving. I've been meaning to make improvements to it for a while -- maybe it could take your ping into account, and maybe keep track of how much CPU load it's using on each deputized computer, to keep from overtaxing people... but at the moment it doesn't make sense to focus on that. So in the next update, I'm just going to disable it. When a subserver disconnects, monsters and pets will just stand perfectly still for a moment until the subserver resyncs. I suspect that will be a lot less disruptive. We'll see how it goes!

Treasure and Skill Changes

So there's lots of engine work going on under the hood. But there's also a lot of very visible new stuff going into the next update: lots of skill and treasure changes. These fall into three categories.

Percentage-Based Treasure Mod Descaling

We aren't yet at the point where we're fine-tuning skills and treasure mods. I'll be fixing a few of the most problematic individual abilities and treasure effects, but most of the fine-grained balancing will happen later. Right now I'm still trying to get the "big picture" balancing done. And in the big picture, player DPS is out of whack at high level.

The problem comes from treasure effects that multiply a specific ability by a percent, such as "Fire Bolts Damage +37%". The percentages for these mods are designed in a very naïve way: they assume they're the only buff you have for that ability. To use Fire Bolts as an example, on my balancing spreadsheets that 37% mod is calculated as adding about 100 damage -- which is what it adds if you have no other mods for the level 60 Fire Bolt spell.

But, of course, there are lots of other mods that boost Fire Bolts, and during actual gameplay that 37% generates a much bigger number. This is all very intentional: the basic calculations are naïve on purpose so that you can find ways to crank up your damage. That's part of the fun of building gear sets!

The problem is the calculations are TOO naïve. There are more ways to boost damage than I originally planned, and the resulting big numbers are causing problems when making content. Those problems aren't too bad right now; it's a bit tough to create content that works for lots of builds, but it's still doable. But it will become a severe problem by the time we reach level 125 content. I need to ratchet the damage percentages down a few notches. This doesn't have to happen immediately, but I figure we should get this sort of thing out of the way ASAP, so that we have as much time during beta as possible to analyze the results and make more adjustments.

But fixing this problem is a little bit worrisome. The changes need to be made to the "average build" as if that's a real thing ... but it's not. If I drop the "average" player's overall damage by 25%, and drop monster health similarly, then the "average" player won't be affected much. But in this game full of weird unique builds, player damage is all over the spectrum. These changes will affect different builds very differently. Most skill combinations would still be fine -- maybe a bit slower to kill things, which is fine, as some are too fast anyway -- but some skill builds which are on the lower end of the damage spectrum might become unplayable.

To limit the number of unforeseen problems, I think we'll do the change in two or three steps. In the next update I'll lower those percentage-mods a little bit, and adjust monsters a bit. Then we'll see how it feels, and I can react to problems while they're still relatively small. In a future update I'll do it again, lowering mods and monsters some more. And then a third time if necessary. The downside is there'll be repeated "nerfing patches", which are always stressful and annoying. But the upside is that none of the nerfing patches is super traumatic.

Another option is to do all these broad changes at once, then help players to recover from it. During alpha when we had to make huge stat changes, we created a special "Cheap Transmutation Week" event, where Transmutation didn't consume durability. This let high-level players efficiently re-roll their gear to replace mods they no longer wanted. We could do that again, or something similar.

So there are two ways: slow incremental changes, or we could rip the band-aid off and do it all at once, then help accelerate the healing. Much more dramatic, but maybe less annoying overall? I haven't decided yet, but I am leaning toward the slow incremental plan.

Either way, this change will mostly affect players in the 50+ range, because that's when the mod-stacking becomes problematic. If you're lower level than that, you'll see changes, but not all that much.

Off-Hand Knife

We now have some basic animations for wielding a knife in the off-hand, and off-hand knives should be in the next update!

The Knife skill can still be used by wielding a Dagger in the main-hand (and all existing knives will be considered Daggers). We'll also be adding new off-hand knives (called just "Knives", or maybe "Shivs" or "Stilettos" or something similar, to easily differentiate them from the main-hand weapons.) You can wield a main-hand knife, an off-hand knife, or both -- and I may add a few tricks for people who want to wield two at once.

The more interesting thing to me, though, is the number of other skill combinations this opens up, such as Sword+Knife or Knife+Ice Magic or Druid+Knife. Lots of synergies to explore! Knife was always meant to have both off-hand and main-hand options, so this will help it reach its full potential.

Damage Over Time (DoT) Ability Changes

One balancing problem with the Knife skill is that it relies on DoTs (damage-over-time or "bleed" effects), and DoTs don't scale very well at high level. Why don't they scale? Well, in part that's because of what I mentioned above: monsters are getting out of whack to compensate for very high player damage. Bleed damage scales linearly, and there aren't too many ways to multiply how much damage they do. So in part, bleeds will be fixed by toning down direct damage. But there will also need to be new ways to buff bleeds and poisons.

So part of the work I'm doing for Knife is rethinking how DoTs scale, and adjusting Knife's abilities and treasure mods to work better at high level. I also need to make improvements to how DoTs are displayed on abilities, because right now they don't take your buffs into account correctly. (For instance, the "Gut" ability always says it deals "25 trauma damage over 10 seconds", even if you have trauma-damage buffs that actually make that number much higher.) Reworking this logic will make Knife a lot easier to use and understand -- and in a future update, we'll backfit those improvements to other skills that use DoTs.

New Skill: Priest

The next update will also introduce a new core combat skill, the Priest. This is a support skill that focuses on group healing and recovery, with a smattering of other things for variety.

We call it "Priest" for short, but the skill is specifically "Priest of Arisetsu". Arisetsu is the Goddess of Hope and Warmth, and she will be the patron deity of anyone that uses Priest abilities. Becoming a priest involves making a vow to Arisetsu that you will help her in the direst of times, "when all hope is dying in the world"... but practically speaking, that's not a day-to-day occurrence. Unlike "druid emergencies", there won't be any "priest emergencies" except, perhaps, during special events -- maybe a weekend event, or something like that.

Arisetsu is a well-liked and easygoing god, and anyone can become a Priest of Arisetsu. The only thing Arisetsu refuses to tolerate is the undead, and you can't use the Necromancer and Priest skills at the same time. Other than that, you can pair it with anything.

Priest has the most targeted healing potential of any skill, but some of the Priest's heals have a short "channeling time." If the priest is attacked while channeling, the spell aborts. In other words, Priests work best in a group, where they don't have to be on the front lines. But like most skills, Priest has versatility and room for customization. There's some unique debuffing abilities, some potent undead-killing tools, and a couple of fiery attacks related to Arisetsu's "warmth" aspect. These should allow for some interesting skill combinations. In fact, I'm excited to see someone try to make Fire Magic+Priest work as a deadly solo offensive combo! ... But in general, Priest is a support skill.

Mecanim Update On The Horizon

We have a custom animation synchronization system that's built on top of Unity's original animation engine. Well, a few years ago Unity added a new animation engine (called "Mecanim"), and we're working on upgrading all our code and assets to use mecanim. This is a big undertaking that involves a ton of work, and the work is being done in a separate branch of the code until it's ready. It won't be ready for at least a few updates... but when it's finally ready, it'll be a big deal.

First, it will let us finally fix some of the annoyingly terrible animation glitches that we just can't fix right now due to the rigidity of our old animation setup. Second, it will let us blend animations better, allowing for more reasonable-looking combat animations. Third, we're plugging in new animation features necessary for mounts. (Just to be clear, mounts will take additional work on top of the mecanim update. But that will be one of the big features we focus on after.)

I don't have any kind of an ETA on the Big Mecanim Update yet, but it's on the horizon.

(Unfortunately for us, the Mecanim system uses a single-threaded implementation under the hood, the same as the older Unity animation system. So switching to Mecanim won't magically improve the game's framerate in crowded areas... but mecanim looks a LOT better! That's the point of this update: laying the groundwork for better-looking animations.)

Other Changes

The next update will be in about two weeks -- hopefully less, actually, but let's say two weeks.

Time permitting, we'll also be tweaking some other skills, nerfing some problematic abilities and buffing a few others. But that's not the primary focus of the next update, so it may get pushed out to a different update.

I also want to make some incremental quality-of-life improvements to the UI, but I doubt too much of that will fit into this update; we'll see how it goes.

That's it for now. I'll be back soon with another dev blog. Thanks for playing and supporting us!