MHServerEmu Progress Report - August 2024

The development of MHServerEmu keeps chugging along.

Back to Basics

After getting sidetracked with implementing some of the more “flashy” features in July, such as powers and loot tables, it was time for me to go back and continue some of the backend work I started back in June.

While our area of interest (AOI) implementation did work, it still relied on some of our earliest code from July-August 2023 when we were just starting getting the hang of things. For instance, the code that would put players into regions was a mess of hardcoded hacks, resulting in limitations such as not being able to teleport outside of a client’s area of interest without completely exiting and reentering the region. Combined with the lack of persistence, this resulted in, for example, your data being reset when you would transition between floors of tower regions, and it also made resurrecting avatars at checkpoints very hard to do, which we had to circumvent by doing resurrections on the spot. After painstakingly going through the old code and refactoring various things, those limitations are now gone, which you can see in current nightly builds of the server.

Another set of issues that had to be dealt with was how AOIs were interacting with the environment (areas and cells). When there are no players around world entities, like enemies and NPCs, most of them stop being simulated, meaning their AI turns off and they no longer become interactable for systems like powers. The problem was with how we determined when a player was “nearby”: we reused the same proximity volume around the active avatar we use for periodic entity scans, and turned simulation on or off based on whether an entity was replicated to a client. This caused frequent simulation switching and made enemies “forget” about players at relatively small distances, especially in larger cells, like the one used in Holo-Sim, where the whole region is just one big cell. So we switched to doing simulation updates per cell instead of per individual entity: if a cell is visible to any player, all world entites in it will now be simulated, making the whole experience more consistent.

Here is a diagram to better illustrate what was happening:

AOI Simulation Range

In this example the player’s AOI proximity volume (indicated in green) intersects cells 1, 2, 4, and 5, so those are the cells that are loaded client-side. Cells 0 and 3 are too far away, so the client is not made aware of them by the server, making them not exist from the client’s point of view. There are enemies (indicated in red) in all cells, however, only some of them intersect the player’s AOI volume. Previously, only the enemies in the volume would be simulated, while the stragglers would get turned off. Now, all enemies in all loaded cells (1, 2, 4, and 5) will continue to be simulated even if they are slightly outside the player’s AOI, allowing them to, for example, pursue the player more effectively. Meanwhile, enemies in faraway cells (0 and 3) will still be turned off.

In addition to that, the order of operations when updating proximity was also off. We were doing all entity updates at the same time, and there were cases when messages were being sent out of order, making the client try to put entities into environments that did not exist client-side, or deleting environments that still had entities in them. The whole entity processing queue had to be split into two, with entity removal happening before environment updates, and creation after, when areas and cells were ready. However, there is a catch: an entity can continue to exist client-side even if it is no longer in proximity (through ownership, discovery, or party channels), allowing its map icon to continue being displayed. In this case though, it is removed from the game world client-side when the environment for it stops existing due to no longer being in proximity. For this reason when we separate entity updates into pre-environment and post-environment we also have to take into account entities that gain or lose the proximity policy, with the former needing to be replicated after the environment, and the latter before.

Finally, one annoying thing that had to be fixed was how the current avatar could occasionally get outside of its own proximity when moving fast, causing it to disappear. The reason for this is that avatar movement in this game is mostly client-authoritative, with the server predicting client movement most of the time and only correcting when things get completely out of hand. We had our proximity update trigger tied to locomotion updates arriving from the client, but due to how the locomotion system works, an avatar can move pretty far without sending any updates as long as the movement is predictable, and this caused the aformentioned issues at high speed predictable movements. The obvious fix was to make proximity updates trigger more frequently based on server-side predictions at a given time, which is what we did. If you still see this issue, please be sure to report it!

While these little problems may not seem as important as a whole new gameplay system, like powers or AI, they are the blood vessels that allow everything else to happen, and some of them can be quite tricky to fix without breaking anything, taking a significant amount of time in the process. But with another round of those done, we can go back to more exciting matters.

Persistent Perfection

In September we plan to release our next stable version, 0.3.0, which is going to include most of the progress we were able to achieve over the course of this summer. Releasing a stable version is a moment when we are able to get feedback from some of the less engaged and technically savvy users, and when we released 0.2.0 I went through the feedback and identified two key features that I wanted to implement for the next release: powers doing damage when you hold the button down, and your loot being saved when you transfer to another region or logout (also known as persistence). With the former goal being overachieved if I do say so myself, it was now time to spend the remaining time on preserving your loot.

In April we talked about the game’s serialization system and the various modes it has. Back then we focused on replication, which is the process synchronizing server game state with clients, but the system also supports a database serialization mode used for persistent storage. And since it’s just another mode for the same system that reuses a lot of the same serialization routines, most of it is actually contained in the client. We could in theory implement some kind of custom serialization solution, but since we already had most of it implemented for replication, it made sense to just expand it a little.

One key difference between database and replication serialization is how data references are stored. For example, in February we went over various tricks the game uses for efficiently encoding properties, and as you might imagine, most of them are going to break with a changing game data set. While in runtime the game can rely on sorted file path hashes and their indexes, for cross-version compatibility needed in persistent storage the game references data using 64-bit GUIDs that remain valid even if a data file is moved, renamed, or replaced. We cannot say for sure how those GUIDs were generated, but there does not appear to be any noticable pattern, indicating that they may be hashes of some internal identifiers not exposed in the client. Properties specifically also required reimplementation of a separate PropertyStore structure used to break up property parameters into independent data references that are serialized as their own values, rather than being embedded in the id. Some data that can be derived from other data is also omitted to reduce file sizes. For example, power collections that contain available powers can be restored from the game database by knowing an avatar’s prototype and character level, so they do not need to be serialized.

One potentially neat side effect of using the original serialization implementation is that it was designed with backward compatibility in mind. In theory this should make it easier to bring data from an older version of the game into a newer one, provided we implement versioning routines for game versions we want to support. This is going to make it easier to implement progression servers that would go through versions of the game over a span of time, as seen in other legacy online games, although it remains to be seen how well this concept suits Marvel Heroes in particular.

What this serialization process gives us is blobs of serialized entity data that we need to store somehow. This is one of the rarer times in this project when we have the freedom to be creative and come up with our own solutions rather than trying to fit into an existing system dictated by the client.

For me personally, it is very important for MHServerEmu to remain self-contained and have it be as easy as a mod to set up and run, if someone just wants to play offline single player on their own machine. For this reason we are using SQLite as our default storage backend: it is embedded into the server and does not require separate setup, while also being very reliable and flexible enough to serve our needs. Depending on how many people play on a single server and how much writing load there is going to be, it may also be the only thing we are ever going to need. But there is also some room for additional flexibility via dependency injection: interaction with the database happens via an interface called IDBManager that is implemented by the SQLiteDBManager class. An instance of SQLiteDBManager is injected into the AccountManager during server initialization, and it can be replaced with another implementation that uses a different backend. Right now we also have another implementation called JsonDBManager that wraps JSON serialization in the database API, allowing you to essentially have a save file, like you would in a single player game, without changing the overall system. Similarly, it should be possible to implement a more powerful solution, such as PostgreSQL or MySQL, if the need ever arises.

Dependency Injection Using IDBManager

With great power comes great responsibility, and as we store more and more data we need to make the persistency layer more robust. While some data corruption and/or wipes are inevitable while the server is still in earlier stages of development, we understand how important your stuff is in a loot progression game like this, so we would like to keep them to a minimum. For this purpose we are implementing a number of features that are going to be coming with this persistency update:

  • The server will now automatically create backups of the database file. If something goes wrong with your primary file, you may have a backup in place to fall back to. By default the server creates up to 5 backups with an interval of at least 15 minutes between them, but this is configurable.

  • The server will now create a new database file if one is not present in the Data folder, and it will no longer come with one. This means when you update to a recent nightly build or stable release, you can safely overwrite all files without having to back up Account.db.

  • There is a new automated migration system that will upgrade your existing database file if there are any schema changes, such as columns or tables being added or removed.

As of writing this, we have most of the IO matters sorted out, but some of the game logic still needs more work to be able to properly handle entities actually persisting. For example, we had to hardcode your character level to 60 because this information did not persist on region transfers and log outs, and you would be getting reset to level 1 all the time even if we had leveling implemented. But, if we roll persistence out in the current state, everyone would get saved as level 60 because of this workaround, and when we would eventually implement leveling we would have to force a wipe of all progress to allow everyone to start from level 1. Other examples include stash tab unlocks, selected team-ups, and more. Once those are dealt with, which should not take too long, we are going to be ready to release 0.3.0.

The Road Ahead

As we wrap things up with 0.3.0, we begin looking towards what is ahead of us. So far we have been hitting our target stable release cadence of once every three months, and we plan to continue doing so. Recently we have published a rough roadmap of features we plan for upcoming stable releases: while plans change and specific features will most likely shift around as we get further along, it should give you a broad overview of the current outlook.

One large new feature that is most likely going to be ready for 0.4.0 is the mission system, and AlexBond has already spent the better part of August working on it. It is a very complex system that involves dozens of missions conditions, such as MissionConditionEntityDeath andMissionConditionItemCollect, that trigger various actions, like MissionActionEntityPerformPower, and MissionActionShowMotionComic. The mission system makes use of numerous other gameplay systems, and it is tightly coupled with the spawning system that needed a major overhaul. We will go into more detail on this in one of the future reports.

We are also planning to continue working on itemization pretty soon. The current implementation allows us to pick mostly accurate base types and qualities, but equipment currently has no affixes, which is something we have been holding back on implementing due to not having persistence working. We would not want anyone to feel the pain of getting an amazing drop, knowing that it is going to be lost forever as soon as you go back to hub.

In general, with most of the backend now being in a reasonably decent state, the focus will continue shifting towards in-game systems that are more clearly visible when playing the game.

Time for us to get back to work. Until the next report!