MHServerEmu Progress Report - May 2024

Dread it, run from it, MHServerEmu Progress Report arrives all the same.

Back in Time

This month there has been an unexpected breakthrough. As I was finishing combing through the client’s serialization routines we talked about in the previous report, I came to a realization just how little is actually needed to get in-game in some form. So I had an idea: what if we had a heavily stripped down version of the server, a minimum viable product with as much version-specific functionality removed as possible? In theory, this would allow us to get in-game using pretty much any version of the game client, which could be useful for various reasons. I decided to test this idea in practice by taking a build of version 1.10, dating back to the game’s original launch in June 2013, and seeing if this turns out as simple as I thought. If I could get 1.10 running, any version from that all the way to 1.53 would be feasible.

I started by taking just the MHServerEmu.Core library, which handles aspects such as low level serialization and networking, and did a new lightweight server implementation with it. To my surprise, after just a couple hours of work on a single evening, I was in-game.

I see this as an absolute win.

I see this as an absolute win.

My hypothesis turned out to be correct: not only all post-launch versions of the game are fundamentally very similar, but there is only one thing that can cause the process of putting a player into a game to fail - not having a valid name specified for the player. The only other little thing that needed fixing was the fact that older versions of the game do not allow entities with zero health to move around, but with just a little bit of extra work I was able to solve this.

One thing that is now possible is making use of some of the internal builds we were able to recover from Steam. These builds have all console commands unlocked, some of which can be used to get a better understanding of how the game works under the hood. For example, there is a number of visualization features for various aspects of the navi system, including mesh triangulation, pathfinding, entity bounds, and much more. Here is Avengers Tower with debug visualization for navimesh, collision shapes, physics sweeps, and pathfinding.

Navi Debug Mode

In addition to that, we can now explore some of the content that was never finished and later ended up being removed from the client. Two major examples of such content are the Savage Land patrol zone, also referred to as United Tribes in the game files, and the Thanos raid that was going to be centered around Knowhere. Both of them were mentioned by Gazillion when promoting the Marvel Heroes 2016 rebrand, and early versions of them are present in version 1.48, also known as the last pre-BUE version. The screenshots below represent this content the way it was in December 2016, a little less than a year before the game’s shutdown.

Savage Land Siege Patrol / United Tribes is the one closer to being finished. Although it is referred to as a “patrol zone”, apparently it was going to be more of its own thing, featuring some form of tower defense gameplay. The region layout present in the client resembles three-lane MOBA level design, but with additional twists and turns reminiscent of tower defense games.

United Tribes Map

Players start at the base where they can go down one of three lanes.

United Tribes Base

Here is what the middle of the map looks like.

United Tribes Mid

And on the opposite side of the map there is what appears to be an unfinished boss arena that uses the Red Onslaught’s model from the Axis raid as a placeholder.

United Tribes Boss

On the cosmic side of things we have the Thanos raid. As far as we can tell, it was going to feature Knowhere as a new hub region, similar to Genosha that accompanied the Axis raid, and this same hub was going to be the starting area of the raid itself. This hub features special camera settings, making it rotate around the center as you move on the outer ring.

Knowhere Hub

The raid itself was planned to include eight areas (the descriptions are mostly speculations based on file names and layouts):

  • ThanosRaidPart1Area, containing four subareas - Knowhere_Spoke_A, Knowhere_Spoke_B, Knowhere_Spoke_C, and Knowhere_Spoke_D. Knowhere_Spoke_A is the only section of the raid that features mostly completed environment artwork. All other areas have only greyboxed layouts that occasionally reuse existing assets.

Thanos Raid Spoke A

  • ThanosRaidPart3AreaBlackDwarf (also referred to as Knowhere_Hub_Part2_BlackDwarf in the cell names), which appears to be an arena where players would have fought Black Dwarf.

  • ThanosRaidPart4AreaTransportBay (Knowhere_Transport_Bay_A) looks like a section where players were supposed to fight their way through to a spaceship.

  • ThanosRaidPart5AreaShipCombat is some kind of encounter where players were supposed to defend their spaceship acquired in the previous area.

  • In ThanosRaidPart6AreaSupergiant player would board Thanos’ flagship, Sanctuary II, and fight Supergiant. In the beginning of this area there is what looks like the debris of the ship players acquired and defended in previous areas.

  • The next area is called ThanosRaidPart7AreaGauntlet, and it appears to be an encounter where the raid would split into two teams and fight their way through Sanctuary II in series of three rooms for each team.

  • It is followed by ThanosRaidPart8AreaCorvusProxima , the bridge of Sanctuary II, where players would have fought Corvus Glaive and Proxima Midnight.

  • The same layout is reused for ThanosRaidPart9AreaThanos, and supposedly this is where the fight with Thanos himself was going to happen.

Thanos Raid Thanos Arena

One other version we are now able to examine is 1.53, which is the version of the game that was on the test center when the game was shut down. It features:

  • The costume closet system, which would turn costumes into options you could unlock and select in a separate panel, rather than equippable items.

  • Playable Spider-Woman.

  • An unfinished version of the playable Gladiator Thor. I mean Unworthy Thor. Actually, he is also referred to as just “Odinson”. Whatever his name was going to be, this was going to be an alternative, distinctly Mjolnir-less, version of Thor as a completely separate hero.

  • Apocalypse terminal from the console version of the game.

  • Omega items, which were going to be some form of equipment set system with bonuses for equipping multiple items from the same set.

  • A number of costumes for various heroes, including Apocalypse-themed costumes for Psylocke, Magik, Jean Grey, and Storm, costumes based on the Thor Ragnarok film for Thor, Loki, and Hulk, and other costumes not directly related to a specific event.

Unworthy Gladiator Thor Odinson

One issue with this version is that its newest build, 1.53.0.203 from November 8th 2017, does not contain the assets for all the new content, such as new costumes. If you want to get the most content, you have to go back to at least 1.53.0.62 from two weeks prior to that, with all the extra bugs that entails. Even the earlier build though lacks the assets for some of the new regions, such as a new event terminal referred to as Hel Unleashed / Hela Palace, as well as a five-man version of the Muspelheim raid, both of which appear to had been planned as Thor Ragnarok tie-ins. While we do still plan to support version 1.53, as it is technically very similar to 1.52 we are currently focusing on right now, the definitive post-BUE version of Marvel Heroes is most likely going to be some kind of modded amalgamation of multiple versions.

The biggest takeaway from this development though is that it has definitively proven the viability of restoring pretty much any version of the game we have the client for. Just booting any version of the game now takes literally minutes (except for beta builds that have some additional jank that still needs to be figured out). It would require some effort to go as far back as 1.10, but adapting our 1.52 work for both 1.48 and 1.53 is most likely going to be significantly less difficult than we expected. As for the mini-emulator we have been using, we plan to share its source code relatively soon, once it is in a slightly more polished and user-friendly state.

The Road to Area of Interest

Back in the realm of version 1.52 some significant backend progress has been achieved this month. The next major obstacle we need to overcome to get the game to a more playable state is implementing a proper area of interest system, and for that we had some cleaning up to do.

An area of interest is a server-side representation of what a particular client is aware of. When something changes in the game state (for example, your health goes up or down), only the clients that are interested in the affected entities should be notified. To find the interested clients, we need to be able to iterate connected players and check each of their individual areas of interest. However, in our previous temporary implementation of handling asynchronous network events a player could join or leave the game at arbitrary times, causing the iteration process to fail.

So one aspect that needed to be fixed is putting aside a specific point in the main game loop when new players could be added and removed. Whenever asynchronous events would happen, they would simply enqueue players to be added or removed during the next update. There is code for such behavior in the client, and we modeled our implementation after it. Same as Gazillion, we use a double buffer style approach: we have two instances of each queue, and when it is time to process joining / leaving players, the only thing we do in a lock is swap pointers to these two instances. While we process players in the instance we got, the other instance continues being filled asynchronously with players we are going to process in the next update. In the end this protects the game thread from asynchronous multithreaded weirdness and allows us to safely iterate players within the main loop.

One unfortunate side effect of this implementation was that it broke persistent player data saving. To understand how this happened we need to look at the bigger picture. Here is a rough overview of the server achitecture the game expects based on our observations:

Player Manager

The client connects directly only to the frontend server (FES). FES then relays client messages to the player manager, which distributes players across various game instances hosted by game instance servers (GIS). Because the player manager is aware where each player is, it can relay messages received from the frontend further down to the game instance where they belong. The player manager is the ultimate overseer of the entire game, and it would make sense for it to be the one interacting with the database. For those familiar with Diablo and other Blizzard games, the role of the player manager there would be handled by Battle.net.

We currently follow this structure pretty closely, but with some deviations. We have only a single server, and game instances are managed directly by the player manager without a separate GIS, but the overall flow is the same. So when a player disconnects, it raises an asynchronous event on the frontend server, that is then relayed to the player manager and handled by it. This handling includes removing the disconnected player from the game instance they were in and saving their data. However, now that the processing of joining and leaving players is deferred to a specific point in time, if you save player data to the database straight away, you are going to save an outdated copy of it, because the game instance most likely will have to wait until the next game frame to update the database model that is written. To fix this, we had to come up with a mechanism that also defers database writes until the game instance finishes processing leaving player. Added to the mix is also a possible case of multiple clients attempting to use the same account. Multithreading is fun!

With that out of the way, the next thing that needed to be done was cleaning up our entity management system and implementing inventories. One issue we had still hanging from the days of working with hardcoded packets was that player and avatar entities existed outside of the entity manager, which made it impossible for them to interact with other systems, such as areas of interest that we need to implement. However, if we were to just mix players and avatars with all the crates, cars and Maggia goons spawning all over the world, it would make it somewhat difficult to clean up when a player leaves a game. This is where inventories come in.

When most people think of an inventory, they probably imagine a grid full of epic loot. Marvel Heroes takes a more generalized approch: any entity can be in an inventory of any other entity. So all of your inactive avatars actually exist in the PlayerAvatarLibrary inventory. The avatar you are currently playing as gets put in the PlayerAvatarInPlay inventory. All of your team-ups exist in the PlayerTeamUpLibrary inventory. Same goes for summons that are contained in the AgentSummonedEntities inventory. Things you would expect to be stored in an inventory, such as equipment, is also included. To summarize, the inventory system in the game actually represents ownership relations between entities.

So by implementing the inventory system we could solve the problem of cleaning up all the various entities related to a player simply by recursively iterating all of their owned entities. And this is exactly what I did: we now have all the basic logic needed for adding, moving, and removing entities within inventories, as well as a lot of validation code mirroring the client implementation of the system. There is still some work to do before items you all know and love begin showing up in various inventory grids in the UI, but the implemented functionality is enough to take care of avatars, as well as team-ups, who now show up as unlocked in latest nightly builds.

With all of this cleanup and refactoring we now reach a point where we can work on properly implementing the area of interest system, which is going to be the foundation of all client-server communication going forward. Once that is done, we are most likely going to enter another period of explosive progress, similar to the one we had in March when we got the game database fully working.

Behaviorism and Us

Before we wrap things up, here is also a quick update on what AlexBond has been up to. While I have been focusing on the more foundational aspects of the server, he has been digging into a system that is going to become rather important later on - behavior, also known as AI.

We are most likely going to go into more details on this in a future report, but here is an overview of what it looks like. Behavior affects anything that can do anything on its own: one obvious example of this is enemies, but there are also team-ups, pets, projectiles, and even orbs and boons that drop from various sources. It is implemented as a finite state machine with fourteen possible states: Delay, Despawn, Flank, Flee, Flock, Interact, MoveTo, Orbit, Rotate, SelectEntity, Teleport, TriggerSpawners, UseAffixPower, UsePower, and Wander. Transitions between these states are controlled by various procedural profiles, of which there are over a hundred in version 1.52. Some profiles are more generic, such as ProceduralProfileBasicMeleePrototype, while others are very specific, like ProceduralProfileDrDoomPhase2Prototype. The data for procedural profiles, such as what powers to use and how often, is defined in prototypes contained in the game database.

Surprisingly enough, the client contains not only all of the data, but also the vast majority of the code for procedural profiles. As far as we can tell, the main reason for this is that the AI system was also utilized by bots designed to stress-test servers with artificial load. While the full functionality for this feature is removed in shipping builds, there are enough traces of it for us to reverse engineer.

Over the course of May Alex has reimplemented 114 procedural profiles, and is now working on states. Once some more of the foundational systems are working, such as areas of interest and locomotion, this should bring some more life back to the game. It is not going to be “playable”, because we are most likely not going to have a working power system implementation for quite some time, but it should in theory make things like having a team-up or pet follow you in the game possible.

And with that the third and final progress report of this spring is over. See you in the summer!