MHServerEmu Progress Report: January 2024

MHServerEmu Progress Report - January 2024

Hello there! The first month of 2024 is now almost behind us, and we have been hard at work on all things Marvel Heroes. Let us dive right in!

CalligraphySerializer is a Lie

Before continuing I strongly suggest you read the reports from November 2023 and December 2023 if you have not done so already. You have been warned.

Most of my time in January has been spent untangling the web that is the CalligraphySerializer class. And the fun begins right from the name: despite being called a “serializer”, it literally cannot serialize any data. It is used strictly for deserialization of Calligraphy prototypes. You literally cannot trust things to do what they say they do.

What it actually does can be summarized like this:

It takes a blank prototype instance of a certain type as input.
If the prototype references a parent prototype, the parent data is copied to the child. If the parent has not yet been loaded, it goes through the deserialization process itself, and this continues until all parents are loaded.
All serialized Calligraphy field groups are processed. Regular field group have their fields matched to fields defined in the prototype class, its parents, and mixin fields, and then deserialized using the appropriate parser method. Property field groups represent individual properties and are used to reconstruct property collections.
A prototype can have embedded prototypes, so this whole process is recursive.

As of right now, we have most of the regular field group processing working, but we still need to implement property collection reconstruction, as well as some small things here and there. Our overall current implementation progress is represented in the following diagram:

One of the biggest challenges I have had to deal with implementing this has been matching serialized fields to appropriate class fields. The client uses a custom run-time type information system for this called GRTTI. While it should be technically possible to reimplement GRTTI, it would also be somewhat like reinventing the wheel when our server emulator is written in C#, which already has very strong reflection capabilities that can be used to get a very similar result.

C# reflection has its drawbacks though: while extremely flexible, it is notoriously slow, and when you have to use it literally hundreds of thousands of times, it really adds up. In one of our unoptimized testing cases it took over 80 seconds to load all prototypes, while the original game did it in 3. In the end, with some targeted result caching, I have been able to get it to a pretty reasonable time of about 12 seconds, which should be more than good enough for our current needs. There is always more room for more optimization, but that is going to come later on. It should also be noted that all of this is a one-time cost during server startup, and if you do not have prototype frontloading enabled, you are most likely not going to notice this at all.

Another big issue that has come up has been taking care of mixins. While most embedded prototypes can be handled with recursion, there are some very specific cases where you have so-called mixin prototypes that need to be deserialized as if they were a part of the prototype itself. Their field groups are mixed with regular field groups (which is where I am guessing the name is coming from), and there are cases where you have entire collections of mixin prototypes. There is a significant amount of extra code that handles mixins specifically, and they are used in literally only three prototypes classes (out of over a thousand) in the entire game: LocomotorPrototype and PopulationInfoPrototype in AgentPrototype, ProductPrototype in ItemPrototype, and lists of ConditionPrototype and ConditionEffectPrototype in PowerPrototype. So in the case of list mixins there is literally a single prototype class that this entire subsystem exists for.

While there has been very good progress, the work on CalligraphySerializer is still not done. It is now possible to access a lot of the game data from code and use it in features like region generation, but there are still some aspects of mixin fields that need to be finished, as well as the property system, which is tightly tangled with the entire game.

Pipeline Improvements

Another thing I have been able to work on this month is some general improvements to our delivery pipeline.

First of all, MHServerEmu now clearly states its version, build time, and build configuration:

This is a pretty minor change, but it is going to make it much easier to figure out and solve issues as they arise.

Also, we now have automated nightly builds powered by GitHub actions and nightly.link. Thanks to that, if you want to try MHServerEmu out, instead of building the source code yourself, you can just download the latest build here. Those builds are based on the the latest master branch code, and are in general a decent representation of our current progress.

These small steps bring us closer towards a big milestone, which would be our first official binary release, tentatively numbered 0.1.0. We are going to talk more about that in the future.

Here Comes a New Challenger!

This month our development team has expanded by 50%: Kawaikikinou has joined me (Crypto137) and AlexBond in restoring Marvel Heroes back to its glory.

MHServerEmu is a very complex project that involves many moving parts, so some time had to be spent to bring our new team member up to speed with what we are working on, but there are already some results that can be seen. So far Kawaikikinou has been mostly helping AlexBond with ironing out issues with region generation (more on that later), as well as improving the reliability of our code by introducing unit testing.

Another thing I am personally very excited about is the new region visualization tool he developed. Knowing is half the battle, and getting a better visual idea of what exactly is going on on the backend is going to be very helpful in the long run.

The Joy of Region Generation with AlexBond

This time I am joined by a co-writer: in this section AlexBond is going to go in-depth on what is happening behind the scenes with region generation.

Hello everyone. I am AlexBond, and I am working on reimplementing the region generation. We are lucky to have the region generation code ifself present in the client, but it is disabled. Most likely it was among common game files, which is why it ended up in the client, or maybe the developers had the option to enable it for testing. In any case, in two months I was able to reimplement the region generation in C#.

Of course, I am not without faults, so we ended up with a number of bugs. So this month me and Kawaikikinou have been testing this new code and fixing all the issues. As of the time of writing, we have tested regions up to chapter 3, and it is also possible to load into other chapters with some workarounds. Our goal is to get an exact copy of a region generated by the client when it has generation enabled. We do the comparisons using logs and the IDA debugger. Over the course of testing I was able to get a slightly better understanding of the generation process, I would like to share with you what I have discovered.

Region Prototype

A region prototype contains information about the type of generator a region uses. There are three region generators in total, but for now we are going to take a look at the most complicated and interesting one - SequenceRegionGenerator.

SequenceRegionGenerator is a generator that describes a tree of areas, and this tree contains all possible variations with weights for each one. Branches of this tree contain area prototypes. Overall this can be represented as a kind of a chain sequence:

Area 1 - Area 2 - Area 3 - an so on

The ProcessSequence() function runs recursively through the tree of areas. During its run it uses a random seed to determine sequences (PickSequence()), areas (PickArea()), their positions and connections to other areas (PickAreaPlacement()), and in the end it runs the generator of the current area. This recursion goes through the entire tree from the end to the beginning.

For example, in Madripoor it goes like this:

LowTown -> Cove -> BambooForest -> Beach

If an area generator runs into an error, all previous areas are destroyed and rebuilt until the error disappears. To prevent infinite rebuilds, there is a special parameter called MaxGenerationTimeInSec, but currently I am not using it (perhaps I should).

Area Prototype

An area prototype defines the type of generator and the cell generation rules. In total there are seven different area generators, but I would like to talk about the most commonly used ones - WideGridAreaGenerator and CellGridGenerator (GridAreaGenerator).

CellGridGenerator

CellGridGenerator is a rectangle of cells, and the size of this rectangle is defined in the area prototype.

The generation process includes ten attempts of creating an arrangement of cells that adheres to certain requirements. Now I am going to describe creation functions.

InitializeContainer() - creates a CellContainer with a size of CellsX by CellsY. This is just a rectangle for holding cells that are going to be added.

EstablishExternalConnections() - defines which sides are going to have cells for connecting to adjacent areas.

GenerateRandomInstanceLinks() - defines connections to various RandomInstances (not all regions contain those).

CreateRequiredCells() - defines groups of cells to be placed:

RequiredSuperCells - places consisting of multiple cells (mostly 2x2), like the Fall Tribe village in Savage Land, or buildings.
NonRequiredSuperCells - this type is used only in the MutateMarshArea.
RegionTransitionSpec - these are mostly places for waypoints and entrances to various treasure rooms. These places are determined using teleports contained in RegionConnectionNodePrototype that exit to this region.
RandomInstances - a list of random places, these are not in the game…
RequiredCells - these are special places that must be present in an area, like shops or cafes (e.g. Madripoor_Lower_BobaTeaPOI_A).
RequiredPOICells - cells from POIGroups, I cannot say for sure what they are (they appear in Wakanda, DangerRooms, CowRegion).
NonRequiredNormalCells - another group of cells, used in HellsKitchen (Brownstone_C_Barricade_ESW_A).

Now we have a filled CellContainer with a number CellRefs, and if it is successful we can move onto destroying and adding cells to the current area.

ProcessDeleteExtraneousCells() - this process uses RoomKillChancePct to determine the number of cells that need to be destroyed using the RoomKillMethod. There are three deletion methods - Random, Edge, and Corner. When deleting cells there are checks for paths between imporant points, and only the cells that can be safely removed without breaking those paths are removed.

ProcessDeleteExtraneousConnections() - removes redundant connections between cells using ConnectionKillChancePct.

ProcessCellTypes() - the main function that randomly picks cells for creation. If a cell has not been previously defined, it is going to be picked from the appropriate CellSet based on cell type.

At this point CellGridGenerator either succeeds or fails. If it fails, all areas are destroyed and regenerated.

WideGridAreaGenerator

WideGridAreaGenerator differs by having GenerateRoads() and CreateProceduralSuperCells(), as well as a different implementation of ProcessCellTypes() that picks cells based not on their type, but RequiredWalls.

GenerateRoads() - defined by the Roads prototype in the generator. This function uses the DijkstraRoad() method based on the Dijkstra’s algorithm for finding the shortest path between nodes.

CreateProceduralSuperCells() - if the ProceduralSuperCells field contains data, it creates a random set for a 2x2 square and rotates it randomly.

Here is an example of generation for CH0204Q36AIMLabAreaA:

CellsX = 4
CellsY = 3
RoomKillChancePct = 60
ConnectionKillChancePct = 15
RoomKillMethod = Corner

X	X	X	X
X	X		X
B			A

As we can see, in this case the path from A to B is preserved, and 60% of the cells were removed using the corner method.

This is what we get in-game:

Now let us take a look at the next area, CH0204Q36AIMLabAreaC:

CellsX = 3
CellsY = 3
RoomKillChancePct = 60
ConnectionKillChancePct = 15
RoomKillMethod = Random

X		B
C		X
X	X	X

As we can see, in this case the path from B to C is preserved and 60% of the cells were removed using the random method.

And here is what it looks like in-game:

Hopefully now it should be more clear how all of this functions.

As for my work, recently I have merged the new prototype system written by Crypto137, and now the testing of generation is much faster, and it is going to be easier to merge with the main repository. But we still have a lot of tests to do, so please be patient.

Back to Crypto to wrap things up. We are getting very close to finally reaping the fruits of our efforts over the past few months, and hopefully we will be able to deliver results that are more visible in-game soon enough. We are just as excited as you are to finally see some things come together. Until next time!