Open source: HotPatcher development progress

It’s been more than half a year since the last version of HotPatcher was released. It has good support for UE4, but lacks support for UE5, and many people in the group frequently ask about the adaptation status.

The last Release version of HotPatcher, v81.0, was released on 20230604. It only supported UE5.1.0 and did not support WP and Nanite. Subsequently, Epic launched UE5.2 and 5.3, but the plugin was not adapted. Although compilation errors could be fixed manually, the feature support in UE5 was in a semi-finished state.

Taking advantage of the Qingming Festival holiday, I worked hard for several days, performing comprehensive optimization and adaptation support for HotPatcher. It can now be compatible with the latest versions from UE4.21 to UE5, and a new Release version has been published: v82.0. The latter part of this article will introduce the adaptation status, precautions, and update log of HotPatcher in UE5.

Additionally, in 2020, I wrote an article: Some Thoughts and Ideas on Open Source. Over the past two years, I’ve had some new ideas about maintaining open-source projects, which I’ll summarize here.

Foreword

The main reasons why HotPatcher hasn’t been updated for so long are:

Busy with work, not much spare time to handle adaptation tasks.
The fragmentation of UE5 versions is severe, and each version has significant changes, making it troublesome to achieve compatibility across multiple engine versions.
Many Mods have been developed based on HotPatcher, which also need to be adapted as HotPatcher is updated.

The workload required is still considerable. It’s not that there was no adaptation work done before; adapting to UE5 and its implementation mechanism involved several modifications and verifications. We also needed to ensure that support for UE5 wouldn’t affect the plugin’s compatibility with other engine versions or with Mods, which is why progress was slow.

I often saw people in the group urging and asking about UE5’s adaptation status, and I could only helplessly say I would get to it when I had time.

Maintaining open-source projects is actually a very time-consuming endeavor, and I’m not considering using it for profit. When both my day job and personal life are tight on time, supporting open-source projects out of passion has to be deferred.

Many people have told me they are willing to pay to support me in prioritizing UE5 adaptation, and some companies even wanted to buy out my adapted version as a closed-source tool.

But I don’t want to commercialize it. I started developing this project in 2019, and it has been five years now. In these five years, I’ve written a lot of code, adapted many engine versions, and ensured cross-platform and cross-version compatibility for the code, with the same code supporting UE4.21~UE5 simultaneously. This is very difficult.

Five years, this project is like a child I’ve raised myself; I wouldn’t charge for it or sell it. Charging would change its nature for me; the effort invested over these years far exceeds its monetary value. Moreover, it’s already being used in a large number of commercial projects, and selling it would be like killing the goose that lays the golden eggs; it’s not something I can do out of ethics.

I still hope it can remain as it was initially. I simply enjoy writing code, and I like making tools useful and capable of generating more value; that’s what makes me happy.

Regarding income, I haven’t directly earned revenue from this project. However, my blog has a tip jar, and occasionally friends tip tens, twenties, or even hundreds of yuan, totaling no more than a few thousand yuan. While not all of it can be counted as HotPatcher donation income, it’s better than nothing overall. Compared to the amount, the emotional value it brings is higher.

I’ve always believed that commercializing tool development projects is never a good business. This is even more true for game development tools; the audience is too narrow, and most programmers can handle their own needs, so why buy from others? Furthermore, if something can be obtained for free, people often won’t pay, and piracy is also a common situation.

Literary people disdain each other, and programmers do too.

Moreover, if I were to charge, and my schedule didn’t allow me to quickly respond to issues and adapt versions, I would feel very ashamed. Without charging, there’s no psychological burden. So I’d rather keep it free and open-source than have my work be accused of being a cash grab.

Being bothered by these trivial matters would be putting the cart before the horse. Therefore, for this project, I insist on no selling, only donations.

However, the income from donations is unpredictable and completely insufficient to support myself. Open source can only be a hobby, not a means of livelihood.

Therefore, in the future, my maintenance and support for open-source projects may maintain a relatively low update frequency and energy investment. I’ll focus on my day job and life first, and then contribute out of passion when I have spare capacity.

Adapting to UE5

Let’s talk about HotPatcher’s adaptation to UE5.

Currently, HotPatcher’s adaptation for UE5 has been completed. It can properly package and mount all types of assets, and supports WP and Nanite. I have verified it on UE5.3.2, 5.2.1, UE4.25, and 4.27, and no significant issues were found.

The UE4 engine can still be used out of the box, but UE5 is slightly more complex. Due to existing bugs in the UE5 engine, some HotPatcher features cannot function properly in UE5 and require some fixes using the source-code version of the engine.

Although I made many efforts during adaptation, hoping to implement it purely with a plugin and avoid modifying the engine as much as possible, the plugin’s capabilities are limited after all. Some of UE’s code implementations really make me want to exclaim that it’s a pile of crap.

Below, let’s discuss the specific bugs and the fixing process.

Crash issue with FShaderMapResource_InlineCode::CreateRHIShaderOrCrash in UE5.3.2 (not found in UE5.2)

In UE5.3.2, when packaging a patch containing Shaders, the following crash will occur:

1
2

LogOutputDevice: Error: Ensure condition failed: oldValue == newValue  [File:D:\build\++UE5\Sync\Engine\Source\Runtime\Core\Private\HAL\ThreadingBase.cpp] [Line: 311] 
LogOutputDevice: Error: oldValue(1) newValue(0) If this check fails make sure that there is a FTaskTagScope(ETaskTag::EParallelRenderingThread) as deep as possible on the current callstack, you can see the current value in ActiveNamedThreads(102), GRenderingThread(6110a4c0), GIsRenderingThreadSuspended(0)

Fix method: Remove the call to IsInParallelRenderingThread() from CreateRHIShaderOrCrash. Epic is aware of this issue and will fix it in UE5.4.

The issue where FShaderLibraryCooker::EndCookingLibrary in CookOnTheFlyServer is not controlled by the engine’s bShareMaterialShaderCode.

Normally, FShaderLibraryCooker‘s Init and Shutdown would only be executed in CookOnTheFlyServer if PackageSetting->bShareMaterialShaderCode is true. However, in UE5, only the Init-related logic is checked, while Shutdown is not checked at all and will always execute.

This causes HotPatcher to produce errors when packaging ShaderLibrary, preventing COOK from executing properly. The only options are to uncheck SharedShaderLibrary in the Patch configuration or fix the engine code.

Note: In UE5.7 and later versions, this issue requires modifying the UCookOnTheFlyServer::ShutdownShaderLibraryCookerAndCompilers function by adding an IsUsingShaderCodeLibrary() check within the function body.

Fixing solutions:

Modify the implementation in CookOnTheFlyServer.cpp to add a bShareMaterialShaderCode check (or IsUsingShaderCodeLibrary()) to the FShaderLibraryCooker operations.
Or disable SharedShaderLibrary in HotPatcher’s packaging configuration:

Both of these issues are entirely caused by UE’s own bugs and cannot be resolved at the plugin level. Anyone intending to use HotPatcher in UE5 must manually modify these two locations in the engine for all HotPatcher features to work correctly.

Similarly, the second issue also prevents the Launcher version of the UE5 engine from properly packaging ShaderLibrary Paks using HotPatcher; either InlineShader must be used, or the source-code version is required.

Although I insist on maintaining version compatibility with a single codebase, these engine bugs make my OCD feel worse than swallowing a fly, but there’s no other way. Until they are resolved in the UE mainline, it can only be used as described above.

Update Log

v82.0

v82.0 has undergone extensive optimizations and bug fixes compared to the previous version.

Regardless of which engine version you use HotPatcher with, it is strongly recommended that you update to the latest version to facilitate troubleshooting should issues arise.

The updates are as follows:

Fixed a bug where *.m.ubulk files were not packaged in some cases.
Optimized PAK packaging time, with caching and multi-threading optimizations for scanning large numbers of files, significantly increasing packaging speed.
Added support for Accompany’s COOK mode.
Added more performance markers.
Optimized log levels.
Supported cache-ddc-only Cook mode.
Automatically load all ShaderLibrary from the ../../../PROJECT_NAME/ShaderLibs directory when the module starts.
Supported COOK for Nanite and WP.
Supported differential updates for WP (due to WP’s One File Per Actor mechanism, modifying a WP scene does not change the level assets themselves. I extended a detection mechanism to enable differential detection for WP).
Optimized access to FExternFiles.
Optimized Release record of external files from absolute paths to marked paths.
Fixed the issue of generating fake files during COOK in UE5.
Fixed the issue where exported release paths contained //.
Fixed the issue where PatcherProxy obtained inaccurate PackageTracker results.
Optimized PackageTracker implementation, removing already Tracked assets.
Optimized PackageTracker, enabling recursive Tracker analysis.
Unified ParallelFor locks in CollectPakCommand to avoid race conditions.
Supported packaging a list of LongPackageNames via the -AddAssetsByFile= parameter.
Fixed the issue where ShaderLib was not packaged into the pak.
Allowed adding Pre and Post tasks within the Cook Cmdlet.
Optimized plugin Slate implementation, supporting secondary menus in the Toolbar to directly open corresponding pages.
Supported monitoring process Crash, and storing Crash files in Saved/HotPatcher/Crashs.
Compatible with both GameFeaturePacker and ShaderPatcher Mods (both need to be updated to the latest version).

Currently, v82.0 has been officially released and can be pulled and updated from hxhb/HotPatcher. Older versions of the plugin will also display new version update prompts when opening the Patch page.

Supplementary Information

Update by 2024-04-13 13:45

For WP map updates: added some supplementary information.

For regular maps, modifying a Blueprint base class automatically triggers changes in child Blueprints and maps that reference it:

However, for WP, the situation is more complex:

With WP’s one file per actor enabled, modifying a Blueprint does not necessarily trigger map updates.
Analysis logic needs to be implemented for this situation and will be released in the next version.

Donations

If the HotPatcher project has helped you, and you wish for HotPatcher to improve further, you can scan the donation QR codes below to support the author.

WeChat Pay	Alipay

After donating, you can click on the HotPatcher Donor Information Collection Form to submit your donation information. I will periodically collect and display it on the project page.

Conclusion

After 5 years of development and evolution, HotPatcher has become the most popular asset management and packaging solution. This update, with its UE5 compatibility and extensive optimizations, I hope it will continue to shine and generate more value in the UE5 generation of engines.