Open source: HotPatcher development progress

开源杂谈:HotPatcher的开发进展

Since the last version release of HotPatcher over half a year ago, its support for UE4 has been commendable, but its support for UE5 has been lacking. There have been frequent inquiries in the group about adaptation.

The last Release version of HotPatcher was v81.0, released on 20230604, which only supports UE5.1.0 and does not support WP and Nanite. Subsequently, Epic released UE5.2 and 5.3, but the plugin has not been adapted. Although one can modify compilation errors themselves, functionality support in UE5 is somewhat in a beta state.

Taking advantage of the Qingming holiday, I worked intensively for a few days to comprehensively optimize and adapt HotPatcher to be compatible with the latest versions of UE4.21 to UE5. A new Release version has been published: v82.0. The latter part of this article will discuss the adaptation of HotPatcher in UE5, precautions, and the update log.

Additionally, in 2020, I wrote an article: Some Thoughts and Ideas on Open Source, and over the past two years, I’ve had some new thoughts on maintaining open source projects. Let’s summarize together.

Introduction

The lack of updates for HotPatcher for so long is mainly due to several reasons:

  1. Work commitments have left me little spare time to handle adaptation work.
  2. The fragmentation of UE5 versions is quite severe, with significant changes in each version, making it cumbersome to achieve compatibility across multiple engine versions.
  3. Many Mods have been developed based on HotPatcher, which also need to be adapted alongside HotPatcher.

The workload required is still considerable; it is not that I haven’t done any adaptation work before. I have made several modifications and verifications to adapt UE5 while ensuring that the support for UE5 does not affect the compatibility of the plugin with other engine versions and Mods, hence the slow progress.

I often see members in the group urging and inquiring about the adaptation status for UE5, and all I can do is helplessly say that I will address it when I have time.

Maintaining an open-source project is indeed a very time-consuming endeavor, and I do not consider it for profit. When my primary job and personal life take up all my time, supporting an open-source project that I am passionate about has to be postponed.

Many people have expressed a willingness to pay to prioritize my adaptation to UE5, and some companies even wanted to buy out my adapted version as a closed-source tool.

However, I prefer not to commercialize it. This project has been in development since 2019, and it has been five years now. Over these five years, I have written a lot of code and adapted to many engine versions while ensuring code cross-platform and cross-version compatibility, supporting both UE4.21 and UE5 with the same codebase is not an easy feat.

In five years, this project feels like a child I raised, and I won’t sell or charge for it. Charging would change its nature for me; the effort I have invested over the years far exceeds its monetary value. Moreover, it is already being used in many commercial projects, selling it would be tantamount to killing the goose that lays the golden eggs, and morally I cannot do that.

I hope it can retain its original form. I simply enjoy writing code, making tools useful, and creating more value, which is what makes me happy.

Regarding income, I have never directly profited from this project. However, my blog has a donation feature, and occasionally friends donate ten, twenty, or even a few hundred dollars, totaling no more than a few thousand dollars. Although this cannot all be considered as donation income for HotPatcher, overall, it’s better than nothing. In comparison to the amount, the emotional value it brings is much greater.

I have always felt that commercializing developer tools has never been a good business, especially with game-related tools; the audience is too narrow, and most needs can be met by programmers themselves—why buy someone else’s? Furthermore, if one can get it for free, people will often choose not to pay, and piracy is a widespread issue.

Scholars disdain each other, and programmers do too.

Additionally, if I were to start charging and my time didn’t allow me to respond quickly to issues and adapt versions, I would feel very ashamed. Without charging, I don’t have psychological burdens. So I prefer to keep it open source for free rather than have what I created be criticized as a money grab.

Getting disturbed by these trivial matters would be counterproductive. Therefore, regarding this project, I insist on not monetizing and relying solely on donations.

Moreover, the income from donations is unpredictable and completely insufficient to support myself. Open source can only be regarded as a hobby and not a livelihood.

So in the future, my maintenance and support for open source projects may maintain a relatively low update frequency and energy investment. First, focus on my primary job and life, and if there’s extra energy, I will contribute out of passion.

Adapting to UE5

Next, let’s talk about the adaptation of HotPatcher to UE5.

Currently, the adaptation of HotPatcher to UE5 has been completed, and it can package and mount all types of resources normally, and supports WP and Nanite. I have verified it in UE5.3.2, 5.2.1, UE4.25, and 4.27, and found no significant issues.

The UE4 engine can still be used out of the box, but UE5 is somewhat more complicated. Due to current bugs in the UE5 engine, some functionalities of HotPatcher do not work correctly with UE5 and require some fixes in the source version of the engine.

Although I have made many efforts during adaptation to implement functionality purely through the plugin without modifying the engine, the capabilities of the plugin are inevitably limited. Some code implementations in UE are genuinely frustrating and can be described as a total mess.

Now I will discuss specific bugs and the fix process.

  1. The crash issue with FShaderMapResource_InlineCode::CreateRHIShaderOrCrash in UE5.3.2 (this issue was not discovered in UE5.2).

In UE5.3.2, when packaging patches that include shaders, the following crash occurs:

1
2
LogOutputDevice: Error: Ensure condition failed: oldValue == newValue  [File:D:\build\++UE5\Sync\Engine\Source\Runtime\Core\Private\HAL\ThreadingBase.cpp] [Line: 311] 
LogOutputDevice: Error: oldValue(1) newValue(0) If this check fails make sure that there is a FTaskTagScope(ETaskTag::EParallelRenderingThread) as deep as possible on the current callstack, you can see the current value in ActiveNamedThreads(102), GRenderingThread(6110a4c0), GIsRenderingThreadSuspended(0)

Fix: Remove the call to IsInParallelRenderingThread() in CreateRHIShaderOrCrash. Epic is already aware of this issue and will fix it in UE5.4.

  1. The issue in CookOnTheFlyServer where FShaderLibraryCooker::EndCookingLibrary is not controlled by the engine’s bShareMaterialShaderCode.

Under normal circumstances, FShaderLibraryCooker‘s Init and Shutdown should only be executed in CookOnTheFlyServer when PackageSetting->bShareMaterialShaderCode is true. However, in UE5, only the Init related logic is checked, while Shutdown has no checks and will be executed at any time.

This results in errors in HotPatcher when packaging the ShaderLibrary, making it impossible to execute COOK normally. The only solutions are to either not check SharedShaderLibrary in the Patch configuration or to fix the engine code.

Fix solutions:

  1. Modify the implementation in CookOnTheFlyServer.cpp to add the bShareMaterialShaderCode check for FShaderLibraryCooker operations.

  2. Alternatively, uncheck SharedShaderLibrary in HotPatcher’s packaging configuration:

Both of these issues are caused entirely by UE’s own bugs, which cannot be resolved at the plugin level. Anyone wishing to use HotPatcher in UE5 needs to modify these two points in the engine to ensure all functionalities of HotPatcher work correctly.

Similarly, the second issue also prevents the Launcher version of the UE5 engine from properly using HotPatcher to package ShaderLibrary Pak; one must either use InlineShader or need to use the source version.

Although I insist on maintaining version compatibility with the same codebase, these bugs in the engine make me feel worse than eating a fly. There’s nothing I can do but use the methods I’ve described above until these are resolved in the UE trunk.

Update Log

v82.0

Compared to the last version, v82.0 features a large number of optimizations and bug fixes.

No matter which engine version you are using HotPatcher with, it is highly recommended to update to the latest version, as it will facilitate troubleshooting if issues arise.

The updates include:

  1. Fixed a bug where *.m.ubulk files were not packaged in certain cases.
  2. Optimized the packaging time of PAK, adding caching and multithread optimizations for scanning a large number of files, significantly improving packaging speed.
  3. Supported Accompany’s COOK mode.
  4. Added more performance markers.
  5. Optimized log levels.
  6. Supported cache-ddc-only COOK mode.
  7. Automatically loaded all ShaderLibraries from the ../../../PROJECT_NAME/ShaderLibs directory when modules are started.
  8. Supported COOK for Nanite and WP.
  9. Supported differential updates for WP (due to WP’s One File Per Actor mechanism, changing the WP scene does not result in changes to level resources; I expanded a detection mechanism for WP’s differential detection).
  10. Optimized access to FExternFiles.
  11. Optimized the release records of external files from absolute paths to tagged paths.
  12. Fixed the issue of fake files generated during COOK in UE5.
  13. Fixed the issue with paths containing // during export release.
  14. Fixed the issue with the PatcherProxy inaccurately obtaining PackageTracker.
  15. Optimized the implementation of PackageTracker to exclude already tracked resources.
  16. Optimized PackageTracker to allow recursive Tracker analysis.
  17. Unified locks in CollectPakCommand’s ParallelFor to avoid contention issues.
  18. Supported the -AddAssetsByFile= parameter to pass a list of LongPackageNames for packaging.
  19. Fixed the issue of ShaderLib not being packaged into pak.
  20. Allowed adding Pre and Post tasks in the Cook Cmdlet.
  21. Optimized the plugin’s Slate implementation to support secondary menus in the Toolbar for quick access to the corresponding pages.
  22. Supported listening for process crashes and will store crash files in Saved/HotPatcher/Crashs.
  23. Compatible with the GameFeaturePacker and ShaderPatcher mods (both must be updated to the latest version).

Currently, v82.0 has been officially released and can be pulled and updated from hxhb/HotPatcher. The Patch page for the old version of the plugin will show a prompt for the new version.

Supplementary Information

Update by 2024-04-13 13:45

Regarding updates for WP maps: Adding some supplementary information.

For ordinary maps, if the blueprint base class is modified, it will automatically trigger changes in sub-blueprints and maps that reference it:

However, for WP, the situation is a bit more complex:

  1. WP has enabled one file per actor, and modifying blueprints may not necessarily trigger updates to the map.
  2. Logic for analysis needs to be implemented for this situation, which will be released in the next version.

Donation

If the HotPatcher project has helped you and you hope it can do even better in the future, you can support the author by scanning the donation QR codes below.

WeChat Payment Alipay

After your donation, you can click on HotPatcher Donor Information Collection Form to submit your donation information, which I will collect periodically and display on the project page.

Conclusion

After five years of development and growth, HotPatcher has become the most popular resource management and packaging solution. This update enhances compatibility with UE5 and includes numerous optimizations, and I hope it can continue to shine and create more value in the UE5 era.

The article is finished. If you have any questions, please comment and communicate.

Scan the QR code on WeChat and follow me.

Title:Open source: HotPatcher development progress
Author:LIPENGZHA
Publish Date:2024/04/07 10:01
World Count:9.6k Words
Link:https://en.imzlp.com/posts/16808/
License: CC BY-NC-SA 4.0
Reprinting of the full article is prohibited.
Your donation will encourage me to keep creating!