Z's Blog

In some previous articles, the issue of UE hot updates missing shaders and using default materials has been introduced, along with a Shader Patch solution. For details, please refer to the articles:

• UE Hot Update: Create Shader Patch
• [UE Hot Update: Questions & Answers#Hot Update Resources Not Working / Material Missing](https://imzlp.com/posts/16895/#Hot Update Resources Not Working-Material Missing).

The fundamental reason for the runtime loss of materials is that the shaders dependent on newly added or modified resources were not packaged, leading to reading failures at runtime. This article will introduce the strategy and pros and cons of shader updates, analyze the mechanisms within the engine, and provide an optimized solution that combines the advantages of Shader Patch and Inline Shader Code, which has been implemented in HotPatcher.

A series of articles has been introduced regarding the engineering practices of hot updates in UE, capable of achieving version comparison and differential updates based on resources from the original project. However, by default, resource updates depend on file updates; if a resource changes, the entire file must be repackaged. In UE, resource changes after Cook do not cause the entire file to be updated; only certain bytes are modified during serialization. In this case, a file-based patch mechanism can significantly reduce the size of the patch. This article outlines the generation and loading scheme of binary patches based on HotPatcher, allowing for easy generation of binary patches.

To fulfill this requirement, I have ported HDiffPatch to UE and used it as the default DIFF/PATCH algorithm for binary differential patches in HotPatcher; it can also easily extend other algorithms based on the Modular Feature approach.

UE4.27 has introduced a new mechanism that allows UE to be compiled into a library, enabling it to be embedded in external programs, which can then drive the engine’s execution and message communication. However, the official documentation has not yet been released. I analyzed the relevant code in the engine and proposed a practical solution for UE as Lib, which can theoretically be implemented in engine versions before 4.27, although it may be inconsistent with the official documentation released later. The related research content will also be continuously updated in this article.

Unreal Insights is a profiling tool provided by UE since 4.23, which allows for an easy view of the flame graphs of various threads during game execution. However, the official documentation only provides connection methods for PC and Android, not directly supporting iOS, which is not very convenient. After some research, it was found that the main issue with iOS is related to network policies. This article provides a solution for real-device profiling and port communication between UE and iOS devices, supporting real-time profiling and cook on the fly, as well as data exchange between iOS devices and PCs.

In the context of the large-scale project and the remote working demand caused by the pandemic, collaborative editing under development has become particularly important. In UE’s resource mechanism, maps are single resources; although they can be split using the Sub-level format, the smallest element remains a single resource. When different designers modify the same resource, it can lead to file conflicts, and merging is not as straightforward as with text. Furthermore, when each person is responsible for editing a single sub-level, there is no way to preview the entire scene’s effect in real-time, which is a bottleneck in collaborative development.

After UE4.23, UE officially launched a multi-user collaboration mechanism that allows multiple people to edit the same map simultaneously without causing conflicts. It can also synchronize other changing resources and take effect in real time. As a supplement to version control, it effectively resolves issues of synchronized collaboration.

This article records the enabling process, usage specifications, network strategies, and other issues. It will also extract an independent server-side program without relying on the complete engine, allowing for convenient deployment of the server side.

I previously wrote an article about integrating the ZSTD compression algorithm into UE: ModularFeature: Integrating ZSTD Compression Algorithm into UE4, and extracted the Oodle compression algorithm library from UE5 for use in UE4: Oodle Compression. Recently, I analyzed their compression, decompression, CPU, and memory consumption, testing on three platforms: WindowsNoEditor, Android_ASTC, and iOS, as well as the performance of Oodle’s Kraken using different compression levels on different platforms.

Finally, at the end of May 2021, UE5 is set to launch its first preview version, Unreal Engine 5 — Early Access Program starts at 10 PM on May 26, Wednesday China time!.

I have been looking forward to UE5 for a long time; last year, Nanite and Lumen created a huge stir, and after a year’s wait, I finally get to experience it. Over the past year, there haven’t been many technical updates about UE5. I have summarized and analyzed some information disclosed by Epic regarding UE5, which can be found in notes/ue5. This article will document some experiences using UE5, focusing on the changes in development methods compared to UE4, the transition from UE4 to UE5, and a technical pre-research on applying UE5 in production environments. I will keep updating this article in the near future.

In UE, there is a PSO Cache mechanism, which stands for Pipeline State Object Caching. It is used to pre-record and construct the shader information dependent on the materials used at runtime. When the project first utilizes these shaders, this list can speed up the shader loading/compilation process. The PSO Cache saves rendering states, vertex declarations, primitive types, render target pixel formats, and other data to files, enhancing shader loading efficiency. This article primarily introduces the enabling and construction process of PSO Cache, and will analyze the loading process of PSO Cache in the engine, along with implementing hot-update PSO methods, error handling, etc. The principle of PSO Cache will be analyzed in detail later.

In a previous article, I introduced the development notes and some engineering practices for Mac/iOS: UE4 Development Notes: Mac/iOS Edition. This article serves as a sister piece, documenting the standardized environment, debugging tools, engineering practices, and analysis of related engine code that I used while developing on Android with UE. It records some pitfalls encountered in the project, primarily organized from my previous notes notes/ue. Future Android-related content will also be updated in this article.

In game development, program performance is a critical issue that needs to be prioritized. To cover the largest user base possible, it’s essential to consider the operational effects on mid to low-end devices and to ensure compatibility with a wide range of hardware configurations. In this context, analyzing and optimizing the performance bottlenecks in games becomes key.

Loading more resources into memory at runtime is essentially a space-for-time approach. Frequent disk I/O is very time-consuming; by preloading resources into memory, high-speed reading can be achieved. However, memory resources are also limited and cannot be used without restrictions. This is particularly significant for mid to low-end mobile devices, where devices with 4GB or even less memory still hold a considerable market share. Therefore, memory usage must be optimized, and excessive memory consumption can lead to being terminated by the system.

Memory optimization is fundamentally about seeking a balance between loading efficiency and memory usage. The goal is to satisfy the need for compatibility with more low-end devices while also maximizing the use of available memory to improve program efficiency without triggering out-of-memory (OOM) situations.

I plan to write several articles related to performance optimization, beginning with memory analysis in UE. This article will introduce commonly used memory analysis tools and methods, as well as compile memory optimization techniques applicable within UE projects. This information has been previously recorded in the form of notes at notes/ue, and future memory-related content will be supplemented to this article.