Z's Blog

Welcome to my blog. This is a technical site focused on game development. My open-source projects on GitHub: github.com/hxhb and Open-Source projects.
The articles in the blog include a C++ series, Hot Update for UnrealEngine, Plugins and tools development, Game Resource Management series, and thoughts series, etc.

In my spare time, I research and develop technical solutions and preview the results: Showcase. If you are interested in any of the projects, feel free to email me at imzlp@foxmail.com.

UE as Lib

Dictionary-based Shader Compression

Remake PAK at Runtime

Multi-stage Resource Inspection

Splitting Game Resource

Human Pose Tracking in UE

In addition to this site, I also created a community-driven Chinese knowledge base site for Unreal Engine https://ue5wiki.com/, hosted on GitHub Pages. Contributions of technical content are welcome to help build the Chinese Unreal Engine technology community! Please see the submission rules: Wiki Content Writing Format and Rules .

Runtime Acquisition of Decryption Key

For Pak’s encryption and signing, the corresponding key must be obtained at runtime.

In the definitions of UE_REGISTER_ENCRYPTION_KEY and UE_REGISTER_SIGNING_KEY above, it can be seen that there are two extern:

1
2

extern void RegisterEncryptionKeyCallback(void (*)(unsigned char OutKey[32])); 
extern void RegisterSigningKeyCallback(void (*)(TArray<uint8>&, TArray<uint8>&));

These two functions are the engine’s methods for obtaining keys:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

typedef void(*TSigningKeyFunc)(TArray<uint8>&, TArray<uint8>&);
typedef void(*TEncryptionKeyFunc)(unsigned char[32]);

void RegisterSigningKeyCallback(TSigningKeyFunc InCallback)
{
	FCoreDelegates::GetPakSigningKeysDelegate().BindLambda([InCallback](TArray<uint8>& OutExponent, TArray<uint8>& OutModulus)
	{
		InCallback(OutExponent, OutModulus);
	});
}

void RegisterEncryptionKeyCallback(TEncryptionKeyFunc InCallback)
{
	FCoreDelegates::GetPakEncryptionKeyDelegate().BindLambda([InCallback](uint8 OutKey[32])
	{
		InCallback(OutKey);
	});
}

These two functions are called (i.e., the constructors of FEncryptionKeyRegistration and FSigningKeyRegistration).

Recall the initialization process of the Monolithic module mentioned in a previous article: UE Plugin and Tool Development: Basic Concepts #Monolithic Mode.
They are created when the module starts.

When loading PAK, FCoreDelegates::GetPakEncryptionKeyDelegate() is invoked to obtain the key:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

void FPakPlatformFile::GetPakEncryptionKey(FAES::FAESKey& OutKey, const FGuid& InEncryptionKeyGuid)
{
	OutKey.Reset();

	if (!GetRegisteredEncryptionKeys().GetKey(InEncryptionKeyGuid, OutKey))
	{
		if (!InEncryptionKeyGuid.IsValid() && FCoreDelegates::GetPakEncryptionKeyDelegate().IsBound())
		{
			FCoreDelegates::GetPakEncryptionKeyDelegate().Execute(OutKey.Key);
		}
		else
		{
			UE_LOG(LogPakFile, Fatal, TEXT("Failed to find requested encryption key %s"), *InEncryptionKeyGuid.ToString());
		}
	}
}

Then it is used to decrypt data:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

void DecryptData(uint8* InData, uint32 InDataSize, FGuid InEncryptionKeyGuid)
{
   if (FPakPlatformFile::GetPakCustomEncryptionDelegate().IsBound())
   {
      FPakPlatformFile::GetPakCustomEncryptionDelegate().Execute(InData, InDataSize, InEncryptionKeyGuid);
   }
   else
   {
      SCOPE_SECONDS_ACCUMULATOR(STAT_PakCache_DecryptTime);
      FAES::FAESKey Key;
      FPakPlatformFile::GetPakEncryptionKey(Key, InEncryptionKeyGuid);
      check(Key.IsValid());
      FAES::DecryptData(InData, InDataSize, Key);
   }
}

Thus achieving the entire process from build-time encryption to runtime decryption. ## Serialization Structure of Pak

It was mentioned in the previous article that UE’s PAK is a typical Archive structure:

Generated through certain serialization rules, and it is also read at runtime according to format rules:

1. When mounting PAK, first read PakInfo from the end of the file to obtain the offset for the Index
2. Get EncodePakEntrys based on the Index offset, which can acquire PakEntry information for each file
3. Each PakEntry records the information of the current file, guiding how to read and decompress
   It is also an important part of UE’s virtual file system.

However, since UE’s source code is open, the serialization structure of PAK itself lacks confidentiality. If the encryption key is obtained through reverse engineering, the files within the PAK can be directly decrypted using the official UnrealPak engine’s logic.

Additionally, there are some third-party tools, such as UModel or UnrealPakViewer, that can achieve this as well. Therefore, in addition to the keys and decryption obfuscation, the serialization structure of the PAK itself also requires modification.

Reinforcement Plan

Drawbacks of the Default Plan

Based on the previous two sections, the following conclusions can be drawn:

1. The decryption key is directly compiled into the code as a macro definition.
2. By using the engine source code + static analysis, it is relatively easy to locate the function for obtaining the key.
3. The exported symbols in the code can directly expose the location of the decryption function.
4. The PAK strings in the code (logs/ProfilingTag) can assist in locating the critical decryption function.
5. The default use of AES encryption has obvious code characteristics, making it easy to analyze.
6. The serialization format of PAK is quite common, allowing retrieval of keys to decrypt using public engines like UnrealPak or tools like UModel/UnrealPakViewer.

Consequently, based on these situations, we need to make various modifications to the encryption scheme to increase the cost of decryption as much as possible. However, due to security demands, I will only introduce some encryption concepts and will not provide specific code.

Key Reinforcement

The first section mentioned that keys are generated through Base64 encoding after a macro definition and obtained during runtime. We can write some unique key obfuscation logic to add a layer of processing to the key itself, preventing it from being directly obtained through static analysis.

Symbol Removal

By default, if symbols are not removed, when opening the executable program with tools like IDA, the function names can be viewed directly:

This way, the decryption function can be easily identified, and the key can be captured. Therefore, all packages for the release platform need to have their symbols removed.
After symbol removal:

You can refer to my previous article: Ultimate Optimization of UE Android APK Size

Removal of PAK Strings

For the same reason, the engine prints out various logs related to mounting PAK. Analyzing the access positions of these log strings can also assist in locating the decryption logic:

Once the decryption function is found, attaching a debugger allows you to retrieve the actual key from memory.

Thus, removing PAK-related strings is very necessary.

Log Removal

For the removal method, we hope to retain logs in Dev while removing them in Shipping:

1
2
3
4
5
6
7
8
9
10

//++[lipengzha] Remove PAK-related strings in shipping
#ifndef DISABLE_PAK_LOG_IN_SHIPPING
	#define DISABLE_PAK_LOG_IN_SHIPPING 0
#endif
#if UE_BUILD_SHIPPING && DISABLE_PAK_LOG_IN_SHIPPING
	#define UE_PAK_LOG(...)
#else
	#define UE_PAK_LOG(LogCategory,LogLevel,Format, ...) UE_LOG(LogCategory,LogLevel,Format,##__VA_ARGS__)
#endif
//--

Thus, we can encapsulate the UE_LOG macro, forwarding it by default in Dev and replacing it with an empty macro in Shipping.

Removal of BOOT_TIMING

We also need to remove strings introduced by SCOPED_BOOT_TIMING:

1
2
3
4
5
6
7
8
9
10
11
12

//++[lipengzha] Remove PAK-related strings in shipping
#ifndef REMOVE_PAK_TEXT_IN_SHIPPING
	#define REMOVE_PAK_TEXT_IN_SHIPPING 0
#endif
#if UE_BUILD_SHIPPING && REMOVE_PAK_TEXT_IN_SHIPPING
	#define UE_PAK_LOG(...)
	#define UE_PAK_SCOPED_BOOT_TIMING(...)
#else
	#define UE_PAK_LOG(LogCategory,LogLevel,Format, ...) UE_LOG(LogCategory,LogLevel,Format,##__VA_ARGS__)
	#define UE_PAK_SCOPED_BOOT_TIMING(x) SCOPED_BOOT_TIMING(x)
#endif
//--

Then replace all UE_LOG occurrences in IPlatformFilePak.cpp and IPlatformFilePak.h with UE_PAK_LOG, and all SCOPED_BOOT_TIMING with UE_PAK_SCOPED_BOOT_TIMING.

This way, the related strings will be automatically removed during Shipping.

PAK Structure Obfuscation

In the previous section (Serialization Structure of Pak), I introduced the serialization method of the PAK structure. Therefore, for commercial projects, obfuscating the structure itself is also necessary.

On the basis of maintaining the original UFS file system, modifications can be made to the serialization of the PAK, altering the organization of the structure and encrypting or modifying the data to avoid public engines and open-source tools from being able to decrypt once they obtain the key. This can also ensure the uniqueness of each pak structure and decryption to prevent cases where one is cracked, leading to a cascade of breaches.

This part will not be elaborated on further, as each project must implement it individually.

Decompression Algorithm Obfuscation

As files are typically compressed using a compression algorithm after being packaged into a PAK, they need to be decompressed at runtime. Additionally, which compression algorithm was used is serialized in the PakInfo, with records in each PakEntry’s Index.

Therefore, some processing can also be applied in this area so that even if the key and PAK structure are reverse-engineered, an extra obfuscation behavior can occur during the actual file decompression:

Replacement of AES Algorithm

Since AES is standard in UE, it can also be located via signature codes during reverse engineering. Thus, AES can be replaced, or logical modifications can be made to the code.
What algorithm to specifically replace it with needs to be evaluated based on the project situation.

Dynamic Key Distribution for Key Assets

According to the previous analysis, if all assets use a single default key, it is directly compiled into the code. If this key is cracked, all resources may face the risk of exposure. Furthermore, local decryption has low security.

In such cases, for certain key assets or files (such as not yet released commercial content), the asset may be pre-embedded in the client but not yet available. Using a common key in this scenario poses significant risks.

The engine itself also provides logic for dynamically registering decryption keys:

1
2
3
4
5
6
7

class CORE_API FCoreDelegates  
{  
public:
	// Callback for registering a new encryption key  
	DECLARE_DELEGATE_TwoParams(FRegisterEncryptionKeyDelegate, const FGuid&, const FAES::FAESKey&);
	// ...
};

Thus, a dynamic key mechanism can be implemented within the update process, ensuring that the encrypted PAK does not contain the decryption key in the program body. Only after a specific time and version is released can the key be dynamically delivered, allowing the client to use it. This ensures that early unpacking and leakage does not occur.

Summary

This article introduced the default encryption logic of UE, analyzed potential risks, and provided some feasible reinforcement strategies for reference.

Note: The article analyzed the engine’s encryption process and reverse engineering methods solely to promote safer practices and is not a hacking tutorial. Any game reverse engineering-related activities are unrelated to the author. Thank you!

When developing UE plugins, we often provide a large number of configurable parameters for flexible control, used to manage the specific execution logic and behavior of the plugins.

This article is the eighth in my UE Plugin Development Series, and will introduce my thoughts and implementations regarding the configurability of plugins during the development process, along with practices in project configuration, task configuration, dynamic parameter replacement, etc., to make the plugin configuration process as flexible and easy to use as possible.

In game projects, when we package for various platforms, we always hope that the package for each platform can be minimized for easier distribution, and there are specific size requirements for some platforms.

For UE, it contains massive code and numerous plugins, and during the Build phase, it generates a lot of reflection glue code, resulting in a significant increase in code segments during compilation. Taking the Android platform as an example, this leads to a sharp increase in the size of libUE4.so, putting pressure on both the package size and runtime memory.

Moreover, some necessary and additional resources brought in by the engine can take up hundreds of MB, making the size of an empty APK easily reach several hundred MB! Not only to meet the platform’s requirements, but it is also necessary to trim down the size of the UE package from the perspective of package size and memory optimization.

This article will take Android as an example and introduce optimization ideas and practices for the cuttable parts in the UE package from various aspects, optimizing both the APK size and the runtime memory usage of native libraries. The strategies can also be reused on other platforms.

Recently, I encountered an extremely bizarre bug involving two maps. One map, A, can be accessed with its PAK placed in the engine’s automatic mount directory, but it cannot be accessed from the hot update directory. The other map, B, behaves completely oppositely: it is abnormal in the automatic mount directory but works normally in the hot update directory.

At first glance, the issue appears entirely elusive, with two mutually exclusive behaviors occurring within the same logical framework. Moreover, the hot update mount and the automatic mount only differ in timing and priority, so this problem shouldn’t theoretically exist.

While the issue can ultimately be resolved on the business logic side, this behavior involves another very obscure path within the engine. Understanding why and how it works is crucial. Therefore, I analyzed the engine’s code based on this behavior, came to a reasonable conclusion, and devised a method to detect and mitigate this issue.

This article assumes that readers have some basic knowledge of UE hot updates; if in doubt, please refer to other articles in this blog’s hot update series for more information.

For personal knowledge management, I am very keen on creating my own local-first and open source self-hosted services. This way, I can avoid relying on any platform, have complete autonomy over my data, and be able to migrate services at any time. Options like Obsidian, Hexo blog, and the memos introduced in this article all meet these criteria.

The protagonist of this article, memos, is an open-source lightweight note-taking service that allows you to take notes in a way similar to posting on Weibo, supporting TAG marking and citations. It features an account system and permission management, allowing web access at any time, and notes can be public or private, offering great flexibility. I chose it because it complements Obsidian; Obsidian is still too heavy for my needs—it works well on PC but has a poor mobile experience. Thus, I need a lightweight, always-available note-taking service.

This article will introduce how to deploy a Memos service on a VPS using Docker, along with Nginx to bind the domain name, Certbot to issue and automatically update SSL certificates, and regular backups of the memos database, as well as some optimization configurations I made for the service.

In previous articles, I primarily introduced how HotPatcher can be applied in the hot update process. However, HotPatcher is not just capable of fulfilling the needs for hot updates; it is also a flexible resource packaging framework.

As the project development reaches a certain stage, a massive amount of resources needs to be packaged, and waiting for a complete package to be built can take a long time. If a real device encounters issues, the serial process of repair -> package -> verify can become exceedingly lengthy. There are often cases of script, data, or other resource errors that lead to package abnormalities.

In such cases, waiting hours to generate a complete package prevents smoke testing from proceeding. Even if you rebuild the package, there’s no guarantee that there won’t be other overlooked issues; otherwise, you have to start over again.

Based on this pain point, we urgently need a method to quickly fix resource issues within packages and validate them during the development phase. HotPatcher provides this functionality, allowing you to utilize it fully without waiting for a complete package.

This article will introduce how to utilize the editor features provided by HotPatcher to quickly package preset configurations and specific resources, accelerating the validation process of resources on real devices and significantly improving testing efficiency.

Since the last version release of HotPatcher over half a year ago, its support for UE4 has been commendable, but its support for UE5 has been lacking. There have been frequent inquiries in the group about adaptation.

The last Release version of HotPatcher was v81.0, released on 20230604, which only supports UE5.1.0 and does not support WP and Nanite. Subsequently, Epic released UE5.2 and 5.3, but the plugin has not been adapted. Although one can modify compilation errors themselves, functionality support in UE5 is somewhat in a beta state.

Taking advantage of the Qingming holiday, I worked intensively for a few days to comprehensively optimize and adapt HotPatcher to be compatible with the latest versions of UE4.21 to UE5. A new Release version has been published: v82.0. The latter part of this article will discuss the adaptation of HotPatcher in UE5, precautions, and the update log.

Additionally, in 2020, I wrote an article: Some Thoughts and Ideas on Open Source, and over the past two years, I’ve had some new thoughts on maintaining open source projects. Let’s summarize together.

During the New Year’s holiday, my iPhone 12 dropped and caused some malfunctions, making it barely usable. However, recently the battery also swelled and it easily shut down automatically, rendering it completely unusable. After switching to a backup phone, it was set aside for a while.

I recently thought of it again, as it could still power on normally and most of its main functions could still operate, so I speculated that it was only partially faulty and not completely damaged. It would be a pity to completely scrap it, and it seems I can only swap it for a stainless steel basin.

I took it apart to analyze the cause of the faults and attempted to buy parts for repair. This article documents the disassembly and resolution process.

This year, I plan to celebrate the Spring Festival in Shenzhen and the next stop for my Greater Bay Area trip is Macau. Traveling from Shenzhen to Macau is relatively convenient, so I am making arrangements for the trip. I’ve done some research and itinerary planning that can serve as a reference for those visiting Macau for the first time.