UE热更新：需求分析与方案设计

Game hot updates are a way for players to obtain the latest game content without reinstalling the game. They are widely used in network games on both PC and mobile platforms, as quick adjustments, bug fixes, and content updates are often needed after a game goes live. If even the smallest change requires players to update the app through the App Store, or manually download and install it from a website, and given that different platforms have inconsistent review rules and feedback times, operations could become chaotic.

While there may be relatively mature solutions for hot updates in other engines, comprehensive discussions on implementing hot updates in UE4 are not commonly found. Fortunately, I have previously analyzed and implemented UE4’s hot update mechanism and plan to write a couple of articles to document my thoughts and implementation plans, along with a runnable demo, which I hope will be of help to those who need it.

To conveniently collect and manage common questions and solutions regarding hot updates and HotPatcher, I’ve created a new article to document and organize them: UE4 Hot Update: Questions & Answers. If you encounter issues, please check this FAQ page first.

This article will cover two sections: Requirement Analysis and Solution Design, focusing on my thoughts and summaries during the research on hot update solutions. The specific implementation of hot updates will be detailed in the next article.

Core Requirement Analysis

As it involves hot updates, we essentially need to delay the game’s content updates until runtime. From the simplest yet most crucial process perspective, the execution structure looks like this:

Based on the flowchart above, we can break down the specific tasks needed for hot updates into the following questions that need to be addressed:

1. What content in UE can be hot updated?
2. How to package the content that can be hot updated and manage those updates?
3. How to use the resource packages downloaded for hot updates in UE?
4. How to compare the latest versions of local and server content?
5. Download and verify hot update files.

The most critical requirement in hot updates is to address the issues of how to package resources and conduct version management. The download process may vary across different applications, so I’ll only cover the basic capability for implementing a hot update download and update process.

Let’s analyze these questions one by one.

What content in UE can be hot updated?

From a programmatic perspective, UE provides C++ and Blueprints as the development languages and scripts offered by the engine. Since C++ is a compiled language, all changes require compilation, which means C++ code cannot be hot updated. However, Blueprints, which are essentially assets (uasset), do not require reinstallation for updates, so game logic written in Blueprints can be hot updated.

However, since Blueprints are resources, any modification to Blueprint logic requires execution of Cook to package, making updates inconvenient, and collaborating on Blueprint projects is challenging to manage, necessitating the integration of a text-based scripting language.

In domestic game development, integrating Lua as the business script is common, and Tencent has open-sourced two UE plugins that integrate Lua, namely sluaunreal and UnLua. These can be used in UE for scripting business logic instead of using Blueprints.

I chose UnLua for my project and integrated several common Lua libraries, editor extensions, common library exports, and fixed some bugs based on UnLua’s official version. It’s open-sourced on GitHub: debugable-unlua, and I will periodically merge the official version.

In a previous article, I wrote about the use of UnLua: UE4 Hot Update: Lua Programming Guide Based on UnLua.

Additionally, all uasset resources in the project (maps, blueprints, models, animations, textures, UMG, audio, fonts, etc.) are also updatable. UE’s uasset requires cooking before packaging. The meaning of Cook is to convert UE’s platform-independent internal format into a specific platform format since various platforms utilize their proprietary formats or storage formats that perform better on their platforms.

For example, when packaging a UE project for Windows, it generates files with a path relationship similar to the example below:

D:\Examples\PackageExample\Package>tree /f
Volume in Drive C has no label.
 Volume Serial Number is BAB5-5234
C:.
└─WindowsNoEditor
    │  Manifest_NonUFSFiles_Win64.txt
    │  PackageExample.exe
    │
    ├─Engine
    │  ├─ ...
    │
    └─PackageExample
        └─Content
            └─Paks
                PackageExample-WindowsNoEditor.pak

The *.pak files under Content/Paks represent all non-code resources packaged by UE; the game reads resources from these paks at startup, which contain not only uasset content but can also be understood as: pak files are the resource packs necessary for the game’s operation. The content supported in paks is what UE hot updates can support.

By default (if no files are set to be ignored), UE4 will put all project resources into a single Pak file during packaging, including the following contents:

In the following descriptions, there are a few key terms: PROJECT_NAME is the project name, and PLATFORM_NAME is the name of the platform on which it was packaged.

• During packaging, all resources participating in Cook from the project are included in the Pak; I will supplement future articles regarding which resources the engine cooks during packaging.

• Engine Slate resource files in Engine\Content\Slate\, such as fonts/images, etc.

• Relevant language files in the engine’s Content\Internationalization.

• The locmeta/locres files in Content\Localization under the engine and enabled plugin directories.

• The uproject file of the project, mounted at ../../../PROJECT_NAME/PROJECT_NAME.uproject.

• The uplugin files for all plugins enabled in the project, mounted at paths relative to ../../../Engine/ or ../../../PROJECT_NAME/Plugins/.

• The Intermediate\Staging\PROJECT_NAME.upluginmanifest file in the project directory, mounted at ../../../PROJECT_NAME/Plugins/PROJECT_NAME.upluginmanifest.

• The engine’s ini files, including all except Editor ini and BaseLightmass.ini/BasePakFileRules.ini in the engine’s Engine/Config.

• Platform-specific ini files under Engine/Config/PLATFORM_NAME.

• The ini files of the enabled plugins, located in the config directory of the plugin.

• The AssetRegistry.bin produced during Cook.

• The PLATFORM_NAME\Engine\GlobalShaderCache*.bin files produced during Cook.

• The PLATFORM_NAME\PROJECT_NAME\Content\ShaderArchive-*.ushaderbytecode files produced during Cook.

• Non-uasset files added via Project Setting-Packaging-Add Non-Asset Directory*, etc.

All the files mentioned above can be hot updated in UE.

Relevant configurations can be seen in Engine\Config\BaseGame.ini:

+EarlyDownloaderPakFileFiles=...\Content\Internationalization\...\*.icu
+EarlyDownloaderPakFileFiles=...\Content\Internationalization\...\*.brk
+EarlyDownloaderPakFileFiles=...\Content\Internationalization\...\*.res
+EarlyDownloaderPakFileFiles=...\Content\Internationalization\...\*.nrm
+EarlyDownloaderPakFileFiles=...\Content\Internationalization\...\*.cfu
+EarlyDownloaderPakFileFiles=...\Content\Localization\...\*.*
+EarlyDownloaderPakFileFiles=...\Content\Localization\*.*
+EarlyDownloaderPakFileFiles=...\Content\Certificates\...\*.*
+EarlyDownloaderPakFileFiles=...\Content\Certificates\*.*
; have special cased game localization so that it's not required for early pak file
+EarlyDownloaderPakFileFiles=-...\Content\Localization\Game\...\*.*
+EarlyDownloaderPakFileFiles=-...\Content\Localization\Game\*.*
+EarlyDownloaderPakFileFiles=...\Config\...\*.ini
+EarlyDownloaderPakFileFiles=...\Config\*.ini
+EarlyDownloaderPakFileFiles=...\Engine\GlobalShaderCache*.bin
+EarlyDownloaderPakFileFiles=...\Content\ShaderArchive-Global*.ushaderbytecode
+EarlyDownloaderPakFileFiles=...\Content\Slate\*.*
+EarlyDownloaderPakFileFiles=...\Content\Slate\...\*.*
+EarlyDownloaderPakFileFiles=...\*.upluginmanifest
+EarlyDownloaderPakFileFiles=...\*.uproject
+EarlyDownloaderPakFileFiles=...\global_sf*.metalmap

Besides the built-in resource types, to achieve program hot updates, the key point is that non-resource files can also be packaged into paks. This addresses the issue mentioned earlier regarding the updates needed for using UnLua as the business script.

How to package hot update content and manage versions?

From the previous section, we learned what content can be hot updated in UE. Another crucial aspect of hot updates is: how to package the desired content to produce a downloadable hot update file?

All platforms in UE include pak files during packaging (some are embedded, while others are separate). Therefore, our requirement is to extract the desired resources into a pak. UE uses the UnrealPak tool to package files into a pak. UnrealPak also provides commands to view the files within a pak and to extract files from it. This part can be referenced in a previous article of mine: UE4 Toolchain Configuration and Development Tips#UnrealPak Parameters, where you can find additional articles on the topic.

During packaging, UE calls UnrealPak to generate pak files, collecting resource information during packaging into a pak-commandlist.txt file, which records the resources to be included in the pak and their mount points.

The basic command is as follows:

UnrealPak.exe D:\TEST.pak -create="XXXXXXX.txt"

For simple pak generation, we can directly call UnrealPak. However, another significant point in the hot update process is: how to control which resources are packaged into the specified pak?

UE provides a feature similar to Chunk, but it’s cumbersome, requiring each resource to have a designated chunk ID, which only gets generated during packaging, making it inconvenient to control which resources are included in each pak.

Moreover, by default, UE can only add non-resource files from the Content path during packaging, causing many files to be difficult to update conveniently. Although the Project Launcher also offers patching operations, the resources packed are a black box without the ability to know precisely which files are included. It’s also not possible to generate a patch based on another patch, rendering the official packaging method ineffective for our hot update needs.

Therefore, I developed a plugin to address these issues, which is open-sourced on GitHub: hxhb/HotPatcher, with introduction documentation available at: UE4 Resource Hot Update Packaging Tool HotPatcher.

Its function is to easily specify which resources and files to package into which pak within the UE editor. It also offers a one-click Cook for multiple platforms, allowing for multiple platform pak generation, version information export, and iterative patching.

The core concept is: while generating the base package, you can use HotPatcher to export resource information, and when changes occur in the project, import the resource information from the base package and compare it with the current project’s resource information to find the differences, packaging those into the pak.

This article does not delve into the detailed usage of HotPatcher; specific usage documentation and parameter descriptions can be found in the provided document link.

In summary, using hxhb/HotPatcher facilitates UE hot update resource packaging and version management!

How to use the resource packages downloaded for hot updates in UE?

Here, resource packages refer to the pak files generated in the previous section.

By default, UE provides three automatic mounting paths for paks:

# relative to Project Path
Content/Paks/
Saved/Paks/
# relative to Engine Path
Content/Paks

In my previous notes, I recorded the process of loading pak files when the engine starts: UE4: Loading Pak Files When Engine Starts.

Placing the generated pak directly into one of these three directories will typically result in the automatic loading of all paks from these paths without enabling Signing. From a testing perspective, simply placing the packaged pak file in one of these folders and starting the game will automatically mount the pak.

This leads to another question: How can UE know which pak file is the latest?

Since hot updates are meant to replace outdated resources with newly updated ones, it is essential to identify which pak contains all the latest resources.

Taking the three paths for automatic pak mounting as an example, the engine prioritizes them based on folder structure:

// Runtime\PakFile\Private\IPlatformFilePak.cpp
int32 FPakPlatformFile::GetPakOrderFromPakFilePath(const FString& PakFilePath)
{
  if (PakFilePath.StartsWith(FString::Printf(TEXT("%sPaks/%s-"), *FPaths::ProjectContentDir(), FApp::GetProjectName())))
  {
    return 4;
  }
  else if (PakFilePath.StartsWith(FPaths::ProjectContentDir()))
  {
    return 3;
  }
  else if (PakFilePath.StartsWith(FPaths::EngineContentDir()))
  {
    return 2;
  }
  else if (PakFilePath.StartsWith(FPaths::ProjectSavedDir()))
  {
    return 1;
  }

  return 0;
}

As observed, pak files located in the Content/Paks directory and prefixed with the project name have the highest default priority, followed by Project/Content/Paks > Engine/Content/Pak > Saved/Paks.

Furthermore, the naming of pak files also affects their priority.

Files ending with _Num_P.pak, where Num is a number, confer higher priority to patch packages compared to regular paks. In IPlatformFilePak.cpp, _P.pak files have their PakOrder increased by 100 by default, and the larger the number before _P.pak, the higher the loading priority.

This priority array is used during mount pak, where each pak file must be specified, aiding the engine in accurately locating the newest version of files during resource searches and loading.

However, if malicious users discover these rules and deliberately disrupt the naming of pak files, it could lead to program errors, so it’s not advisable to solely rely on naming conventions to establish priority; it’s best to perform a validation during hot updates.

Additionally, for hot update needs, you need to control the timing of mounting the pak yourself, and for this, you need to invoke the MountPak function myself. I’ve written a wrapper function:

bool MountPak(const FString& PakPath, int32 PakOrder, const FString& InMountPoint)
{
	bool bMounted = false;
#if !WITH_EDITOR
	FPakPlatformFile* PakFileMgr=(FPakPlatformFile*)FPlatformFileManager::Get().GetPlatformFile(FPakPlatformFile::GetTypeName());
	if (!PakFileMgr)
	{
		UE_LOG(LogTemp, Log, TEXT("GetPlatformFile(TEXT(\"PakFile\") is NULL"));
		return false;
	}
	
	PakOrder = FMath::Max(0, PakOrder);
	
	if (FPaths::FileExists(PakPath) && FPaths::GetExtension(PakPath) == TEXT("pak"))
	{
		const TCHAR* MountPount = InMountPoint.GetCharArray().GetData();
		if (PakFileMgr->Mount(*PakPath, PakOrder,MountPount))
		{
			UE_LOG(LogTemp, Log, TEXT("Mounted = %s, Order = %d, MountPoint = %s"), *PakPath, PakOrder, !MountPount ? TEXT("(NULL)") : MountPount);
			bMounted = true;
		}
		else {
			UE_LOG(LogTemp, Error, TEXT("Faild to mount pak = %s"), *PakPath);
			bMounted = false;
		}
	}

#endif
	return bMounted;
}

When pak files are downloaded from the server, this function can be called to mount them into the engine. As long as you ensure that the PakOrder is correct, you do not need to worry about the pak; the engine will automatically locate the latest version of the resources when loading files.

Note: Typically, the hot update process is executed in a new map; no resource loading actions should happen before the hot update, as resources loaded before the hot update will not be refreshed even if the new pak is mounted afterward.

How to compare local versions with the latest versions on the server?

Different applications can implement various handling processes. The key idea is that at game startup, the hot update module first requests the latest hot update version info from the server. The client then scans the local downloaded patches based on the current package’s version info, comparing the two datasets to determine which versions need downloading.

A simple approach is to write a JSON file that records all patch information (filename, MD5 values), which the client downloads each time it starts and parses. The client locally scans the pak files in a designated folder, compares their filenames and MD5 values to produce a legitimate local pak list, and then contrasts that with the server’s version list to identify differences for downloading.

How to download and verify hot update files?

In the previous section, we discussed how to request the server and analyze the current user’s necessary patch versions. This section will cover how to download and verify them in UE.

UE provides a cross-platform HTTP library, which can be used through the HTTP module under Online:

HttpRequest = FHttpModule::Get().CreateRequest();
HttpRequest->OnRequestProgress().BindUObject(this, &UDownloadProxy::OnDownloadProcess);
HttpRequest->OnProcessRequestComplete().BindUObject(this, &UDownloadProxy::OnDownloadComplete);
HttpRequest->SetURL(InternalDownloadFileInfo.URL);
HttpRequest->SetVerb(TEXT("GET"));
HttpRequest->ProcessRequest();

However, I must reiterate that simply using this default HTTP download method would require writing to the file only after the download is complete. If the file is large, the write process is time-consuming and could block other operations. Moreover, since the files need verification, I opted for MD5. MD5 is a hash algorithm that requires the file to be completely read from start to finish to compute the result. If you attempt to separate downloading, storing, and verifying into three steps, it would waste a lot of time.

Therefore, I wrapped up a download library that allows for downloading and storing simultaneously/downloading and calculating MD5 simultaneously, which means that the file is downloaded and stored locally as well as its MD5 value is calculated for verification all in one process. It also supports pause/resume and segmented downloads, which can be altered for breakpoint continuation.

This plugin is also open-sourced on GitHub: ue4-dtkit, and supports iOS/Android/Windows/Mac across four platforms.

Within this plugin, I wrapped a MD5Wrapper.hpp that can be used for MD5 calculations elsewhere, utilizing the OpenSSL library.

After downloading and obtaining the MD5 of the file, compare it with the MD5 from the version information on the server; if they match, you can proceed to MountPak.

Conclusion

This article primarily introduces the concepts of hot updates in UE and the tools that can be utilized, with a focus on resource packaging in UE4. Using the tools I created, such as hxhb/HotPatcher, significantly simplifies this process. As for the downloading and verification flow, this is just for reference, and you can implement your method according to your process.

The next article on hot updates will specifically implement an example based on the concepts and tools presented here. The project and code will also be open-sourced when time allows. However, based on the ideas and tools provided in this article, it should not be too difficult for you to implement your solution.

Resources and documentation mentioned in this article:

• hxhb/HotPatcher
• UE4 Resource Hot Update Packaging Tool HotPatcher
• ue4-dtkit
• debugable-unlua
• UE4 Hot Update: Lua Programming Guide Based on UnLua
• UE4: Loading Pak Files When Engine Starts
• UE4 Toolchain Configuration and Development Tips#UnrealPak Parameters

Hot Update Series Articles

• UE4 Hot Update: Requirements Analysis and Solution Design
• UE4 Resource Hot Update Packaging Tool HotPatcher
• UE4 Hot Update: Automation Process Based on HotPatcher
• UE4 Hot Update: Splitting Base Package
• UE4 Hot Update: Asset Management and Audit Tools
• UE4 Hot Update: Create Shader Patch
• UE4 Hot Update: Questions & Answers