UE Hot Update: binary patch solution for resources

A series of articles has been introduced regarding the engineering practices of hot updates in UE, capable of achieving version comparison and differential updates based on resources from the original project. However, by default, resource updates depend on file updates; if a resource changes, the entire file must be repackaged. In UE, resource changes after Cook do not cause the entire file to be updated; only certain bytes are modified during serialization. In this case, a file-based patch mechanism can significantly reduce the size of the patch. This article outlines the generation and loading scheme of binary patches based on HotPatcher, allowing for easy generation of binary patches.

To fulfill this requirement, I have ported HDiffPatch to UE and used it as the default DIFF/PATCH algorithm for binary differential patches in HotPatcher; it can also easily extend other algorithms based on the Modular Feature approach.

Overview

The implementation of binary differential patches based on resources has already been widely applied in some games, such as League of Legends, which has implemented a binary patching scheme in its update mechanism. Some implementation schemes are based on the differences of complete packages, which is unsuitable for UE since the resources packaged by UE are in Pak format. The arrangement order of resources during Pak packaging is not fixed, and cannot guarantee the same arrangement, making direct comparisons of Pak unreasonable, yielding poor performance. Therefore, my scheme is to perform DIFF on the Cooked files of the resources before they are packaged into Pak, then pack the differential results into Pak.

In UE, the modified files that ultimately update to the player’s device as uasset are all files after they are Cooked. The following image serves as an example:

The comparison of the same resource before and after modification after cooking shows that the Cooked file only changed some data, while most of the data remained consistent. In this situation, updating the entire file seems wasteful, and the benefits of binary patches are significant.

Thus, the objectives are:

1. Generate binary patches based on the Cooked results of UAsset during packaging.
2. Replace the original Cooked resource file with the patch file.
3. Apply the patch when loading resources.

To meet these three needs, it is necessary to intervene in the packaging process and modify the engine code.

Binary Patches of Resources

The uasset packaged by UE is a Cooked file, which can be read after the game is packaged. To implement binary DIFF/PATCH for uasset, it is necessary to obtain the Cooked files in the base package and the latest Cooked files for comparison.

Thanks to the mechanism of HotPatcher, it is very convenient to acquire the current patch’s uasset information, unpack the files in the Patch list from the base package’s Pak, and perform DIFF with the latest Cooked files.

Therefore, the steps and processes of performing DIFF on the entire resource are:

1. Retrieve the list of resources in the current patch.
2. Cook the resources in the patch.
3. Unpack the resources from the base package’s Pak currently listed in the patch.
4. Use HDiffPatch to create a new binary patch between the new and old resources.
5. Remove the original Cooked file from the Patch information and use patch as a substitute.

This mechanism has been provided in the HotPatcher plugin, and can be enabled in the Binaries Patch section of the Patch configuration page.

You can specify the Pak of the base package for a certain platform and the decryption key, as well as perform some filter operations, supporting file rule matching (size, type), etc. The size of the patches is still quite considerable:

Binary DIFF/PATCH Algorithm

I have ported HDiffPatch to UE in the form of a plugin that can be downloaded from GitHub: hxhb/HDiffPatchUE. Together with HotPatcher, it can be used in the same project for patching.

For binary DIFF/PATCH algorithms, I have currently chosen HDiffPatch, which has performance advantages over BsDiff:

I have ported the HDiffPatch code into a UE module and made some corrections for cross-platform issues. Once HotPatcher is launched, you can choose it in Binaries Patch - Binaries Patch Type.

I have encapsulated its code into two functions for convenient DIFF/PATCH operations.

UFUNCTION(BlueprintCallable)
static bool CreateCompressedDiff(const TArray<uint8>& NewData, const TArray<uint8>& OldData, TArray<uint8>& OutPatch);
UFUNCTION(BlueprintCallable)
static bool PatchCompressedDiff(const TArray<uint8>& OldData, const TArray<uint8>& PatchData, TArray<uint8>& OutNewData);

Of course, as mentioned before, if you do not want to use HDiffPatch as the algorithm for creating binaries, you can implement other algorithms by injecting them. Since it is based on a Modular Feature implementation, it can be integrated non-intrusively, just by adding a module that implements the following interface and registering it into the BINARIES_DIFF_PATCH_FEATURE_NAME Modular Features in StartupModule.

struct IBinariesDiffPatchFeature: public IModularFeature
{
	virtual ~IBinariesDiffPatchFeature(){};
	virtual bool CreateDiff(const TArray<uint8>& NewData, const TArray<uint8>& OldData, TArray<uint8>& OutPatch) = 0;
	virtual bool PatchDiff(const TArray<uint8>& OldData, const TArray<uint8>& PatchData, TArray<uint8>& OutNewData) = 0;
	virtual FString GetFeatureName()const = 0;
};

Runtime PATCH

Note: The implementation of the PakFile module has not supported PakCache, so it needs to be disabled.

After completing the previous steps, the binary patches have been generated and packaged into Pak files.

For resources created with binary patches, they also need to be accessed via patches during runtime. This section provides an implementation idea, but it is not a complete solution. This part involves too much content and requires engine modifications. I provide a rough practical solution aimed at verifying the feasibility of binary patches. A lot of optimization is still needed according to project requirements.

When UE loads files from Pak, it sorts them according to PakOrder, which is also the foundation for UE’s hot update implementation. The reading of files from Pak is implemented in IPlatformPakFile.cpp, which is located in the UE’s PakFile module.

The original Pak loading process:

1. Read File A.
2. Read the original Cooked file of A from Pak.
3. Return the file handle.

However, after creating a patch, the Pak does not contain the original Cooked file, so it is necessary to execute the patch operation on the basis of the old resource to restore the latest file:

1. Read File A.
2. Read the old A file + A’s patch file from Pak.
3. Perform runtime patching to restore the source file.
4. Return the source file handle.

The patching operation is usually part of the Pre Patch process; while it can be done in memory during runtime, it incurs significant performance costs. Therefore, prior to entering the game, during the hot update process, all downloaded resources can be patched so that, during runtime, the only difference is whether the data is read from Pak or from disk.

To implement this process, the UE’s PakFile module needs to be modified. However, it is recommended not to directly modify the PakFile module but to create a class that inherits from FPakPlatformFile:

class PATCHPAKFILE_API FPatchPakPlatformFile: public FPakPlatformFile
{
public:
	virtual IFileHandle* OpenRead(const TCHAR* Filename, bool bAllowWrite = false) override;
	virtual IAsyncReadFileHandle* OpenAsyncRead(const TCHAR* Filename)override;
	virtual const TCHAR* GetName() const override { return TEXT("PatchPakFile"); }
};

Reading files from Pak is achieved by calling the OpenRead and OpenAsyncRead interfaces, where the above process can be implemented.

The core pseudocode for the main flow is as follows:

IFileHandle* FPatchPakPlatformFile::OpenRead(const TCHAR* Filename, bool bAllowWrite)
{
	IFileHandle* Handle = NULL;
	if(HasPatch(Filename))
	{
		Handle = GetLowerLevel()->OpenRead(GetPatchedAssetPath(Filename),bAllowWrite);
	}
	return !Handle ? FPakPlatformFile::OpenRead(Filename, bAllowWrite) : Handle;
}

The asynchronous process follows the same idea.

Once FPatchPakPlatformFile is implemented, it can be placed into the engine’s runtime. The engine’s code needs to be modified to replace the PakFile module with PatchPakFile so that our code can take effect.

Modify the Launch module’s LaunchEngineLoop.cpp file in the LaunchCheckForFileOverride function:

// From
IPlatformFile* PlatformFile = ConditionallyCreateFileWrapper(TEXT("PakFile"), CurrentPlatformFile, CmdLine);
// To
IPlatformFile* PlatformFile = ConditionallyCreateFileWrapper(TEXT("PatchPakFile"), CurrentPlatformFile, CmdLine);

This will allow the implemented reading process to be utilized when reading files from Pak, achieving patching behavior.

Conclusion

This article introduces a scheme for creating binary resource patches in UE, implementing binary resource patches based on HDiffPatch on the foundation of HotPatcher, and validating their feasibility through real-time or pre-patch methods. Further optimizations are needed for practical projects since the restored files are the original Cooked files and have not undergone encryption, raising resource security issues that also need to be addressed. If time permits, I will add more details on enhancing patch performance, encrypted resources, and other implementations.

To facilitate testing, I provide a ThirdPerson demo, which can be downloaded via the link: CompressionLab_WindowsNoEditor.7z.

The two files in the red box, one has gone through BinariesPatch and the other is the original Cooked file, can perform the same function and can be placed in CompressionLab/Content/Paks for testing.