Z's Blog

UE Hot Update: split the basic package

UE热更新：拆分基础包
Posted on In , Views:
Symbols count in article: 7.5k Reading time ≈ 19 mins.

In previous articles, we introduced the implementation mechanism of UE hot updates and the automation process of hot updates. Recently, we plan to continue writing a few articles to discuss the process and rules of resource package management in UE hot updates.

Of course, different types of projects will have different packaging strategies, and resource management does not have a universal best strategy. This article mainly introduces the engineering practice of splitting the base package in the hot update process, involving the modification of the engine to implement a common splitting method for Android/iOS. We hope to provide some useful ideas for projects in different businesses.

In practical engineering, we abandoned the default package splitting scheme provided by UE and implemented a flexible package splitting scheme based on the HotPatcher framework. For details, see the article: Resource Management: Reshaping UE’s Package Splitting Scheme.

When projects develop to the mid- to late-stages, there will be a large number of maps and art resources. For mobile games, the size of the package is also quite sensitive, and Android has a 2G package size limitation. Therefore, when a project reaches a certain stage, splitting the package becomes a necessary consideration, and a giant installation package is not conducive to promotion.

In the case of hot updates, splitting the base package needs to consider two factors:

  • Reducing the package size without affecting gameplay
  • Minimizing players’ download waiting time

The trade-offs and implementations of these two factors are more or less related to specific game businesses. Here, we will only split the base package from the implementation level; different businesses can split according to their own business rules.

UE’s resource management can split resources during packaging, using Asset Manager or AssetPrimaryLabel to categorize resources by type, directory, map, dependency analysis, etc. This is the so-called Chunk mechanism.
For the operation method of Chunk division, refer to UE’s documentation:

UE’s Chunk mechanism can turn a single giant pak that is created by default into several smaller pak files based on the specified splitting granularity (when Priority is the same, there may be a situation where a resource exists in multiple chunks). If you take maps and their dependent resources as the chunk division mechanism, you can include the initial key maps in the base package and dynamically download the remaining resources during hot updates or runtime.

The key to splitting the base package lies in two points:

  1. Being able to split resources (chunks) based on custom classifications
  2. Being able to control the rules for packing resources into the base package

The first point can be achieved through the Chunk mechanism of Asset Manager. So, how can the second point be achieved?

Android

For the Android platform, because a package larger than 2GB will cause packaging failures, UE provides the ObbFilter functionality by default to specify which files should be added to OBB (pak/mp4), etc.

The control method only requires adding configurations.

1
2
3
4
5
# Config/DefaultEngine.ini
[/Script/AndroidRuntimeSettings.AndroidRuntimeSettings]
+ObbFilters=-pakchunk1-*
+ObbFilters=-pakchunk2-*
+ObbFilters=-pakchunk3-*

The rules of ObbFilters starting with - serve as exclusion rules, which filter out chunk1-3 pak from the base package and can be used for subsequent download processes.

It is also possible to specify a combination of Exclude and Include rules:

1
2
+ObbFilters=-*.pak
+ObbFilters=pakchunk0-*

The first step ignores all pak files and then explicitly adds pakchunk0-*.pak to the obb.

iOS

However, iOS does not provide this functionality. To achieve iOS filtering mechanisms similar to Android, I looked into the UE packaging code for iOS, and it can be implemented in the following way (modifying the iPhonePackager code). The thought and implementation process is recorded as follows.

Note: The IPA packaging method I use is remote building. For details, see the previous article: UE4 Development Notes: Mac/iOS.

In the earlier sections, it was mentioned that UE provides file filtering rules for packaging into OBB for Android:

1
2
3
# Config/DefaultEngine.ini
[/Script/AndroidRuntimeSettings.AndroidRuntimeSettings]
+ObbFilters=-pakchunk1-*

However, UE does not provide a corresponding operation for iOS. By default, all pak files for iOS are packaged into the IPA.

To unify the base package rules of Android and iOS, I implemented a filtering rule functionality on iOS similar to Android. Here’s a simple introduction.

I used Mac for remote packaging. The process involves compiling code to generate IPA on Mac, pulling it back to Windows for cooking, generating Pak files, and finally unpackaging the original IPA, adding Pak and other files to form the final IPA.

My requirement is to specify custom filtering rules that can ignore certain files and not pack them into the IPA. This operation actually lies within the process of unpackaging and repackaging the IPA. After reviewing UE’s code, I found that this operation is performed through the iPhonePackager standalone program, which means that I need to modify the code of this program.

After debugging and analysis, I found that the actual operation of repackaging the IPA occurs in the following function:

1
2
3
4
5
Programs/IOS/iPhonePackager/CookTime.cs
/** 
* Using the stub IPA previously compiled on the Mac, create a new IPA with assets
*/
static public void RepackageIPAFromStub();

This function is located in the iPhonePackager - CookTime class.

1
2
3
4
5
6
7
8
9
10
11
12
static public void RepackageIPAFromStub()
{
	// ...
	string SourceDir = Path.GetFullPath(ZipSourceDir);
	string[] PayloadFiles = Directory.GetFiles(SourceDir, "*.*", Config.bIterate ? SearchOption.TopDirectoryOnly : SearchOption.AllDirectories);
 	foreach (string Filename in PayloadFiles)
    {
       // read file to memory, add to ZipFileSystem
       // generate stub and ipa
    }
  	//...
}

The operation to perform is to intervene in this process and filter the list of files in PayloadFiles based on our custom rules.

The process can be divided into the following steps:

  1. Read Filter configuration from the project
  2. Create the actual filter
  3. Use the filter to check whether files should be included in the IPA during the RepackageIPAFromStub file traversal process

This can be achieved with just a few dozen lines of code. First, an IniReader class needs to be added:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
using Tools.DotNETCommon;
using System.Runtime.InteropServices;
using Ini;

namespace Ini
{
	public class IniReader
	{
		private string path;

		[DllImport("kernel32")]
		private static extern int GetPrivateProfileString(string section, string key, string def,
			StringBuilder retVal, int size, string filePath);

		public IniReader(string INIPath)
		{
			path = INIPath;
		}

		public string ReadValue(string Section, string Key)
		{
			StringBuilder ReaderBuffer = new StringBuilder(255);
			int ret = GetPrivateProfileString(Section, Key, "", ReaderBuffer, 255, this.path);
			return ReaderBuffer.ToString();
		}
	}
}

Then, create the filter in the RepackageIPAFromStub function:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
FileFilter IpaPakFileFilter = new FileFilter(FileFilterType.Include);
{
	string ProjectDir = Directory.GetParent(Path.GetFullPath(Config.ProjectFile)).FullName;
	// Program.Log("ProjectDir path {0}", ProjectDir);
	string EngineIni = Path.Combine(ProjectDir,"Config","DefaultEngine.ini");
	// Program.Log("EngineIni path {0}", EngineIni);
	IniReader EngineIniReader = new IniReader(EngineIni);
	string RawPakFilterRules = EngineIniReader.ReadValue("/Script/IOSRuntimeSettings.IOSRuntimeSettings", "IPAFilters");
	Program.Log("RawPakFilterRules {0}", RawPakFilterRules);
	string[] PakRules = RawPakFilterRules.Split(',');		
	// foreach(string Rule in PakRules) {Program.Log("PakRules {0}", Rule);}
  
	List<string> PakFilters = new List<string>(PakRules);
	if (PakFilters != null)
	{
		IpaPakFileFilter.AddRules(PakFilters);
	}
}

Here, the value of IPAFilters is read from the project’s Config/DefaultEngine.ini under [/Script/IOSRuntimeSettings.IOSRuntimeSettings], with rules similar to Android, allowing specification of both Exclude and Include rules, but all rules must be written in one line and separated by commas.

1
2
[/Script/IOSRuntimeSettings.IOSRuntimeSettings]
IPAFilters=-*.pak,pakchunk0-*

Finally, you also need to detect whether the files match our specified filtering rules during the iteration of Payload files in RepackageIPAFromStub:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
static public void RepackageIPAFromStub()
{
	// ...
	string SourceDir = Path.GetFullPath(ZipSourceDir);
	string[] PayloadFiles = Directory.GetFiles(SourceDir, "*.*", Config.bIterate ? SearchOption.TopDirectoryOnly : SearchOption.AllDirectories);
	foreach (string Filename in PayloadFiles)
	{
		if (!IpaPakFileFilter.Matches(Filename))
		{
			Program.Log("IpaPakFileFilter not match file {0}", Filename);
			continue;
		}
		// Program.Log("IpaPakFileFilter match file {0}", Filename);
	}
	//...
}

Now, when packaging for iOS, files will be added according to the specified filtering rules, achieving consistent behavior with Android.

The log during the packaging process is as follows (the above code has been commented out):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
Saving IPA ...
ProjectDir path C:\BuildAgent\workspace\PackageWindows\Client
EngineIni path C:\BuildAgent\workspace\PackageWindows\Client\Config\DefaultEngine.ini
RawPakFilterRules -*.pak,pakchunk0-*
PakRules -*.pak
PakRules pakchunk0-*
IpaPakFileFilter match file C:\BuildAgent\workspace\PackageWindows\Client\Saved\StagedBuilds\IOS\Assets.car
IpaPakFileFilter match file C:\BuildAgent\workspace\PackageWindows\Client\Saved\StagedBuilds\IOS\Info.plist
IpaPakFileFilter match file C:\BuildAgent\workspace\PackageWindows\Client\Saved\StagedBuilds\IOS\LaunchScreenIOS.webp
IpaPakFileFilter match file C:\BuildAgent\workspace\PackageWindows\Client\Saved\StagedBuilds\IOS\Manifest_DebugFiles_IOS.txt
IpaPakFileFilter match file C:\BuildAgent\workspace\PackageWindows\Client\Saved\StagedBuilds\IOS\Manifest_NonUFSFiles_IOS.txt
IpaPakFileFilter match file C:\BuildAgent\workspace\PackageWindows\Client\Saved\StagedBuilds\IOS\mute.caf
IpaPakFileFilter match file C:\BuildAgent\workspace\PackageWindows\Client\Saved\StagedBuilds\IOS\ue4commandline.txt
IpaPakFileFilter match file C:\BuildAgent\workspace\PackageWindows\Client\Saved\StagedBuilds\IOS\cookeddata\fgame\content\movies\logo.mp4
IpaPakFileFilter match file C:\BuildAgent\workspace\PackageWindows\Client\Saved\StagedBuilds\IOS\cookeddata\fgame\content\movies\sparkmore.mp4
IpaPakFileFilter match file C:\BuildAgent\workspace\PackageWindows\Client\Saved\StagedBuilds\IOS\cookeddata\fgame\content\paks\pakchunk0-ios.pak
IpaPakFileFilter not match file C:\BuildAgent\workspace\PackageWindows\Client\Saved\StagedBuilds\IOS\cookeddata\fgame\content\paks\pakchunk1-ios.pak
IpaPakFileFilter not match file C:\BuildAgent\workspace\PackageWindows\Client\Saved\StagedBuilds\IOS\cookeddata\fgame\content\paks\pakchunk2-ios.pak
...

As can be seen, the filtering rules have taken effect. Additionally, this will not have any other adverse effects on the packaged bundle (of course, this may result in losing resources by default, and a download mechanism needs to be implemented, which can refer to my previous articles in the hot update series).

Note: Since iPhonePackager is a Program-type program that does not depend on the engine, it can be compiled and then copied for use with non-source versions of the engine.

Note: In non-remote builds, you cannot modify the iPhonePackager code when packaging the iOS package directly on Mac, as non-remote builds will not utilize it. To achieve the same effect, it is necessary to modify the process in IOSPlatform.Automation.cs, adding the above code to the Package function to implement the filtering behavior.

Furthermore, by default, UE should not compile the C# program’s project on Mac. You can modify the AutomationTool on Windows and then copy it to Mac.

Priority Packaging Fails

When splitting the base package, I wanted to minimize resource redundancy by controlling the PrimaryAssetLabel‘s Priority, but tests showed that it did not work (resources with both high and low priority were present in the same chunk):

Looking at the code in the engine, resource management is obtained via AssetManager::GetPackageManagers:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
Engine\Source\Runtime\Engine\Private\AssetManager.cpp
bool UAssetManager::GetPackageManagers(FName PackageName, bool bRecurseToParents, TSet<FPrimaryAssetId>& ManagerSet) const
{
  IAssetRegistry& AssetRegistry = GetAssetRegistry();

  bool bFoundAny = false;
  TArray<FAssetIdentifier> ReferencingPrimaryAssets;
  ReferencingPrimaryAssets.Reserve(128);

  AssetRegistry.GetReferencers(PackageName, ReferencingPrimaryAssets, EAssetRegistryDependencyType::Manage);

  for (int32 IdentifierIndex = 0; IdentifierIndex < ReferencingPrimaryAssets.Num(); IdentifierIndex++)
  {
    FPrimaryAssetId PrimaryAssetId = ReferencingPrimaryAssets[IdentifierIndex].GetPrimaryAssetId();
    if (PrimaryAssetId.IsValid())
    {
      bFoundAny = true;
      ManagerSet.Add(PrimaryAssetId);

      if (bRecurseToParents)
      {
        const TArray<FPrimaryAssetId> *ManagementParents = ManagementParentMap.Find(PrimaryAssetId);

        if (ManagementParents)
        {
          for (const FPrimaryAssetId& Manager : *ManagementParents)
          {
            if (!ManagerSet.Contains(Manager))
            {
              // Add to end of list, this will recurse again if needed
              ReferencingPrimaryAssets.Add(Manager);
            }
          }
        }
      }
    }
  }
  return bFoundAny;
}

By default, the Priority value is not utilized at all.

Thus, I need to modify this part of the engine’s implementation so that when retrieving the resource’s associated PrimaryAssetId, it captures the list of Labels with the maximum (or highest identical) Priority. However, if labels are used at runtime, it becomes a bit counterintuitive. It is recommended that the Priority function be effective only during packaging, and at runtime, loading a PrimaryAssetLabel should not affect resource management. The engine launch parameters can be checked through FParse::Param to detect this.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
FPrimaryAssetId AddPrimaryAssetId = PrimaryAssetId;
if(FParse::Param(FCommandLine::Get(),TEXT("labelpriority")))
{
  TArray<FPrimaryAssetId> ManagerArray = ManagerSet.Array();
  FPrimaryAssetRules Rules = GetPrimaryAssetRules(PrimaryAssetId);
  for(const auto& Manager:ManagerArray)
  {
    FPrimaryAssetRules ExistRules = GetPrimaryAssetRules(Manager);
    if(Rules.Priority > ExistRules.Priority)
    {
      ManagerSet.Remove(Manager);
    }
    if(Rules.Priority < ExistRules.Priority)
    {
      AddPrimaryAssetId = FPrimaryAssetId();
    }
  }
}
if(AddPrimaryAssetId.IsValid())
{
  ManagerSet.Add(PrimaryAssetId);
}

By enabling -labelpriority during cook time, resources will only exist in chunks with a higher Priority value.

End

Through the above operations, you can realize consistent base package filtering rules for Android/iOS, packing the most critical resources into the base package. The remaining pak files can be extracted from Saved/StagedBuilds after completing the base package, to be downloaded at runtime according to the startup of the hot update platform, or designed according to the project’s type and requirements for a runtime download scheme.

The article is finished. If you have any questions, please comment and communicate.

Scan the QR code on WeChat and follow me.

Title:UE Hot Update: split the basic package
Author:LIPENGZHA
Publish Date:2021/01/27 21:51
Word Count:7.5k Words
Link:https://en.imzlp.com/posts/13765/
License: CC BY-NC-SA 4.0
Reprinting of the full article is prohibited.
Your donation will encourage me to keep creating!