In the game development process, hot updating is a very important capability. It allows us to update features and fix bugs without changing the package. However, it is not clear what can be hot updated, what cannot, and what risks are involved, as there are currently no detailed articles on this topic.
In this article, I will share some thoughts on the capabilities and safety of hot updates based on my development of HotPatcher and practical hot update projects. It is essential to ensure that the capability and stability of hot updates are guaranteed when the project goes live, and during the hot update iteration phase, it should be clear what can be hot updated, allowing us to effectively assess update risks. Additionally, through the construction of peripheral capabilities, risk checks can be conducted in advance, automating the hot update process, so that only the version PM needs to control the timing of patch construction and release without requiring program involvement in the hot update process.
Hot Update Process
Patch Construction
Based on the hot update packaging mechanism of HotPatcher: tracking changes in the original project, automatically analyzing all resources that need to be packaged, and generating differential patches with the previous version.
After the patch is generated, it will be uploaded to the CDN for subsequent testing and release processes.
In this hot update construction process based on this solution, the initiation and release of the hot update patch are controlled by the version PM, without program involvement, and the process is fully automated.
Runtime
At runtime, it is necessary to implement a dynamic downloading process for patches, preferably executing the mounting uniformly after the download is complete:
The priority of the released patches is incremental:
- The priority of the patch is higher than that of resources within the installation package and whole downloads.
- The latest patch (x.3) has a higher priority than the second latest patch (x.2), ensuring that the latest patch can cover old resources.
This ensures that the latest resources can override the old ones, achieving the update goal.
Update Capability
Updatable Content
Resources
The vast majority of resources in the game can be hot updated (contents that cannot be updated are introduced later). Active modifications to assets can be packaged normally, including shaders influenced by modified materials, which can also be incrementally tracked.
Additionally, HotPatcher can analyze passive changes in resources that have been caused, which will also be packaged.
The following situations fall under passive changes (which HotPatcher will automatically track):
- Modifications to parent blueprints that affect child blueprints.
- Modifications of UMG base controls that affect all controls referencing it in Instanced mode.
- Modifications of master materials that affect all material instances derived from it.
- ….
HotPatcher will analyze all lists of passive changes caused by actively modified assets and check if the resources for passive changes exist in the base version (to avoid introducing unnecessary new resources, only affecting the changed portions).
Moreover, it is necessary to minimize resources loaded before the completion of hot updates (such as key engine-dependent resources and resources for the hot update interface), as these cannot be hot updated and should be minimized as much as possible. To facilitate the investigation and cleaning of unnecessary assets, I provide an auxiliary function in HotPatcher that records all assets loaded before the hot update at startup, allowing precise identification of resources that cannot be hot updated, which can also be exposed in subsequent pre-checks.
Files
Most non-asset files required in the game can be hot updated:
- Script codes (lua/ts/py, etc.)
- db
- pb
- Multi-language (game.locres)
- Voice (wwise/wem)
- PSO
- Other custom non-asset files
As long as they can be tracked in the packaging process and are loaded (or reloadable) only after the hot update is completed, they can be hot updated. For localization, engine modifications may be necessary to delay loading until after the hot update, which needs to be evaluated based on business requirements.
ini
I provided a detailed introduction in a previous article: UE Hot Update: Reloading and Applying Config
In short, updates to the Ini generally support these three configuration situations:
ConsoleVariables
1
2[ConsoleVariables]
launcher.precompile.pso=1Object Config
1
2
3[/Script/UdpMessaging.UdpMessagingSettings]
EnableTransport=True
bAutoRepair=TrueDevicesProfiles
1
2
3
4
5
6
7
8
9[iPhoneXXX DeviceProfile]
.CVars=sg.ViewDistanceQuality=2
.CVars=sg.AntiAliasingQuality=2
.CVars=sg.ShadowQuality=2
.CVars=sg.PostProcessQuality=2
.CVars=sg.TextureQuality=2
.CVars=sg.EffectsQuality=2
.CVars=sg.FoliageQuality=2
.CVars=sg.ResolutionQuality=100.0
Note: IOS cannot dynamically modify
r.MobileContentScaleFactor. Android may experience click offsets. So if dynamic modifications to DeviceProfiles are desired, theMobileContentScaleFactorvalue shouldn’t be modified, and handling should be deferred to the next start.
Additionally, any ini configurations loaded during engine startup occur too early to be delivered via hot update; modifications to these require repackaging (though putting patches into the auto-mount directory is possible, it is a risky action that is not recommended, which will be detailed in the hot update safety section).
Recommendation: Modifications to the configurations in DeviceProfiles in the online version of ini should be avoided unless necessary. If changes are essential, a detailed evaluation should determine whether the specific content can be hot updated and the risks involved.
PSO
Although UE provides a mechanism to embed PSO in the installation package during building, we have not adopted this because the process of collecting PSO before limiting the build installation package is not flexible enough.
Our current PSO process is independent of the package building:
- During package building, no PSO-related elements are involved.
- After the installation package is generated, PSO is automatically collected.
- Once collection is completed,
stable.upipelinecacheis automatically generated and can be released as a hot update.
This separates PSO into an independent process, rather than a prerequisite. It also allows for direct hot updating of PSO (I will write a piece about PSO hot updates when time permits). PSO can be recollected and hot updated after several versions have been released.
Deleting Files
In most cases, we only need to focus on newly added and modified parts. However, one might ask, if I delete an asset or file, can this deletion be delivered through a hot update?
Yes! However, regarding deletion at the UFS layer, what is achieved is not the actual physical removal of files from the user’s disk. Since the files are packaged into PAKs, completely removing them from PAK requires regeneration, which poses some risks when executed locally.
Instead, it’s achieved by enabling UFS to not find the actual files, thus achieving the effect of “deletion”.
By analyzing the engine code, it can be known that the UnrealPak response file can pass the -delete parameter:
1 | "../../../Blank425/Blank425.uproject" "../../../Blank425/Blank425.uproject" -delete |
When it’s packaged as Pak, an empty Entry is actually added with a deleted marker:
Using UnrealPak’s -list, it appears similar to normal files but is specific in that it has no size, an offset of 0, and SHA1 of 0:
At runtime, when searching for files from the mounted PAK in UFS, if the deleted-marked Pak has a higher priority, it will trigger FoundDeleted:
Multiple deletion markers can be added as well:
1 | "../../../Blank245/Content/StarterContent/Maps/StarterMap.umap" "../../../Blank245/Content/StarterContent/Maps/StarterMap.umap" -delete |
The list of deleted files will also participate in the computation of the MountPoint:
In fact, if you want to mark a resource as deleted, the actual physical files do not need to exist because they will not actually be packaged into pak. Instead, an empty PakEntry is created via the MountPath’s path.
When UFS searches for files, it must first find this empty PakEntry; failing to load the real resources is how we achieve the intended “deletion” action.
The file marking deletion and the creation process of PAK:
- Read all lines from PakList.
- Check for the presence of the -delete token.
- If present, mark the Entry as
bIsDeleted.![]()
- In
CollectFilesToAdd, files marked asbIsDeletedwill be skipped and the actual disk files will not be searched.![]()
- In
CreatePakFile, each element ofFilesToAddwill be checked forbIsDeleted, and the corresponding PakEntry will be flagged withFlag_Deletedto indicate deletion, without packaging the local files those elements point to (this explains why any specified path can be marked for deletion).![]()
This implements the process from paklist.txt -> UnrealPak -> Deleted.pak -> runtime loading of PakEntry.
HotPatcher has automated this process. For deleted resources and files, if the patch configuration uses bIgnoreDeletedFiles=false, they will be marked for deletion in the output patch.
This effectively enables the hot update of the behavior “delete file“.
Contents that Cannot be Hot Updated
Contents that cannot be hot updated refers to those that cannot be dynamically downloaded to players in the game and can only be repackaged for release.
Unable to hot update sections include:
- Native/C++ code (code that affects runtime, VM-executable forwarded codes are not included here)
- Third-party libraries
- Package signatures, Entitlements
- Files within the package (e.g., models for GCloudVoice)
- Android Manifest, IOS PLIST, etc.
Parts not recommended for hot updating:
- uproject, uplugin
- ush/usf/global shader
- Modification of basic materials such as WorldGridMaterial (e.g.,
/Engine/EngineMaterials, as the timing of the pak mount and ShaderLibrary loading may differ, potentially causing crashes). - Resources that have been loaded before the completion of the hot update (such as key engine-dependent resources and resources for the hot update interface; these cannot be hot updated, so they should be minimized to maintain the smallest dependencies).
Beyond those that cannot be fully hot updated, the remaining elements can still be updated cautiously. They can be downloaded and placed in the auto-mount directory, assigned a higher priority than the assets within the installation package, and then made effective after a restart.
This is due to UFS’s mechanism, which loads files from PAK with the current highest priority, so if we ensure that the patch’s priority is highest at the engine’s startup, all files within UFS will be effective.
However, for safety considerations, I still include them in the list of contents that cannot be hot updated (even though the mechanism allows for such capabilities, they should not be used lightly). This is because the stability and safety of commercial products take precedence, and we need to make trade-offs between update capability and safety. The next subsection will elaborate on this.
Hot Update Safety
Because hot updates are very important, as they are the entry point to the game and the foundation of version iterations, ensuring robust hot updates themselves is crucial. Some projects might have encountered similar pitfalls, where a hot update patch caused the game to fail to start normally, necessitating a package switch. Therefore, I have elevated hot update safety to a very important level, even at the expense of some update capability.
Based on my practical experience, the focus on hot update safety should encompass the following aspects:
- The updated content should be stable and harmless.
- The updated content should be reversible.
- The updated content should not affect the next program startup.
- The updated content should not impact other hot update capabilities.
Thus, to achieve this, I have undertaken the following actions:
- Completely avoid using an automatic mounting mechanism; all patches are controlled by business logic for mounting, completely eliminating automatic mounting actions. This is because the timing of automatic mounting is too early; issues that arise could be fatal.
- Based on point 1, reject behaviors that depend strongly on a restart to take effect; all update content must be effective in real-time (evaluated based on project situations, e.g., handling of non-asset properties like
r.MobileContentScaleFactor). - Standardize the mounting/loading order and timing of PAK and ShaderLibrary to avoid mismatches between shader and pak content.
- Control priority strictly according to priority.
- Every initial game startup to the completion of the hot update should be identical to a new installation scenario, without any special logic. This ensures that no matter what we update, it can go through the hot update process normally, leaving room for remedy if a patch has issues.
Besides the hot updates themselves, resource management also requires strict control:
- Ensure each hot update is precisely traceable, accurately recording information about each changed resource, file, and their cooked information, along with detailing the current version’s repository and generated patch information.
- Check each asset’s specifications for each hot update, avoiding anomalies brought into the live environment by asset modifications.
Regarding resource management, after each hot update patch construction, we fully document the information for the current version:
Additionally, we archive patches and Metadatas:
This allows us to trace back the changes of each hot update version at any time, and we can check if the patches generated align with expectations.
Furthermore, for assets participating in hot updates, we conduct asset checks on all packages included. If rules are triggered, they are concurrently archived and publicly notified:
This allows clear visibility of potential risks within the current patch, facilitating assessment of any negative impacts from the patch. I have previously written several articles on the resource checking aspects. If you are interested, you can refer to the articles in the Resource Management category.
Conclusion
Related articles in the blog: Hot Update Series, Resource Management Series. Some content mentioned in this article will be covered in more detail in these articles, and those interested can check them out.
This article introduces some of my thoughts on the update capability and security of UE hot updates. Based on the actual project launch situation, the hot update mechanism is fault-tolerant, serving as a reference for practical implementation.
