In the process of game development, hot update is a very important capability that allows us to update features and fix bugs without replacing the entire package. However, there aren’t many detailed articles currently introducing what can be hot-updated, what cannot, and what carries risks.
In this article, I will introduce my thoughts on hot update capabilities and hot update security during the development of HotPatcher and project hot update practices. This ensures that the hot update capabilities and stability are guaranteed when the project goes online, and that it’s clear what can be hot-updated during the hot update iteration phase, allowing for a good assessment of update risks.
Additionally, by building peripheral capabilities, conducting risk assessments in advance, and automating the hot update process, only the version PM needs to control the patch construction and release timing, with no need for programmers to be involved in the hot update process at all.
Hot Update Process
Patch Construction
Based on the hot update packaging mechanism of HotPatcher: it tracks original project changes, automatically analyzes all resources that need to be packaged, and generates differential patches compared to the previous version.
After the patch is generated, it will be uploaded to CDN for subsequent testing and release processes.
Based on this hot update construction scheme, the initiation of patch construction and release are controlled by the version PM, no programmer involvement is required, and the intermediate process is fully automated.
Runtime
At runtime, a dynamic patch download process needs to be implemented, ideally, mounting should be performed uniformly after the download is complete:
The priority of hot-updated patches is incremental:
- The patch priority is higher than resources within the installation package and full-downloaded resources.
- The latest patch (x.3) has higher priority than the second newest patch (x.2), ensuring that the latest patch can overwrite old resources.
This ensures that the latest resources can overwrite older ones, achieving the update goal.
Update Capabilities
Updatable Content
Resources
Most resources within the game can be hot-updated (content that cannot be updated is introduced later). Active modifications and additions to assets can be packaged normally, including shaders affected by material modifications, which can be incrementally tracked.
Furthermore, HotPatcher can analyze resources that cause passive changes, and these will also be packaged.
The following situations are considered passive changes (HotPatcher tracks them automatically):
- Parent Blueprint modification affects child Blueprints
- UMG base widget modification affects all widgets that reference it in an Instanced manner
- Modification of a parent material affects all its material instances
- ….
HotPatcher analyzes all passive change lists caused by actively changed assets, and also checks if the passively changed resources exist in the base version (to avoid introducing unnecessary new resources, only affecting changed parts).
In addition, it’s necessary to minimize resources loaded before the hot update is complete (such as critical engine-dependent resources, hot update UI resources, etc.). These cannot be hot-updated and should be minimized.
To facilitate troubleshooting and cleaning up unnecessary assets, I have provided helper functions in HotPatcher which can record all assets loaded before the hot update at startup. This allows for precise knowledge of which resources cannot be hot-updated, and also allows them to be exposed in subsequent pre-checks.
Files
Most non-asset files required in the game can be hot-updated:
- Script code (lua/ts/py etc.)
- db
- pb
- Multilingual (game.locres)
- Voice (wwise/wem)
- PSO
- Other custom non-asset files
As long as they can be tracked during packaging and are loaded (or reloadable) after the hot update is complete, they can be hot-updated. For example, localization may require some engine modifications to defer loading until after the hot update; this needs to be assessed based on specific business requirements.
ini
My previous article had a detailed introduction: UE Hot Update: Config Reload and Application
Simply put, Ini updates typically support these three configuration scenarios:
ConsoleVariables
1
2[ConsoleVariables]
launcher.precompile.pso=1Object Config
1
2
3[/Script/UdpMessaging.UdpMessagingSettings]
EnableTransport=True
bAutoRepair=TrueDevicesProfiles
1
2
3
4
5
6
7
8
9[iPhoneXXX DeviceProfile]
.CVars=sg.ViewDistanceQuality=2
.CVars=sg.AntiAliasingQuality=2
.CVars=sg.ShadowQuality=2
.CVars=sg.PostProcessQuality=2
.CVars=sg.TextureQuality=2
.CVars=sg.EffectsQuality=2
.CVars=sg.FoliageQuality=2
.CVars=sg.ResolutionQuality=100.0
Note: IOS cannot dynamically modify
r.MobileContentScaleFactor. Android may experience click offset. Therefore, if you want to dynamically modify DeviceProfiles, you cannot change the value ofMobileContentScaleFactorand need to handle the delay until the next startup yourself.
Furthermore, ini configurations loaded during engine startup are too early to be delivered via hot update and require modification by repackaging the entire application (although putting patches in the auto-mount directory is possible, it is a dangerous practice and not recommended; this will be detailed in the hot update security section later).
Suggestion: For the online version, configurations in DeviceProfiles within ini files should not be changed unless absolutely necessary. If modification is essential, the actual content to be modified needs to be specifically evaluated for hot update feasibility and risks during the hot update release.
PSO
Although UE provides a mechanism to package PSO into the installation package during packaging, we did not adopt it because I find the process of collecting PSO before limiting the construction of the installation package not flexible enough.
Our current PSO process is independent of building the installation package:
- When building the installation package, no PSO-related elements are involved.
- After the installation package is built, PSO is collected automatically.
- After collection,
stable.upipelinecacheis automatically generated and can be released as a hot update.
This separates PSO as an independent process, and it is no longer a prerequisite. It also allows for direct hot updating of PSO (I will write an article on PSO hot updates if I have time later). PSO can be re-collected and hot-released after several hot update versions have been published.
Deleting Files
In most cases, we only need to focus on additions and modifications. But some might ask, if I delete an asset or file, can this deletion be delivered via hot update?
Yes, it can! However, for deletion at the UFS level, what is to be achieved is not truly physically removing the file from the user’s disk. Because files are packaged into PAKs, if you want to completely delete from a PAK, you need to regenerate the PAK, which carries some risks when executed locally.
This can be achieved by making UFS unable to find the actual file, thus serving as a ‘deletion’.
If actual deletion is required, runtime PAK recombination is needed, see my previous article: Runtime PAK Recombination Scheme in Unreal Engine.
Analyzing the engine’s code reveals that the -delete parameter can be passed via UnrealPak’s response file:
1 | "../../../Blank425/Blank425.uproject" "../../../Blank425/Blank425.uproject" -delete |
When packaging it into a Pak, an empty Entry is actually added and marked as Deleted:
Viewing it with UnrealPak’s -list, it appears like a normal file, but it’s special, having no size, offset of 0, and SHA1 of 0:
When searching for files from mounted PAKs in UFS at runtime, if a PAK marked for delete has a higher priority, FoundDeleted will be triggered:
Multiple deletion markers can also be added:
1 | "../../../Blank245/Content/StarterContent/Maps/StarterMap.umap" "../../../Blank245/Content/StarterContent/Maps/StarterMap.umap" -delete |
The list of deleted files also participates in MountPoint calculation:
In fact, if you want to mark a resource as deleted, the actual physical file does not need to exist because it won’t actually be packaged into the pak. An empty PakEntry is simply created using the MountPath.
By making UFS prioritize finding this empty PakEntry when searching for files, the actual resource cannot be loaded, which is the “deletion” requirement we aim to achieve.
File marking for deletion, PAK creation process:
- Read all lines from PakList
- Check for
-deletetoken - If present, mark the Entry as
bIsDeleted![]()
- In
CollectFilesToAdd, files marked withbIsDeletedwill be directly skipped, and the actual disk file will not be searched for.![]()
- In
CreatePakFile, each element ofFilesToAddwill be checked forbIsDeleted, and the corresponding PakEntry will be marked withFlag_Deletedto indicate deletion. Furthermore, the local file pointed to by that element will not be packaged into the Pak (this is also why any path can be specified and marked for deletion).![]()
This completes the workflow from paklist.txt -> UnrealPak -> Deleted.pak -> runtime loading of PakEntry.
HotPatcher has automated this process. For deleted resources and files, if bIgnoreDeletedFiles=false is used in the Patch configuration, they will be marked as deleted in the generated patch.
This enables the delivery of “file deletion“ actions via hot update.
Content That Cannot Be Hot-Updated
Content that cannot be hot-updated refers to content that cannot be dynamically downloaded and updated to players within the game and can only be released by repackaging the entire application.
Parts that cannot be hot-updated:
- Native/C++ code (code affecting Runtime; code that can be forwarded to a VM for execution is not included here)
- Third-party libraries
- Package signatures, Entitlements
- Files within the package (e.g., GCloudVoice model files)
- Android Manifest, IOS PLIST, etc.
Parts not recommended for hot update:
- uproject, uplugin
- ush/usf/global shader
- Modification of base materials like WorldGridMaterial (e.g.,
/Engine/EngineMaterils; different mounting times for PAKs and ShaderLibrary loading might lead to crashes) - Resources already loaded before the hot update is complete (critical engine-dependent resources, hot update UI resources, etc., these cannot be hot-updated, need to be minimized, keeping dependencies minimal)
Apart from those that absolutely cannot be hot-updated, the remaining ones are not impossible to update if necessary; just download them and place them in the auto-mount directory, set a higher priority than the assets within the installation package, and then restart to take effect.
Because UFS’s own mechanism is to load files from the PAK with the highest priority at the current time, as long as we ensure that the patch has the highest priority as soon as the engine starts, then all files within UFS can take effect.
However, for security reasons, I still include them in the list of content that cannot be hot-updated (the mechanism can provide such capabilities, but they should not be easily used). Because the stability and security of commercial products are more important factors, we need to make trade-offs between update capabilities and security, which will be detailed in the next subsection.
Hot Update Security
Because hot updates are very important, they are the entry point into the game and the cornerstone of version iteration, thus, making the hot update itself robust is crucial.
Some projects might have encountered similar pitfalls, where a hot update patch was delivered, causing the game to fail to start properly, requiring a complete re-package. Therefore, I’ve elevated hot update security to a very important position, even at the cost of sacrificing some update capabilities.
Based on my practical experience, the focus on hot update security should include the following aspects:
- The updated content should be stable and harmless.
- The updated content should be revocable.
- The updated content should not affect the next program startup.
- The updated content should not affect other hot update capabilities.
Therefore, to achieve this, I have done the following work:
- Completely abandoning the auto-mount mechanism, all patches are mounted by business logic control, completely eliminating auto-mount behavior. Because auto-mounting occurs too early, if problems arise, they are fatal.
- Based on point 1, rejecting strong reliance on restart for effectiveness; all updated content must take effect in real-time (can be evaluated according to project circumstances, such as handling non-asset attributes like
r.MobileContentScaleFactor). - Unifying the mounting/loading order and timing of PAKs and ShaderLibrary, to avoid mismatches between Shaders and assets within PAKs.
- Controlling priority strictly according to the mount Order.
- Every initial game startup, until the hot update is complete, is consistent with a fresh installation, with no special logic. This ensures that no matter what content we update, the hot update process can proceed normally, thus leaving room for remediation if there are issues with the patch.
In addition to hot updates themselves, resource management also needs strict control:
- Ensure that every hot update is precisely traceable, accurately recording information about every changed resource and file, as well as their cooked information, and recording the current version’s repository information and generated patch information.
- Check the asset specifications for each hot update to avoid introducing exceptions caused by asset modifications into the live network.
For resource management, after each hot update patch construction, we fully record the current version’s information:
And archived patches and Metadatas:
This allows us to trace back the changes of each hot update version at any time based on this data, and also check if the generated patch meets expectations.
Additionally, for assets involved in hot updates, we perform asset checks on all assets included in the package, and if rules are triggered, they will be archived synchronously and reminded in groups:
This allows for a clear understanding of potential risks within the current patch, facilitating the evaluation of the patch’s negative impact and preventing issues from being introduced into the live network.
I have written several articles on asset inspection before; if interested, you can refer to the articles in the Resource Management category.
Conclusion
Related articles in the blog: Hot Update Series, Resource Management Series. Some content mentioned in this article will have more detailed introductions in these articles; those interested can go and view them.
This article introduces some of my thoughts on UE hot update capabilities and hot update security. Based on actual project launch conditions, the hot update mechanism has had zero faults and can serve as a reference for engineering practices.
