In game development, hot update is a crucial capability, allowing us to update features and fix bugs without replacing the entire package. However, which elements can be hot-updated, which cannot, and which carry risks, are topics not extensively covered in current articles.
In this article, I will share my thoughts on hot update capabilities and safety, derived from my development of HotPatcher and practical experience with project hot updates. This ensures that when the project goes live, the hot update capability and stability are guaranteed. During the hot update iteration phase, it also provides clarity on what can be hot-updated, allowing for accurate assessment of update risks. Furthermore, by building peripheral capabilities, risks can be identified in advance, and the hot update process can be automated, requiring only the version PM to control patch building and release timing, with no programmer involvement needed in the hot update process.
Hot Update Process
Patch Building
Based on HotPatcher‘s hot update packaging mechanism: it tracks original project changes, automatically analyzes all resources that need to be packaged, and generates a differential patch against the previous version.
After generation, patches are uploaded to CDN for subsequent testing and release.
Based on this hot update building solution, the initiation of hot update patch building and release are controlled by the version PM, without programmer involvement, and the intermediate process is fully automated.
Runtime
At runtime, a dynamic patch download process needs to be implemented. It is best to perform unified mounting after the download is complete:
The priority of hot-updated patches is incremental:
- Patches have a higher priority than resources within the installation package and fully downloaded resources.
- The latest patch (x.3) has a higher priority than the second latest patch (x.2), ensuring that the latest patch can overwrite old resources.
This ensures that the latest resources can overwrite older ones, achieving the update goal.
Update Capability
Updatable Content
Assets
Most assets within the game can be hot-updated (content that cannot be updated is introduced later). Active modifications and additions to assets, including shaders affected by material changes, can all be properly packaged and incrementally tracked.
Furthermore, HotPatcher can analyze and package resources that cause passive changes.
The following situations are considered passive changes (HotPatcher automatically tracks these):
- Parent Blueprint modification affecting child Blueprints.
- UMG basic widget modification affecting all widgets that reference it in an Instanced manner.
- Master material modification affecting all its material instances.
- ….
HotPatcher analyzes all passive changes caused by actively modified assets and checks if these passively changed resources exist in the base version (to avoid introducing unnecessary new resources, only affecting the changed parts).
Additionally, it’s necessary to minimize resources loaded before the hot update is complete (such as engine-dependent critical resources, hot update UI resources, etc.). These cannot be hot-updated and should be reduced as much as possible.
To facilitate troubleshooting and cleaning up unnecessary assets, I’ve provided helper functions in HotPatcher that can record all assets loaded before the hot update at startup. This allows for precise identification of which resources cannot be hot-updated, and enables their exposure in subsequent pre-checks.
Files
Most non-asset files used in the game can be hot-updated:
- Script code (lua/ts/py etc.)
- db
- pb
- Multi-language (game.locres)
- Voice (wwise/wem)
- PSO
- Other custom non-asset files
As long as they can be tracked during packaging and are loaded (or reloadable) after the hot update is complete, they can be hot-updated. For example, localization might require some engine modifications to defer loading until after the hot update, which needs to be evaluated based on specific business requirements.
ini
I previously discussed this in detail in an article: UE Hot Update: Config Reload and Application
Simply put, Ini updates typically support these three configuration scenarios:
ConsoleVariables
1
2[ConsoleVariables]
launcher.precompile.pso=1Object Config
1
2
3[/Script/UdpMessaging.UdpMessagingSettings]
EnableTransport=True
bAutoRepair=TrueDevicesProfiles
1
2
3
4
5
6
7
8
9[iPhoneXXX DeviceProfile]
.CVars=sg.ViewDistanceQuality=2
.CVars=sg.AntiAliasingQuality=2
.CVars=sg.ShadowQuality=2
.CVars=sg.PostProcessQuality=2
.CVars=sg.TextureQuality=2
.CVars=sg.EffectsQuality=2
.CVars=sg.FoliageQuality=2
.CVars=sg.ResolutionQuality=100.0
Note: iOS cannot dynamically modify
r.MobileContentScaleFactor. Android may experience click offset. Therefore, if you want to dynamically modify DeviceProfiles, you cannot change the value ofMobileContentScaleFactor; you need to handle deferring it until the next launch.
In addition, ini configurations loaded during engine startup are too early to be delivered via hot update and require modifications through a new package release (although placing the patch in an auto-mount directory is possible, it’s a dangerous practice and not recommended, as will be explained in the hot update safety section later).
Recommendation: Do not modify DeviceProfiles configurations in the ini of the online version unless absolutely necessary. If modifications are required, a specific assessment of whether the actual changes can be hot-updated and their risks should be conducted during the hot update release.
PSO
Although UE provides a mechanism to embed PSO into the installation package during packaging, we did not adopt it because we found the process of collecting PSO before limiting the installation package build to be inflexible.
Our current PSO process is independent of building the installation package:
- No PSO-related elements are involved when building the installation package.
- After the installation package is built, PSO is automatically collected.
- After collection,
stable.upipelinecacheis automatically generated and can be released as a hot update.
This way, PSO is separated as an independent process, no longer a prerequisite. It also enables direct hot updating of PSO (I will write an article about PSO hot update later if I have time). PSO can be re-collected and hot-updated after several hot update versions have been released.
Deleting Files
In most cases, we only need to focus on additions and modifications. However, some might ask: if I delete an asset or file, can this deletion behavior be delivered via hot update?
Yes, it can! However, for deletion at the UFS level, the goal is not to physically remove the file from the user’s disk. Since files are packed into PAKs, fully deleting them from a PAK requires regenerating the PAK, which carries some risks when executed locally.
Instead, it achieves the “deletion” effect by making UFS unable to find the actual file. If actual deletion is required, runtime PAK reorganization is needed, as detailed in my previous article: Runtime Reorganization Solution for Pak in Unreal Engine.
Analyzing the engine’s code reveals that the -delete parameter can be passed via UnrealPak’s response file:
1 | "../../../Blank425/Blank425.uproject" "../../../Blank425/Blank425.uproject" -delete |
When packaged into a Pak, an empty Entry is actually added and marked as Deleted:
Viewing it with UnrealPak’s -list shows it acts like a normal file, but is special: it has no size, offset is 0, and SHA1 is also 0:
When searching for files from an already mounted PAK in UFS at runtime, if a Pak marked for deletion has a higher priority, FoundDeleted will be triggered:
Multiple deletion markers can also be added:
1 | "../../../Blank245/Content/StarterContent/Maps/StarterMap.umap" "../../../Blank245/Content/StarterContent/Maps/StarterMap.umap" -delete |
The list of deleted files also participates in MountPoint calculation:
In fact, to mark a resource as deleted, the actual physical file doesn’t need to exist, because it won’t actually be packaged into the pak. Instead, an empty PakEntry is created via this MountPath.
By making UFS prioritize finding this empty PakEntry when searching for files, and thus failing to load the actual resource, we achieve our “deletion” requirement.
File marked for deletion, PAK creation process:
- Read all lines from PakList.
- Check for
-deletetoken. - If present, mark the Entry as
bIsDeleted.![]()
- In
CollectFilesToAdd, files markedbIsDeletedare directly skipped; no attempt is made to find the actual disk file.![]()
- In
CreatePakFile, each element inFilesToAddis checked forbIsDeleted, and the corresponding PakEntry is given aFlag_Deletedmark to indicate deletion. Furthermore, the local file pointed to by that element will not be packaged into the Pak (this is also why any path can be specified to mark for deletion).![]()
This completes the workflow from paklist.txt -> UnrealPak -> Deleted.pak -> runtime loading of PakEntry.
HotPatcher has automated this process. For deleted resources and files, if the Patch configuration uses bIgnoreDeletedFiles=false, they will be marked as deleted in the generated patch.
This way, the behavior of “deleting files” can be delivered via hot update.
Non-Hot-Updatable Content
Non-hot-updatable content refers to content that cannot be dynamically downloaded and updated to players within the game; it requires a new full package release.
Parts that cannot be hot-updated:
- Native/C++ code (code affecting Runtime; code that can be forwarded to VM for execution is not included here).
- Third-party libraries.
- Package signatures, Entitlements.
- Files within the package (e.g., GCloudVoice model files).
- Android Manifest, iOS PLIST, etc.
Parts not recommended for hot update:
- uproject, uplugin
- ush/usf/global shader
- Modifications to base materials like WorldGridMaterial (e.g.,
/Engine/EngineMaterils; different timing for pak mounting and ShaderLibrary loading might lead to crashes). - Resources already loaded before the hot update is complete (engine-dependent critical resources, hot update UI resources, etc. These cannot be hot-updated and should be minimized to maintain minimal dependencies).
Apart from those that are completely non-hot-updatable, the remaining items can be updated if absolutely necessary by downloading them, placing them in an auto-mount directory, setting a higher priority than in-package assets, and then restarting for the changes to take effect.
Because UFS’s inherent mechanism is to load files from the PAK with the highest priority at the current time, as long as we ensure that the patch has the highest priority as soon as the engine starts, all files within UFS can take effect.
However, for safety considerations, I still list them as non-hot-updatable (the mechanism can provide this capability, but it shouldn’t be used lightly). Stability and security are more critical factors for commercial products, and we need to make trade-offs between update capability and safety, as will be detailed in the next subsection.
Hot Update Safety
Because hot update is extremely important, serving as the entry point into the game and the foundation for version iteration, it’s crucial to ensure the hot update mechanism itself possesses strong robustness.
Some projects might have experienced similar pitfalls where a deployed hotfix patch prevents the game from starting properly, necessitating a full package replacement. Therefore, I’ve elevated hot update safety to a very critical level, even if it means sacrificing some update capabilities.
Based on my practical experience, attention to hot update safety should encompass the following aspects:
- Updated content should be stable and harmless.
- Updated content should be reversible.
- Updated content should not affect the next program launch.
- Updated content should not affect other hot update capabilities.
Therefore, to achieve this, I have done the following:
- Completely avoid automatic mounting mechanisms; all patches are mounted under the control of business logic, entirely eliminating automatic mounting. This is because automatic mounting occurs too early, and any issues at that stage can be fatal.
- Building upon point 1, reject actions that heavily rely on a restart to take effect. All updated content must take effect in real-time (can be evaluated based on project specifics, e.g., handling non-asset properties like
r.MobileContentScaleFactor). - Unify the mounting/loading order and timing of PAK and ShaderLibrary to prevent mismatches between shaders and assets within the Pak.
- Strictly control priorities based on priority levels.
- Each initial game launch, up until the hot update is complete, should behave identically to a fresh installation, without special logic. This ensures that no matter what content we update, the hot update process proceeds normally, providing room for recovery if a patch has issues.
Beyond the hot update itself, resource management also requires strict control:
- Ensure every hot update is precisely traceable, meticulously recording information for each changed resource and file, their cooked information, and details of the current version’s repository and generated patch.
- Check asset specifications for each hot update to prevent anomalies caused by asset modifications from being introduced into the live environment.
For resource management, after each hot update patch build, we fully record the current version’s information:
And archived patches and Metadatas:
This allows for tracking changes in each hot update version using this data at any time, and for checking if the generated patch meets expectations.
Additionally, for assets involved in hot updates, we perform asset checks on all assets included in the package. If a rule is triggered, it will be archived simultaneously and a group notification will be sent:
This provides a clear understanding of potential risks within the current patch, facilitating the assessment of its negative impact and preventing issues from being introduced into the live environment.
I’ve written several articles about resource checking before; if you’re interested, you can refer to the articles in the Resource Management category.
Conclusion
Related articles on the blog: Hot Update Series, Resource Management Series. Some content mentioned in this article will be introduced in more detail in those articles; interested readers can go there for more information.
This article introduces some of my thoughts on hot update capabilities and safety in UE hot updates. Based on actual project launches, the hot update mechanism has been fault-free, serving as a reference for engineering practice.
