DevOps is a philosophy based on automation for Continuous Integration (CI) and Continuous Deployment (CD), optimizing all aspects of development, testing, and operations. It emphasizes the integration of software development, testing, and operations, reducing communication costs between departments or processes to achieve rapid and high-quality releases and iterations.
Although the development situation in the gaming field differs significantly from traditional internet development, leveraging the concept of DevOps to integrate all production elements of a project into automation can greatly enhance efficiency.
This article will introduce the basic concepts of automation in UE and share some of my attempts and actual engineering practices regarding automation builds in Unreal Engine.
Forward
I believe that the goals of automation in game development can be summarized into the following five core demands:
- Reduce human involvement in tedious tasks
- Lower the cost of identifying issues while improving efficiency
- Rapidly iterate versions and produce experienceable content
- Parameterized control and expandable processes
These demands correspond to various stages in the development process, and I will analyze the actual pain points related to these demands in UE later. In the employment market of UE, even the concept of a “Build Engineer” has gradually emerged, specifically addressing the build needs of projects.
In work, when facing tasks that involve handling builds, one should maintain a mindset of in-depth research. Because it’s not simply about packaging; it involves optimizing aspects across the entire development process. Identifying bottlenecks and pain points, optimizing and resolving them will naturally reflect their value.
If one’s perspective is limited to merely packaging these superficial build needs, it poses significant disadvantages in terms of accumulating knowledge for technical personnel, making self-accumulation essential.
DevOps Platform
The so-called DevOps platform is, simply put, an execution environment that is convenient and orchestratable. It can host custom machines, orchestrate execution sequences, initiate execution commands, collect results, and schedule tasks.
There are many products in this area, including open-source ones like Jenkins and Github Actions from GitHub. In big companies, there may also be self-developed CI/CD systems, such as Tencent’s Blue Shield.
You can edit the pipelines needed for execution:
This article won’t elaborate on their deployment since, fundamentally, they are just environments that orchestrate processes and initiate execution commands, not the focus of UE’s CI/CD practices.
Build
Engine/Editor
In project development, besides programmers, planners or artists typically do not have development capability and don’t require code, and thus usually do not use the source version of the engine. Instead, they create a binary version of the engine for their use.
Just like the engine installed from EpicLauncher:
One aspect of implementing CI/CD builds in an UE project is to easily build a binary version of the engine and editor, allowing non-programmers to completely avoid the need to compile locally. Moreover, this binary engine can be reused; if the engine code hasn’t changed, there’s no need to rebuild, only incrementally compile the project.
I have detailed this in the article BuildGraph: Building a Binary Engine that Supports Multi-Platform Packaging, so I won’t repeat what’s already been discussed.
The final process to achieve looks like this:
Each execution updates both the engine and Client code, generating a binary engine and project, and for safety considerations, the C++ code of the engine and client can also be excluded.
In simple terms, the implementation is to use BuildGraph to construct a binary engine version from the source version of the engine:
1 | RunUAT.bat BuildGraph -target="Make Installed Build Win64" -script=Engine/Build/InstalledEngineBuild.xml -set:WithMac=false -set:WithAndroid=false -set:WithIOS=false -set:WithTVOS=false -set:WithLinux=false -set:WithHTML5=false -set:WithSwitch=false -set:WithDDC=false -set:WithWin32=false -set:WithLumin=false -set:WithPS4=false -set:WithXboxOne=false -set:WithHoloLens=false -set:GameConfigurations=Development |
Then, using the generated binary engine to compile the project will only compile the modules within the project:
1 | Engine\Build\BatchFiles\Build.bat |
This is the same command that is executed when starting the compilation in VS.
Passing Compilation Parameters
Sometimes, when compiling the binary engine and project, you may want to pass parameters to UE to control compilation behavior. This allows for dynamic control of some feature switches. There are two scenarios when passing compilation parameters: during engine compilation and client project compilation.
When constructing the binary engine using BuildGraph, the default InstalledEngineBuild.xml cannot specify UBT parameters. You can copy a version of InstalledEngineBuild.xml and modify it:
1 | <Option Name="UBTArgs" DefaultValue="" Description="compile UE4Editor UBT Args" /> |
Then you can pass it when calling BuildGraph: -set:UBTArgs=.
For compiling the Client project, you can directly append new parameters to the command used to call UBT to compile the project:
1 | Engine\Build\BatchFiles\Build.bat |
In build.cs or target.cs, you can access the UBT command line parameters to control compilation options:
1 | string[] CmdArgs = Environment.GetCommandLineArgs(); |
Compilation output:
1 | 0>E:\UnrealEngine\Launcher\UE_4.26\Engine\Build\BatchFiles\Build.bat GWorldEditor Win64 Development -Project="E:\UnrealProjects\GWorld\GWorld.uproject" -WaitMutex -FromMsBuild -test |
Based on the passed parameters, the compilation behavior can be dynamically modified, such as adding macros, dependent modules, etc.
Moreover, based on this approach, the impact is limited to the current module and its dependent modules, negating the need for a complete recompilation, and fully utilizing the compilation cache of unchanged modules.
Packaging
Packaging is the most fundamental requirement in UE builds. Within the UE editor, there are two main methods for packaging options.
The first method is File - Package Project:
The second method is ProjectLauncher, allowing you to edit some configuration options, like the Cook platform, Cultures, etc.:
However, they both execute through UAT, invoking BuildCookRun. UAT is an independent program that acts as a task scheduler, launching multiple processes to execute different stage tasks, used to coordinate the entire packaging process, and can control skipping certain stages.
1 | Engine\Engine\Build\BatchFiles\RunUAT.bat |
When we want to package, we can execute the above command.
However, to integrate packaging into the CI/CD system, we also need to encapsulate its parameters, such as:
- Specify platform (Win64/Android/IOS, etc.)
- Cook platform (WindowsNoEditor/Android_ASTC/IOS, etc.)
- Configuration (Debug/Development/Shipping)
- UBT compilation parameters
- Parameters for CookCommandlet
- Whether to enable encryption
- Output path
And so on, all options that need dynamic control must be exposed as parameters.
You can use scripts in Python or other scripting languages for encapsulation:
Passing Compilation Parameters
In the BuildCookRun command, you can specify parameters for UBT using -ubtargs= to dynamically control behavior during compilation.
1 | Engine\Engine\Build\BatchFiles\RunUAT.bat |
You can see in the UBT calling command that the parameters have been passed:
The parameters can also be obtained in build.cs or target.cs:
1 | string[] CmdArgs = Environment.GetCommandLineArgs(); |
Passing Cook Parameters
During packaging, after code compilation is complete, UAT will invoke the engine to execute CookCommandlet, which handles resources.
The default parameters UAT calls Cook with are:
1 | D:\UnrealEngine\Source\UE_4.27.2\Engine\Binaries\Win64\UE4Editor-Cmd.exe |
But how do you pass parameters to Cook via UAT’s command?
UE provides a parameter for BuildCookRun to achieve this need: -AdditionalCookerOptions=.
1 | Engine\Engine\Build\BatchFiles\RunUAT.bat |
When UAT calls CookCommandlet, these parameters will be passed:
1 | D:\UnrealEngine\Source\UE_4.27.2\Engine\Binaries\Win64\UE4Editor-Cmd.exe |
Its purpose is to avoid manual separation and management of the packaging process. By receiving these parameters as plugins, it can automatically intervene in the packaging process.
Realizing effects similar to the one shown below:
I have utilized this approach in several previous articles:
- UE Plugin and Tool Development: Commandlet
- Resource Management: Restructuring UE’s Package Splitting Scheme
- A Flexible and Non-Intrusive Base Package Splitting Scheme
In addition, if you want to use Unreal Insights to analyze the time consumption of various parts of Cook during the packaging process, you can also pass profiling parameters:
1 | Engine\Engine\Build\BatchFiles\RunUAT.bat |
Cmdlet Automation
In UE’s CI/CD practices, there are significant demands for automatically importing and exporting data from the engine. This includes launching the engine via command line to process resources automatically.
A few days ago, I wrote an article introducing the mechanism and engineering techniques of Commandlet, which can be viewed in detail at: UE Plugin and Tool Development: Commandlet.
For example, from my blog articles, Commandlet automation can be applied to:
- Resource Management: Restructuring UE’s Package Splitting Scheme
- Multi-Phase Automation Resource Inspection Scheme in UE
- Automation Practice for Resource Inspection Based on ResScannerUE
- UE Hot Update: Automated Processes Based on HotPatcher
- Export Recast Navigation Data from UE4
Cmdlet automation can export NavMesh data, generate base package data, hot update packages, automatically execute package splitting in the CookCmdlet process, scan resource specifications, and more.
By fully utilizing Cmdlet for automation, human involvement in resource processing demands is minimized, reducing the probability of errors and increasing efficiency.
Pre-check
In engineering practices, it is often necessary to perform pre-processing for cross-platform code compilation and resource checks. This helps reveal problems early and avoids leaving problematic parts until the packaging and runtime stages, ensuring the stability of built outputs.
Compilation
In most projects, development is cross-platform. The same project produces packages for various platforms; the most common ones are Windows/Android/IOS, etc.
In many cases, for “compilation,” the same code needs to be compatible across the following platforms:
- Win64Editor
- Win64
- MacEditor
- Android
- IOS
However, due to environmental differences, discrepancies in compiler standards for different platforms, warning levels, differences in linking libraries, and different compilation parameters in UE for different targets, the same code that compiles successfully on Win64Editor may not compile successfully on Android or IOS.
Therefore, under these circumstances, checking the cross-platform compilation of code is very necessary. After code changes, automatically compiling across all required platforms to check for errors is recommended.
In fact, the command executed for each platform is a wrapper around UBT:
1 | # .bat for Windows, .sh for Mac |
By adjusting the -Target parameter, you can compile different Targets, Platforms, and Configurations.
Additionally, one of the most common compilation issues in daily development is caused by merging translation units in UE.
The specific phenomenon is that code A hasn’t changed for a long time, while changes in other code B lead to errors in A code.
This is due to merged translation units; previously, A and other translation units were merged together, and although A didn’t include all the required header files, another translation unit might have included them, so the merged compilation wouldn’t raise errors.
Thus, in pre-compilation checks, you can disable UnityBuild in the project to allow each translation unit to compile independently, which will also help in detecting such errors.
However, disabling UnityBuild can slow down compilation, so it should only be turned off during pre-compilation checks, while in the formal processes, you should still use it. How to distinguish between these two scenarios during compilation can be achieved by passing compilation parameters as mentioned earlier.
Resources
During project development, a significant portion of runtime bugs are caused by resource issues, such as:
- Artistry changes to the scene GameMode and accidental submissions
- Non-standard resource specifications
- Overwriting someone else’s resources in version control
- Referencing temporary resources
- Referencing resources in the NeverCook directory, causing runtime loss
To avoid these various situations, addressing these problems only when they arise can waste a lot of manpower. Because first, you need to determine whether it’s a program bug or a resource issue, and if it’s the latter, which resource it is, which consumes a lot of extra effort.
To effectively resolve these issues, I previously developed a resource scanning tool to integrate into the project development process. It allows for multi-stage resource checks and minimizes resource issues.
By coordinating four phases: editing, submission, CI/CD, and packaging, it can eliminate resource anomalies at the birth stage!
For more information, please refer to my previous articles:
- Multi-Phase Automation Resource Inspection Scheme in UE
- Automation Practice for Resource Inspection Based on ResScannerUE
- Resource Compliance Inspection Tool ResScannerUE
Efficiency Enhancement
For efficiency improvements during the development stage, I believe they can be divided into the following aspects:
- Development and resource production efficiency
- Issue investigation efficiency
- Debugging efficiency
Development and resource production efficiency can be significantly improved by fully utilizing automation tools and various AI-assisted tools. Especially with the current surge of ChatGPT and Stable Diffusion, utilizing AI tools can quickly extract information and produce content, but it’s essential not to rely entirely on AI as it can still generate incorrect information.
Moreover, during game development, it’s necessary to uncover genuine project pain points; solving these can improve efficiency. For example:
- Rapidly validating code changes without going through the entire packaging process, thereby avoiding waiting times for packaging. This can involve loading
.sofiles externally to realize quick updates of C++ code or providing options for packaging that compiles only the code and basic resources. - Conveniently editing engine startup parameters to avoid manually editing the
ue4commandlet.txtfile on devices. (Efficient Debugging: Launch UE Android App with Command Line Parameters) - Allowing artists, planners, and programmers to package resources locally based on the needs of different modules to quickly verify effects. (UE Resource Hot Update Packaging Tool HotPatcher)
- Archiving rich data for each version, such as which resources were included in the package, type proportions, the amount of redundancy, which resources are non-standard, and recording which norms were triggered, etc., so that when issues arise in game experience, they can be quickly traced back to the relevant resources. (UE Resource Management: Engine Packaging Resource Analysis/Multi-Phase Automation Resource Inspection Scheme in UE)
- Enhancing build efficiency by reducing code and shader compilation time and overall packaging time. (FASTBuild and other Cook optimization strategies)
In my opinion, these are all tasks necessary to apply the DevOps philosophy in the gaming field, which have already been implemented well in our actual projects. Some practices can be found in my previous articles, detailing the implementation ideas and specific features.
However, there’s much more that can be done to improve development efficiency. Many aspects of the UE development process can be optimized through DevOps thinking, and I will write articles later to explore specific areas in-depth.
Enhancing production efficiency, reducing debugging costs, and ensuring quick validation and efficient version iterations are eternal themes of DevOps. Applying scarce human resources to key areas to generate greater value also embodies the essence of cost reduction and efficiency improvement.
