Z's Blog

UE build improvement: optimize the implementation of remote build IOS

UE构建提升：优化远程构建IOS的实现
Posted on Edited on In , Views:
Symbols count in article: 8.3k Reading time ≈ 21 mins.

Remote building is a very convenient way to package iOS in UE, allowing code compilation only on a Mac without COOKing resources, which greatly reduces the hardware requirements for Mac.

Moreover, Macs are quite expensive. In a traditional setup, if multiple independent iOS package build processes are needed, it usually requires purchasing several separate Mac machines for deployment, leading to high hardware costs.

Additionally, the iOS development environment often needs to be updated with iOS system iterations. Every year at WWDC, a new version of iOS is released, requiring support from the latest XCode, which in turn depends on the latest MacOS—a nested chain of dependencies. If the number of Mac build environments increases, updating the build environments of these machines also becomes a cumbersome repetitive task.

Remote building for iOS can effectively solve this issue. A single independent Mac can simultaneously support multiple iOS build processes, and if an update is needed, it only needs to be handled on one machine. Furthermore, the time it takes to compile code is relatively fixed, and incremental compilation can be performed. In theory, as long as not all machines execute a full compilation at the same time, the performance pressure on the Mac won’t be that great.

There are also articles on enabling remote building for iOS in the blog: UE Development Notes: Mac/iOS Edition, which can be referred to for basic configuration.

However, the default implementation of iOS remote building in UE also presents several issues in actual use, failing to achieve a fully automated process from deployment to building. This article aims to optimize the pain points of remote building in real project applications.

RemoteServer Specified Port Number

This is a problem that exists across all UE4 versions. The engine sets the default SSHD port number for RemoteServer to 22, which is indeed the case in most situations. However, for various reasons, the company’s internal network policy does not allow port 22 to be open. If SSH connections are needed, the SSHD must be changed to another port.

In this case, if the RemoteServerName is configured in IP:PORT format in ProjectSettings, it will cause SSHKey lookup failures because UE by default directly concatenates the RemoteServerName to the path.

Therefore, engine support needs to be modified so that the IP:PORT format configuration does not impact SSHKey lookup.

The changed file is the PostInitProperties function in Runtime/IOS/IOSRuntimeSettings/Private/IOSRuntimeSettings.cpp:

This issue has been resolved in UE5. My submitted PR (EpicGames/UnrealEngine/pull/7737) has been merged into UE5, but it still exists in UE4.

Automated Certificate and Provision Import

By default, whether packaging iOS on Windows or Mac, both remote building and direct exporting from Mac require importing the corresponding certificate and provision for the specific bundle ID in the engine. A popup to input the certificate password will also appear:

The entire process is purely manual and very cumbersome. For each machine, the configuration needs to be imported once. If the certificate expires or the provision is updated, all machines must undergo the manual process again, which is torturous.

To address this problem, I investigated the implementation of certificate and provision import in the engine to find a method for complete automation.

The engine contains a standalone program, iPhonePackager, which can import certificates and provisions, as well as install IPA on devices, and perform re-signing of IPA, among other functions. Although it provides a command line, it cannot specify the certificate password, resulting in a popup during execution, similar to the situation when imported in the editor.

So, I modified the code in iPhonePackager to support specifying the certificate password through the command line parameter -cerpassword, avoiding the popup issue.

This allows for silent certificate import through the following command:

1
2
3
4
# Import certificate
IPhonePackager.exe Install Engine -project G:\Client\Game.uproject -certificate G:\Client\Source\ThirdParty\iOS\com.xxxx.yyyy\Development\com.xxxx.yyyy_Development.p12 -cerpassword cerpassword -bundlename com.xxxx.yyyy
# Import provision
IPhonePackager.exe Install Engine -project G:\Client\Game.uproject -provision G:\Client\Source\ThirdParty\iOS\com.xxxx.yyyy\Development\com.xxxx.yyyy_Development.mobileprovision -bundlename com.xxxx.yyyy

However, by default, when compiling the engine, this program is not compiled, making it impossible to use the latest modifications. To avoid this situation, a precompiled iPhonePackager was placed into the project, executing the import through this precompiled version.

So far, silent underlying support for updating certificates and provisions has been implemented, but the timing for importing certificates still needs to be determined.

I chose the timing to write a logic segment in the project’s target.cs to detect the current compilation environment, the path of iPhonePackager, the certificate path, etc., constructing an import command and calling iPhonePackager for execution.

This way, imports can be executed during compilation, and the UE compilation process also conveniently allows passing compilation parameters to UBT, which can be detected in Target.cs and executed as needed.

Additionally, the automatic import feature is best disabled by default and can be enabled via a command line parameter for UBT, allowing automatic certificate import in the pipeline.

The code in Target.cs for parameter detection for certificate import is:

target.cs
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
bool bExportCer = IsContainInCmd("-importcer");
if (Target.Platform == UnrealTargetPlatform.Win64 && bExportCer)
{
    string EngineSourceDirectory = System.IO.Directory.GetCurrentDirectory();  
    string IPhonePackagerExePath = Path.GetFullPath(Path.Combine(EngineSourceDirectory, "..\\Binaries\\DotNET\\IOS/IPhonePackager.exe").Normalize());
    string UProjectFile = Target.ProjectFile.ToString();
    Console.WriteLine("uproject file:" + UProjectFile);
    Console.WriteLine("IPhonePackagerExePath:" + IPhonePackagerExePath);
    if (File.Exists(IPhonePackagerExePath))
    {
        Dictionary<string, string> BundleIdMap = new Dictionary<string, string>();
        BundleIdMap.Add("com.xxx.yyy","password1");
        BundleIdMap.Add("com.xxx.yyy.zzz","password2");
        string[] Configurations = { "Development" };
        string CerRelativePath = "Source/ThirdParty/iOS/";
        
        DirectoryReference ProjectDir = ProjectFile.Directory;
        foreach (KeyValuePair<string, string> BundleIdPair in BundleIdMap)
        {
            string BundleId = BundleIdPair.Key;
            for (int ConfigurationIndex = 0; ConfigurationIndex < Configurations.Length; ConfigurationIndex++)
            {
                string PackageConfiguration = Configurations[ConfigurationIndex];
                string mobileprovision_name = BundleId + "_" + PackageConfiguration + ".mobileprovision";
                string cer_file_name = BundleId + "_" + PackageConfiguration + ".p12";
                string cerPath = Path.Combine(ProjectDir.FullName, CerRelativePath, BundleId, PackageConfiguration, cer_file_name);
                string proversionPath = Path.Combine(ProjectDir.FullName, CerRelativePath, BundleId, PackageConfiguration, mobileprovision_name);

                if (File.Exists(cerPath) && File.Exists(proversionPath))
                {
                    string cerPassword = BundleIdPair.Value;
                    string[] Cmds =
                    {
                        String.Format(
                        "Install Engine -project {0} -certificate {1} -cerpassword {2} -bundlename {3}",
                        UProjectFile, cerPath, cerPassword, BundleId),
                        String.Format(
                        "Install Engine -project {0} -provision {1} -bundlename {2}",
                        UProjectFile, proversionPath, BundleId),
                    };
                    for (int CmdIndex = 0; CmdIndex < Cmds.Length; CmdIndex++)
                    {
                        ProcessStartInfo startInfo = new ProcessStartInfo();
                        startInfo.CreateNoWindow = true;
                        startInfo.UseShellExecute = false;
                        startInfo.FileName = IPhonePackagerExePath;
                        startInfo.Arguments = Cmds[CmdIndex];
                        Console.WriteLine(String.Format("Import Cer&Provision cmd: {0} {1}", startInfo.FileName, startInfo.Arguments));
                        Process.Start(startInfo);
                    }
                }
            }
        }
    }
}

If the -importcer parameter is passed to UBT during compilation:

1
Build\BatchFiles\Build.bat -Target="GameEditor Win64 Development" -Project="G:\Client\Game.uproject" -WaitMutex -importcer

The compilation log will display logs for importing the certificate and provision:

1
2
3
4
5
6
7
2023-08-15 20:13:09:055 : AssemblyDir: G:\UE4_27\Engine
2023-08-15 20:13:09:055 : EngineAssemblyFileName: G:\UE4_27\Engine\Intermediate\Build\BuildRules\UE4Rules.dll
2023-08-15 20:13:09:205 : ProjectDir:G:\Client
2023-08-15 20:13:09:205 : uproject file:G:\Client\FGame.uproject
2023-08-15 20:13:09:205 : IPhonePackagerExePath:G:\Client\Source\ThirdParty\iOS\iPhonePackager\IPhonePackager.exe
2023-08-15 20:13:09:205 : Import Cer&Provision cmd: G:\Client\Source\ThirdParty\iOS\iPhonePackager\IPhonePackager.exe Install Engine -project G:\Client\FGame.uproject -certificate G:\Client\Source\ThirdParty\iOS\com.xxx.yyy\Development\com.xxx.yyy_Development.p12 -cerpassword keystore -bundlename com.xxx.yyy
2023-08-15 20:13:09:256 : Import Cer&Provision cmd: G:\Client\Source\ThirdParty\iOS\iPhonePackager\IPhonePackager.exe Install Engine -project G:\Client\FGame.uproject -provision G:\Client\Source\ThirdParty\iOS\com.xxx.yyy\Development\com.xxx.yyy_Development.mobileprovision -bundlename com.xxx.yyy

New Machine SSH Connection Fingerprint Verification

Since remote building for iOS is essentially initiated from Windows, commands are sent via SSH and rsync, transferring engine and code files, and then pulling back compilation results through rsync.

Key point: SSH Connection.

When an SSH Client connects to a new IP for the first time, it triggers a security mechanism that gives a prompt and waits for the user to input yes:

When you connect to a new server, it asks whether you trust this new server and wish to add it to the known_hosts file. If you input yes, SSH will add the server’s public key to your local machine’s known_hosts file. Thus, on subsequent connections to that server, SSH can compare the public key provided by the server with the public key in the known_hosts file to ensure you’re connecting to the same server, preventing man-in-the-middle attacks.

However, this issue creates a configuration nightmare when using iOS remote building:

  1. If the RemoteServerIP in the project is modified, every PC involved in the connection must manually execute an SSH connection and allow it.
  2. When a new PC machine is deployed and needs to use remote building, the same manual SSH connection must be performed.

If no configuration is made, the engine uses -BatchMode=yes by default for SSH connections, and if user input is required, it will directly lead to build failures! I have previously analyzed this issue; see: Remote Building SSH Connection Error

The entire process is exceptionally cumbersome, contradicting our goal of achieving fully automated builds.

Similarly, to resolve this issue, modifications to the engine’s implementation are needed to allow it to ignore fingerprint verification when using SSH and rsync connections.

The code change is in the ToolChain/RemoteMac.cs, modifying the SSH connection parameters:

SSH connection parameters

rsync connection parameters

Once changes are made, finally, peace returns to the world…

UPL Overwrites plist Modifications

When using remote building for iOS, stub files are generated on the Mac for synthesizing the final IPA on Windows.

The logic we write in UPL will be reflected in the stub generated on the Mac, but a Info.plist file will also be generated in StagedBuilds/IOS on Windows, which contains template information:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
  <dict>
    <key>CFBundleURLTypes</key>
    <array>
      <dict>
        <key>CFBundleURLName</key>
        <string>com.Epic.Unreal</string>
        <key>CFBundleURLSchemes</key>
        <array>
          <string>FGame</string>
        </array>
      </dict>
    </array>
    <!-- .... -->
  </dict>
</plist>

When using iPhonePackager to package the stub generated on the Mac with resources into the final IPA, the Info.plist in the stub is merged with StagedBuilds/IOS/Info.plist.

The command executed by iPhonePackager is:

1
D:\UE4.27\Engine\Binaries\DotNET\IOS\IPhonePackager.exe RepackageFromStage "D:\Client\GWorld.uproject" -config Development -schemename GWorld -schemeconfig "Development" -targetname GWorld -sign -codebased -stagedir "D:\Client\Saved\StagedBuilds\IOS" -project "D:\Client\GWorld.uproject" -provision "com.imzlp.gworld_Development.mobileprovision" -certificate "iPhone Developer: Created via API (XXXX)"

In the Programs/IOS/iPhonePackager/CookTime.cs‘s RepackageIPAFromStub function, the merging process is implemented: iPhonePackager/CookTime.cs#L226-L258

During merging, the following LOG appears:

1
2
3
4
5
6
7
8
9
10
----------
Executing command 'Clean' ...
Cleaning temporary files from PC ...
 ... cleaning: D:\Client\Intermediate\IOS-Deploy\GWorld\Development\
2023-07-10 14:33:45:817 :   
Loaded stub IPA from 'D:\Client\Binaries\IOS\GWorld.stub' ...
 ... 'D:\Client\Binaries\IOS\GWorld.stub' -> 'D:\Client\Binaries\IOS\GWorld.ipa'
     Copy: D:\Client\Binaries\IOS\GWorld.stub -> D:\Client\Binaries\IOS\GWorld.ipa, last modified at 2023/7/10 12:26:52
Found Info.plist (D:\Client\Saved\StagedBuilds\IOS\Info.plist) in stage, which will be merged in with stub plist contents
...

The engine’s implementation of plist merging occurs in such a way that the plist from StagedBuilds is merged into the stub’s plist (note this sequence).

If elements in the StagedBuilds plist already exist in the stub’s plist, the elements in the stub’s plist will be removed first, and then elements from the StagedBuilds plist will be added to the resulting stub’s plist.

The implementation of MergePlistIn is found here: iPhonePackager/Utilities.cs#L399

If we modify the values of existing elements in the StagedBuilds/IOS/Info.plist via UPL, under this logic, our modified values will always be overridden by the default values, leading to inconsistencies between the final IPA and the values in the stub’s plist.

This should be regarded as a bug in the engine; the fix is to let the plist from StagedBuilds merge into the stub’s plist, effectively reversing the aforementioned process.

CookTime.cs
1
2
3
4
5
6
7
8
9
10
11
12
// Merge the two plists, using the staged one as the authority when they conflict
byte[] StagePListBytes = File.ReadAllBytes(PossiblePList);
string StageInfoString = Encoding.UTF8.GetString(StagePListBytes);

byte[] StubPListBytes = FileSystem.ReadAllBytes("Info.plist");

//++[lipengzha] Fix the problem that the modification of plist in UPL is overwritten 
Utilities.PListHelper StageInfo = new Utilities.PListHelper(StageInfoString);
StageInfo.MergePlistIn(Encoding.UTF8.GetString(StubPListBytes));
// Write it back to the cloned stub, where it will be used for all subsequent actions
byte[] MergedPListBytes = Encoding.UTF8.GetBytes(StageInfo.SaveToString());
//--[lipengzha]

Also, compiling the engine separately via UBT will not include the iPhonePackager program, so it’s necessary to execute a separate compilation for it or create a script to compile all tools each time the engine is compiled, which is more secure. Ultimately, a precompiled iPhonePackager can also be submitted to the engine.

Modifying Remote Build Upload Path

In UE4.26 and earlier versions, uploads are defaulted to the ~ directory, which is hardcoded in RemoteMac.cs and cannot be configured:

1
2
3
4
5
6
7
8
// 4.27 or before
// Get the remote base directory
StringBuilder Output;
if(ExecuteAndCaptureOutput("'echo ~'", out Output) != 0)
{
    throw new BuildException("Unable to determine home directory for remote user. SSH output:\n{0}", StringUtils.Indent(Output.ToString(), "  "));
}
RemoteBaseDir = String.Format("{0}/UE4/Builds/{1}", Output.ToString().Trim().TrimEnd('/'), Environment.MachineName);

The output of echo ~ is the absolute path of the current ~.

In UE4.27, the engine introduced a configuration option that allows the RemoteServerOverrideBuildPath to be set in IOSRuntimeSettings:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
// 4.27 or later
// Get the remote base directory
string RemoteServerOverrideBuildPath;
if (Ini.GetString("/Script/IOSRuntimeSettings.IOSRuntimeSettings", "RemoteServerOverrideBuildPath", out RemoteServerOverrideBuildPath) && !String.IsNullOrEmpty(RemoteServerOverrideBuildPath))
{
    RemoteBaseDir = String.Format("{0}/{1}", RemoteServerOverrideBuildPath.Trim().TrimEnd('/'), Environment.MachineName);
}
else
{
    StringBuilder Output;
    if (ExecuteAndCaptureOutput("'echo ~'", out Output) != 0)
    {
        throw new BuildException("Unable to determine home directory for remote user. SSH output:\n{0}", StringUtils.Indent(Output.ToString(), "  "));
    }
    RemoteBaseDir = String.Format("{0}/UE4/Builds/{1}", Output.ToString().Trim().TrimEnd('/'), Environment.MachineName);
}

If configured, it will prioritize that; if not, it will retrieve it using echo ~.

If there is insufficient disk space on the default upload path, this method can be used to change the remote build upload path to another disk.

The article is finished. If you have any questions, please comment and communicate.

Scan the QR code on WeChat and follow me.

Title:UE build improvement: optimize the implementation of remote build IOS
Author:LIPENGZHA
Publish Date:2023/08/16 10:43
Update Date:2023/09/05 20:11
Word Count:8.3k Words
Link:https://en.imzlp.com/posts/50293/
License: CC BY-NC-SA 4.0
Reprinting of the full article is prohibited.
Your donation will encourage me to keep creating!