Z's Blog

Integrate ZSTD compression algorithm for UE4 using ModularFeature

ModularFeature：为UE4集成ZSTD压缩算法
Posted on In , , Views:
Symbols count in article: 9.9k Reading time ≈ 25 mins.

UE uses Zlib by default as the compression algorithm for packaging resources, but it is not the best choice in terms of compression ratio and decompression speed. The efficiency comparison of various compression algorithms can be seen from the Squash Compression Benchmark. I chose the Facebook open-source ZStandard to replace Zlib, as ZSTD provides a good compression ratio while also having decent decompression speed. This article will not only explain how to integrate a compression algorithm into UE but will also briefly introduce a modular organization method for some features in UE—ModularFeature. Using this method makes it easy to replace certain functionalities, and the replacement compression algorithm in this article serves as a practical example.

I integrated ZSTD into UE by writing a plugin. The source code is available on GitHub: hxhb/ue-zstd, which supports Android, Windows, iOS, and macOS. Feel free to give it a star.

Introduction

In fact, the compression ratio of ZStandard is similar to that of zlib when considering compression speed (ZSTD’s compression level is around 10), but ZSTD decompresses faster. ZSTD’s compression levels range from 1 to 22, with the default set to 3. In the plugin I developed, the default level is set to 10, which can be specified by using the -zstdlevel= parameter when starting the engine (if you compiled UnrealPak, you can also specify the -zstdlevel= parameter in the project settings under Pacakge - PakFileCompressionCommandlineOptions).

ModularFeature

All ModularFeature implementations in UE must inherit from IModularFeature. It has no members and serves simply as a generic type. The organization of ModularFeature is managed by the IModularFeatures class (with the default implementation being FModularFeatures). This class provides methods to register, unregister, and retrieve instances of IModularFeature by name. Its core functionality is to associate a Name with a set of implementations. For example, all compression algorithms in UE fall under the COMPRESSION_FORMAT_FEATURE_NAME, which corresponds to an array where each element is an implementation of a compression algorithm; this allows us to specify which compression algorithm to use based on our requirements.

1
2
3
4
5
6
// Runtime/Core/Private/Features/ModularFeatures.cpp
void FModularFeatures::RegisterModularFeature( const FName Type, IModularFeature* ModularFeature )
{
  ModularFeaturesMap.AddUnique( Type, ModularFeature );
  ModularFeatureRegisteredEvent.Broadcast( Type, ModularFeature );
}

As seen above, registering a ModularFeature simply involves adding an element to a Map. Note that ModularFeaturesMap is not a regular TMap; it is a TMiltiMap, allowing a key to correspond to multiple values. It is a static data member of FCompression.

The specific process is:

  1. Register the ModularFeature we created with IModularFeatures during the loading of UE’s Module;
  2. When using the feature, we can search for all implementations of a certain ModularFeature based on the name of our ModularFeature category;
  3. We can then call the implemented functionalities of ModularFeature through our own defined interfaces.

Note that IModularFeatures::GetModularFeature is implemented using templates, allowing for direct retrieval of the specific type of a given Feature category.

Specifying Compression Algorithms

First, let’s take a look at how to replace the compression algorithm used during packaging in UE:

Open Project Settings and find Packing - Pak File Compression Format(s):

It is an FString type where you can enter a string separated by commas. If multiple formats are specified, they will be prioritized in order, falling back to other formats in case of errors or unavailability (such as plugins not being enabled).

This string is passed to the UnrealPak as the -compressionformats= parameter.

1
2
3
4
5
6
// Programs/AutomationTool/Scripts/CopyBuildToStagingDirectory.Automation.cs
string CompressionFormats = "";
if (PlatformGameConfig.GetString("/Script/UnrealEd.ProjectPackagingSettings", "PakFileCompressionFormats", out CompressionFormats))
{
  CompressionFormats = " -compressionformats=" + CompressionFormats;
}

In the PakFileUtilities module, the default compression algorithm is ZLib.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
// Developer/PakFileUtilities/Private/PakFileUtilities.cpp
FString DesiredCompressionFormats;
// look for -compressionformats or -compressionformat on the commandline
if (FParse::Value(CmdLine, TEXT("-compressionformats="), DesiredCompressionFormats) || FParse::Value(CmdLine, TEXT("-compressionformat="), DesiredCompressionFormats))
{
  TArray<FString> Formats;
  DesiredCompressionFormats.ParseIntoArray(Formats, TEXT(","));
  for (FString& Format : Formats)
  {
    // look until we have a valid format
    FName FormatName = *Format;
    if (FCompression::IsFormatValid(FormatName))
    {
      CmdLineParameters.CompressionFormats.Add(FormatName);
      break;
    }
  }
}
// make sure we can always fallback to zlib, which is guaranteed to exist
CmdLineParameters.CompressionFormats.AddUnique(NAME_Zlib);

If a different algorithm is specified, the actual compression algorithm will be retrieved via ICompressionFormat* FCompression::GetCompressionFormat(FName FormatName, bool bErrorOnFailure). Here’s its implementation:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
ICompressionFormat* FCompression::GetCompressionFormat(FName FormatName, bool bErrorOnFailure)
{
  ICompressionFormat** ExistingFormat = CompressionFormats.Find(FormatName);
  if (ExistingFormat == nullptr)
  {
    TArray<ICompressionFormat*> Features = IModularFeatures::Get().GetModularFeatureImplementations<ICompressionFormat>(COMPRESSION_FORMAT_FEATURE_NAME);

    for (ICompressionFormat* CompressionFormat : Features)
    {
      // is this format the right one?
      if (CompressionFormat->GetCompressionFormatName() == FormatName)
      {
        // remember it in our format map
        ExistingFormat = &CompressionFormats.Add(FormatName, CompressionFormat);
        break;
      }
    }

    if (ExistingFormat == nullptr)
    {
      if (bErrorOnFailure)
      {
        UE_LOG(LogCompression, Error, TEXT("FCompression::GetCompressionFormat - Unable to find a module or plugin for compression format %s"), *FormatName.ToString());
      }
      else
      {
        UE_LOG(LogCompression, Display, TEXT("FCompression::GetCompressionFormat - Unable to find a module or plugin for compression format %s"), *FormatName.ToString());
      }
      return nullptr;
    }
  }

  return *ExistingFormat;
}

It is seen that it retrieves all FeatureImplementation from IModularFeatures::Get().GetModularFeatureImplementations<ICompressionFormat>. If you want to add your own compression algorithm, you can write a plugin and include it, thus allowing it to be used by name.

As mentioned at the beginning, to add it, you need to register a Feature using IModularFeature::RegisterModularFeature:

1
IModularFeatures::Get().RegisterModularFeature(COMPRESSION_FORMAT_FEATURE_NAME, ICompressionFormatPointer);

Here, COMPRESSION_FORMAT_FEATURE_NAME is a macro that corresponds to a string CompressionFormat, marking a group of Feature types. ICompressionFormatPointer is the pointer to the encapsulation of the compression algorithm you wish to add, and that object needs to inherit from ICompressionFormat.

1
2
3
4
5
6
7
8
9
10
// Runtime/Core/Public/Misc/ICompressionFormat.h
#define COMPRESSION_FORMAT_FEATURE_NAME "CompressionFormat"

struct ICompressionFormat : public IModularFeature, public IModuleInterface
{
  virtual FName GetCompressionFormatName() = 0;
  virtual bool Compress(void* CompressedBuffer, int32& CompressedSize, const void* UncompressedBuffer, int32 UncompressedSize, int32 CompressionData) = 0;
  virtual bool Uncompress(void* UncompressedBuffer, int32& UncompressedSize, const void* CompressedBuffer, int32 CompressedSize, int32 CompressionData) = 0;
  virtual int32 GetCompressedBufferSize(int32 UncompressedSize, int32 CompressionData) = 0;
};

Additionally, in project settings, you can use PakFileCompressionCommandlineOptions to control the parameters for the compression algorithm. This parameter is passed to UnrealPak in CopyBuildToStagingDirectory.Automation.cs. For now, I won’t elaborate on this; it will be discussed later.

Integrating ZSTD

As discussed earlier, integrating a compression algorithm requires implementing a class that inherits from ICompressionFormat and registering it with IModularFeatures. Taking ZSTD as an example, here’s how to add a compression algorithm.

First, let’s review the four interface functions provided in ICompressionFormat and their semantic requirements:

1
2
3
4
5
6
7
8
9
10
11
struct ICompressionFormat : public IModularFeature, public IModuleInterface
{
  // Get the name of the currently implemented compression algorithm
  virtual FName GetCompressionFormatName() = 0;
  // Execute compression
  virtual bool Compress(void* CompressedBuffer, int32& CompressedSize, const void* UncompressedBuffer, int32 UncompressedSize, int32 CompressionData) = 0;
  // Execute decompression
  virtual bool Uncompress(void* UncompressedBuffer, int32& UncompressedSize, const void* CompressedBuffer, int32 CompressedSize, int32 CompressionData) = 0;
  // Get the maximum size of the buffer for compressed data
  virtual int32 GetCompressedBufferSize(int32 UncompressedSize, int32 CompressionData) = 0;
};

Next, let’s proceed to integrate ZSTD. First, pull the code from facebook/zstd and extract it (you can copy everything from the Lib directory except for the dll directory) into the Source/ThirdParty folder of your plugin. Then add its path to PublicIncludePaths in your plugin’s *.build.cs.

You would need to modify ZSTD’s code because the compilation environment and warning levels in UE may not allow the code to compile correctly. Common operations involve ignoring certain warnings, but ZSTD has a specific scenario where its code containing XXHash may conflict with existing definitions in LiveCoding, leading to redefinition errors, so the XXHash code in ZSTD has to be renamed.

Once the ZSTD code compiles successfully in UE, you can move on to creating and implementing the ZSTD ICompressionFormat:

1
2
3
4
5
6
7
8
9
struct FZstdCompressionFormat : public ICompressionFormat
{
  virtual FName GetCompressionFormatName() override;
  virtual bool Compress(void* CompressedBuffer, int32& CompressedSize, const void* UncompressedBuffer, int32 UncompressedSize, int32 CompressionData) override;
  virtual bool Uncompress(void* UncompressedBuffer, int32& UncompressedSize, const void* CompressedBuffer, int32 CompressedSize, int32 CompressionData) override;
  virtual int32 GetCompressedBufferSize(int32 UncompressedSize, int32 CompressionData) override;

  static int32 Level;
};

This merely derives from ICompressionFormat and adds a static member Level to track which compression level to use in ZSTD.

The remaining task is to identify the functions in ZSTD’s code that can implement the semantics of the ICompressionFormat interface:

1
2
3
ZSTDLIB_API size_t ZSTD_compress(void* dst, size_t dstCapacity, const void* src, size_t srcSize, int compressionLevel);
ZSTDLIB_API size_t ZSTD_decompress(void* dst, size_t dstCapacity, const void* src, size_t compressedSize);
ZSTDLIB_API size_t ZSTD_compressBound(size_t srcSize); /*!< maximum compressed size in worst case single-pass scenario */

By examining ZSTD’s code, you can see that these three functions together can implement all functionalities of ICompressionFormat. The implementation is straightforward—all that is required is to forward function calls:
Note: The GetCompressionFormatName function specifies the name of the compression feature, which I set to zstd. This name will be used in project settings.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
FName FZstdCompressionFormat::GetCompressionFormatName()
{
  return TEXT("zstd");
}

bool FZstdCompressionFormat::Compress(void* CompressedBuffer, int32& CompressedSize, const void* UncompressedBuffer, int32 UncompressedSize, int32 CompressionData)
{
  UE_LOG(LogTemp, Log, TEXT("FZstdCompressionFormat::Compress level is %d"), FZstdCompressionFormat::Level);
  int32 Result = ZSTD_compress(CompressedBuffer, CompressedSize, UncompressedBuffer, UncompressedSize, FZstdCompressionFormat::Level);
  if (Result > 0)
  {
    if (Result > GetCompressedBufferSize(UncompressedSize, CompressionData))
    {
      FPlatformMisc::LowLevelOutputDebugStringf(TEXT("%d < %d"), Result, GetCompressedBufferSize(UncompressedSize, CompressionData));
      // we cannot safely go over the BufferSize needed!
      return false;
    }
    CompressedSize = Result;
    return true;
  }
  return false;
}

bool FZstdCompressionFormat::Uncompress(void* UncompressedBuffer, int32& UncompressedSize, const void* CompressedBuffer, int32 CompressedSize, int32 CompressionData)
{
  int32 Result = ZSTD_decompress(UncompressedBuffer, UncompressedSize, CompressedBuffer, CompressedSize);
  if (Result > 0)
  {
    UncompressedSize = Result;
    return true;
  }
  return false;
}

int32 FZstdCompressionFormat::GetCompressedBufferSize(int32 UncompressedSize, int32 CompressionData)
{
  return ZSTD_compressBound(UncompressedSize);
}

At this point, the integration of ZSTD is complete. The final step is to add this feature to IModularFeatures, making it available for the engine to use.

As I created a plugin, I will write the registration logic in the module’s StartupModule, and unregister it in ShutdownModule.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
#define ZSTD_LEVEL_OPTION_STRING TEXT("-ZstdLevel=")
void FlibzstdModule::StartupModule()
{
  FString CommandLine = FCommandLine::Get();
  if (CommandLine.Contains(ZSTD_LEVEL_OPTION_STRING, ESearchCase::IgnoreCase))
  {
    int32 level;
    FParse::Value(FCommandLine::Get(), *FString(ZSTD_LEVEL_OPTION_STRING).ToLower(), level);
    FZstdCompressionFormat::Level = FMath::Clamp(level, ZSTD_minCLevel(), ZSTD_maxCLevel());
  }

  ZstdCompressionFormat = new FZstdCompressionFormat();
  IModularFeatures::Get().RegisterModularFeature(COMPRESSION_FORMAT_FEATURE_NAME, ZstdCompressionFormat);
}

void FlibzstdModule::ShutdownModule()
{
  IModularFeatures::Get().UnregisterModularFeature(COMPRESSION_FORMAT_FEATURE_NAME, ZstdCompressionFormat);
  delete ZstdCompressionFormat;
}

Using the same registration method as before, I also added a command-line parameter when the engine starts, -ZstdLevel=, which can be used to specify the compression level for ZSTD.

To use the ZSTD algorithm for packaging Pak, you can use my HotPatcher to add the -compressionformats=zstd,zlib parameter in UnrealPakOptions:

To check the compression format used by UnrealPak:

Note: Since the engine is not compiled, you cannot directly use UnrealPak.exe to decompress Pak files compressed with ZSTD.

How to Use

If you’re calling the ExecuteUnrealPak function from the PakFileUtilities module directly in the UE editor, you only need to enable this plugin in your project, similar to how I used HotPatcher.

However, when packaging directly in the editor (including using ProjectLauncher), UE launches a separate process and does not load this plugin, which can cause issues. Therefore, I will also discuss the solution.

Runtime Usage

To use zstd at runtime, you can retrieve the zstd object registered in the engine and use its interfaces to call the ICompressionFormat methods.

You can obtain it through FCompression:

1
ICompressionFormat* ZstdCompressionFormat = FCompression::GetCompressionFormat(TEXT("zstd"), true);

Once you have it, you can call Compress, Uncompress, and other interfaces to perform compression and decompression.

Modifying UnrealPak

When we use File - Package Project - PLATFORM in the editor, the process goes through several stages:

  1. Compiling the project and any dependent plugins.
  2. Cooking the resources in the project.
  3. Using UnrealPak.exe to package the cooked resources.
  4. Copying the packaged results.

Since I integrated zstd, it needs to be used during the resource cooking phase, where the Package - PakFileCompressionFormat(s) in project settings takes effect.

However, since the plugin I created is placed in the project, UnrealPak as a separate program won’t load this plugin, which makes it a bit awkward and unusable directly. To resolve this, modifications should be made to UnrealPak, ensuring it loads the libzstd plugin and registers it with IModularFeature on startup. Since UnrealPak belongs to Programs, you will need the source version of the engine for compilation.

The specific approach is as follows:

  1. Place the plugin in the engine’s Engine\Plugins\Runtime directory.
  2. Run GenerateProjectFiles.bat from the engine directory.
  3. Edit the UnrealPak code to add a dependency on the libzstd module and dynamically load the module in the main function:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
// Engine\Source\Programs\UnrealPak\Private\UnrealPak.cpp
#include "UnrealPak.h"
#include "RequiredProgramMainCPPInclude.h"
#include "PakFileUtilities.h"
#include "IPlatformFilePak.h"
#include "libzstd.h"

IMPLEMENT_APPLICATION(UnrealPak, "UnrealPak");

INT32_MAIN_INT32_ARGC_TCHAR_ARGV()
{
  // start up the main loop
  GEngineLoop.PreInit(ArgC, ArgV);

  FlibzstdModule& LibZstdModule = FModuleManager::LoadModuleChecked<FlibzstdModule>(FName("libzstd"));

  double StartTime = FPlatformTime::Seconds();

  int32 Result = ExecuteUnrealPak(FCommandLine::Get()) ? 0 : 1;

  UE_LOG(LogPakFile, Display, TEXT("Unreal pak executed in %f seconds"), FPlatformTime::Seconds() - StartTime);

  GLog->Flush();

  FEngineLoop::AppPreExit();
  FEngineLoop::AppExit();

  return Result;
}

You can also see my modified UnrealPak code here: UnrealPak_422Source.7z.

Then compile UnrealPak in Development mode. The resulting UnrealPak.exe will now support zstd compression/decompression functionality.

You can also place this UnrealPak.exe in the installed version of the engine to use it.

Here’s my compiled version of UnrealPak that supports ZSTD (UE4.22.3): UnrealPak422_ZSTD.

Now you can specify zstd in the project settings under Package - PakFileCompressionFormat(s):

Passing Parameters to zstd

Remember the PakFileCompressionCommandlineOptions in the project settings that I put aside earlier? This parameter can be passed as command-line arguments to UnrealPak.exe. Any content filed here will be forwarded to UnrealPak.exe. Our modified UnrealPak will load the libzstd module on startup, and we can analyze the command line input in the StartupModule of the libzstd module to control various compression parameters (like compression level):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
#define ZSTD_LEVEL_OPTION_STRING TEXT("-ZstdLevel=")
void FlibzstdModule::StartupModule()
{
  FString CommandLine = FCommandLine::Get();
  if (CommandLine.Contains(ZSTD_LEVEL_OPTION_STRING, ESearchCase::IgnoreCase))
  {
    int32 level;
    FParse::Value(FCommandLine::Get(), *FString(ZSTD_LEVEL_OPTION_STRING).ToLower(), level);
    FZstdCompressionFormat::Level = FMath::Clamp(level, ZSTD_minCLevel(), ZSTD_maxCLevel());
  }

  UE_LOG(LogTemp, Log, TEXT("FZstdCompressionFormat::Compress level is %d"), FZstdCompressionFormat::Level);
  ZstdCompressionFormat = new FZstdCompressionFormat();
  IModularFeatures::Get().RegisterModularFeature(COMPRESSION_FORMAT_FEATURE_NAME, ZstdCompressionFormat);
}

This way, you can fill -zstdlevel=22 in the PakFileCompressionCommandlineOptions to control the compression level that you want to use.

Note that Package - PakFileCompressionCommandlineOptions is inactive by default. If you don’t add any other compression algorithms and use the method of obtaining parameters that I described above, any entries here will have no effect.

Final Thoughts

Using this method, you can integrate other compression algorithms on your own, such as lz4, etc. I will write about the compression ratio and performance analysis of ZSTD during packaging when I have more time.

UPDATE

On April 1, 2021, UE announced the integration of RAD’s Oodle compression algorithm suite into UE in both UE4.27 and UE5. While 4.27 hasn’t been released yet, code and link libraries have already been uploaded. I extracted Oodle and fixed some errors in earlier engine versions, allowing it to be used directly in older engine versions.

For detailed content, please refer to:

The usage method can be referenced in this article by specifying -compressionformats=Oodle as the command-line argument, as well as -compresslevel=Leviathan to designate the compression level. Testing shows that a Pak file compressed with zlib and measuring 295M was reduced to 282M using Oodle, while Oodle’s decompression speed surpasses that of zlib.

The article is finished. If you have any questions, please comment and communicate.

Scan the QR code on WeChat and follow me.

Title:Integrate ZSTD compression algorithm for UE4 using ModularFeature
Author:LIPENGZHA
Publish Date:2020/04/20 21:52
Word Count:9.9k Words
Link:https://en.imzlp.com/posts/8470/
License: CC BY-NC-SA 4.0
Reprinting of the full article is prohibited.
Your donation will encourage me to keep creating!