ZeldaMods (Breath of the Wild)

Resource system

Revision as of 04:42, 1 August 2019 by imported>Ginger (Clarified BFRES resource size estimation for Wii U)

In Breath of the Wild, the resource system is a subsystem that is responsible for managing resources, i.e. content files. It is composed of ResourceMgrTask, ResourceSystem and a few ancillary classes (such as "resource binders" and "handles").

Concepts

Resource

Resources (in the context of the codebase) are C++ classes that are responsible for parsing raw data from content files and storing them in a more convenient data structure.

In general, there is one resource class per file type, not format. As an example, bgparamlist and bxml do not share the same class even though both are AAMP files.

In some cases, the resource is nothing more than a thin wrapper over the underlying file bytes. This is the case for BYMLs and other simple resource types that don't require any additional memory allocation. Such a wrapper is (unofficially) called ResourceBase.

Resource factory

Factories are C++ classes that create resource objects.

During application init, factory instances are created and registered with the resource system. When a resource is loaded, the system will look up the corresponding factory based on the file extension, load the entire file into memory, and pass the data to the factory, which then returns a Resource object on which parse() must be called.

List of resource factories

A complete list of resource factories can be found in the rstb project.

These values were extracted from the Switch and Wii U 1.5.0 executables.

Note that:

  • For ResourceLoadArg3 (used in model/bfres related code), the factory is hardcoded to be the ResourceBase factory.
  • For ResourceLoadArg2 (used for actor resources and physics stuff) and ResourceLoadArg (everything else), the factory is determined from the file extension.
  • Any file for which there isn't any specific factory will use the ResourceBase factory. 

Special cases

Compressed files

Compressed files can be automatically decompressed and loaded if their extension starts with an 's'. The 's' prefix indicates that a file is Yaz0 compressed.

When a resource load is requested, the resource system will in most cases automatically prepend 's' to the extension and use sead::ResourceMgr::tryLoadWithDecomp to try loading a compressed version first. This means that any code that interacts with the resource loading functions must not include the prefix in resource paths.

Exceptions: bfevfl, bcamanim and barslist are always loaded uncompressed[1].

Warning: This behaviour can be overridden by passing some flags to the resource load functions.

Archives

The resource system can be configured to try loading from an SARC archive first.

To load from an archive, a pointer to the archive resource handle should be set in ResourceLoadArg::packRes @ 0x58. If it is null, and if ResourceMgrTask's flag 2 is not set[2], packRes will automatically be set to the global resource pack pointer (res::ResourceMgrTask::sInstance->packRes @ this+0x9c06f0 in Switch 1.5.0). Files in the archive can then be accessed as if they were at the root of the romfs/content partition. For example, if EventFlow/Test.bfevfl is a file in Pack/TitleBG.pack, the path that is for loading is EventFlow/Test.bfevfl, not Pack/TitleBG.pack/EventFlow/Test.bfevfl.

Calls to any of the resource loading functions will automatically check whether the specified resource exists in the archive. If it does exist, it will be loaded from the archive. If the file cannot be found in the archive, the game will ignore the active resource pack and load from the usual file device.

Add-on content

In order to load add-on content, the file device must be set to the AoC file device when calling a resource loading function.

This can be done by calling various aocManager functions that return the AoC file device if the resource path matches specific AoC path patterns and assigning the result to field 0x48 in the ResourceLoadArg class.

If no file device is explicitly set, the resource system will use the default file device (romfs on Switch or content directory on Wii U).

If the file device is set to the AoC file device, Aoc/0010/ will be prepended to the canonical resource path. This is done to avoid conflicts between the base and AoC versions of a resource. Warning: the AoC file device is not always set when loading resources from AoC archives. This is the case for the model and ActorParam code. Conversely, the AoC device is sometimes set even when it is unnecessary.

Cases where the AoC file device is used

If the AoC version is >= 0x200 (in most cases) or >= 0x300 (for DLC Pack 2 content), and if the canonical resource name (without the Aoc/0010/ prefix) matches any of the below patterns:

  • Terrain/A/AocField*
  • Any dungeon pack (Pack/%s.pack): shrines, divine beasts, Final Trial (for shrines, the dungeon number must be > 119)
  • UI/StaffRollDLC/*
  • Map/MainField/* (if Pack/AocMainField.pack was loaded successfully)
  • Map/MainFieldDungeon/*
  • Map/AocField/*
  • Map/CDungeon/* (for DungeonNNN with NNN > 119)
  • Physics/StaticCompound/AocField/*
  • Physics/StaticCompound/MainFieldDungeon/*
  • Physics/StaticCompound/CDungeon/* (for DungeonNNN with NNN > 119)
  • Movie/Demo6*
  • Game/AocField/*
  • NavMesh/AocField/*
  • NavMesh/MainFieldDungeon/*
  • NavMesh/CDungeon/* (for DungeonNNN with NNN > 119)
  • Physics/TeraMeshRigidBody/AocField/*
  • Voice/*/Stream_Demo6*/*.bfstm
  • System/AocVersion.txt

There are two other situations where the Aoc/0010/ prefix is supposed to be prepended:

  • If the load file device is set to the Pack/AocMainField.pack archive file device explicitly
  • If the global resource pack is an AoC dungeon pack

However, the first case is irrelevant because that archive only contains Map/MainField/ (a case that is already handled by the path checks) and the second check appears to never pass.

Failed resource load

If a resource loading failure is detected by the resource system, it will try again up to 30 times.[3]

Memory allocation

Heap size

The size of the resource loading heap the system allocates every time a resource is loaded depends on the value that is listed in the #Resource size table (RSTB). If lookup fails, the game will fall back to the following formula (Switch on 1.5.0): 

alignedFileSize = (actualFileSize + 31) & -32;
factory->getResourceSize()
+ factory->constant
+ factory->getLoadDataAlignment()
+ (signed int)(float)(factory->sizeMultiplier * alignedFileSize)
+ (factory->sizeMultiplier * alignedFileSize >= 0.0 &&
   (float)(signed int)(float)(factory->sizeMultiplier * alignedFileSize) != (float)(sizeMultiplier * alignedFileSize))
+ 0x750

Warning: Failure to add entries to the RSTB for a large number of resources may result in system instability. The purpose of this fallback appears to be to allow loading resources during development even without an entry in the resource size table. As such, the system will often allocate way more memory than needed to account for any dynamic allocation that the resource class may do.

Parent heap

If the heap size is smaller than 0x80000 bytes, the heap is created with the "arena for resource S" (small) heap as parent.

Otherwise, the parent heap is the "arena for resource L" (large).

Resource size table

The Resource Size Table (RSTB) contains a list of canonical resource names and their sizes. It is loaded from ResourceSizeTable.product.rsizetable and used to determine how much memory should be allocated when loading a resource.

Resource size calculation

It is currently unknown how Nintendo has determined the values in the RSTB. However by REing the resource system it was found that the resource loading heap is, for most factories, only used to allocate the file loading buffer (which is as large as the file to load), the C++ resource class, extra allocations depending on the resource class and some extra bytes to ensure data is aligned correctly in memory.

Therefore, the following formula should give a correct size value for all resource types: 

(size rounded up to multiple of 32) + CONSTANT + sizeof(ResourceClass) + PARSE_SIZE

CONSTANT is 0x168 in the Switch version and 0xe4 on Wii U.

PARSE_SIZE is the amount of memory allocated from the resource heap in the Resource::parse function. Determining the exact value of PARSE_SIZE requires reversing that function and tracking calls to operator new() because the amount of dynamically allocated memory depends on the resource.

BFRES resource size estimation (Wii U only)

 
RSTB size analysis for all BFRES files. Y-axis: ratio, X-axis: BFRES file size, dot size: RSTB entry size.

Though current tools cannot yet calculate exact RSTB values for BFRES files, by analyzing the BFRES files Nintendo shipped with the game and their RSTB entries, it is possible to make some general observations which permit loose estimates for setting BFRES RSTB values.

There are two types of BFRES file: model and texture. (Texture files are further divided into regular textures, which include Tex1 in the name, and mipmaps, which contain Tex2 in the name, but there seems to be no difference in RSTB calculations concerning these two types of textures.)

 
RSTB size analysis for BFRES model files. Y-axis: ratio, X-axis: BFRES file size, dot size: RSTB entry size.

For models, the ratio of RSTB entry size to BFRES file size increases early, but lazily, as BFRES file size decreases. The estimations are trickier for models than they are for textures, but they are still fairly simple overall.

  • If your model BFRES is less than 100KB, your RSTB entry should be 200% of your BFRES file size.
  • If your model BFRES is greater than 100KB, but less than 1MB, your RSTB entry should be 175% of your BFRES file size.
  • If your model BFRES is greater then 1MB, but less than 2MB, your RSTB entry should be 150% of your BFRES file size.
  • If your model BFRES is greater than 2MB, your RSTB entry should be 125% of your BFRES file size.
 
RSTB size analysis for BFRES tex1 texture files. Y-axis: ratio, X-axis: BFRES file size, dot size: RSTB entry size.

For textures, the ratio of RSTB entry size to BFRES file size increases late, but sharply, as BFRES file size decreases. This makes the estimates for RSTB entry size for textures incredibly easy.

  • If your texture BFRES is less than 100KB, your RSTB entry should be 200% of your BFRES file size.
  • If your texture BFRES is greater than 100KB but less than 500KB, your RSTB entry should be 110% of your BFRES file size.
  • If your texture BFRES is greater than 500KB, your RSTB entry should be 101% of your BFRES file size.
 
RSTB size analysis for BFRES tex2 texture files. Y-axis: ratio, X-axis: BFRES file size, dot size: RSTB entry size.

Note: These analyses were done on the Wii U 1.5.0 version. Switch version calculations will vary.

  1. ↑ See 0x710120A9F4 in Switch 1.5.0
  2. ↑ 0x71012087EC
  3. ↑ 0x710121087C in Switch 1.5.0 (ResourceBinder::retryLoading -- member function name is unofficial)
Retrieved from "https://zeldamods.org/w_botw/index.php?title=Resource_system&oldid=1292"