Resource system and Help:Creating mods: Difference between pages

From ZeldaMods (Breath of the Wild)
(Difference between pages)
Jump to navigation Jump to search
imported>Leoetlino
 
imported>Leoetlino
 
Line 1: Line 1:
In ''Breath of the Wild'', the '''resource system''' is a subsystem that is responsible for managing resources, i.e. content files. It is composed of [[ResourceMgrTask]], [[ResourceSystem]] and a few ancillary classes (such as "resource binders" and "handles").
Welcome, modders! This page is intended to be a collection of useful information for anybody who wants to create mods. If you want to simply use other people's mods, see [[Help:Playing with mods]].


== Concepts ==
== Getting set up ==
=== Resource ===
To make mods, you'll need to start by dumping the unedited game files from your copy of BotW.
Resources (in the context of the codebase) are C++ classes that are responsible for parsing raw data from content files and storing them in a more convenient data structure.
* [[Help:Playing with mods#Wii U/Cemu|Dumping the game files (Wii U)]]
* [[Help:Dumping games|Dumping the game files (Switch)]]


In general, there is one resource class per file type, not format. As an example, [[bgparamlist]] and [[bxml]] do not share the same class even though both are [[AAMP]] files.
== Files, Tools and Tutorials ==
Since BotW uses a lot of Nintendo's own file formats, you'll need tools to be able to edit them. To use leoetlino's tools, you need to install the latest version of Python (64 bit version), and tick the "Add to PATH" / "Add to environment variables" box in the installer. Then, to install the tool, run <code>pip install TOOL_NAME</code> in the command line.


In some cases, the resource is nothing more than a thin wrapper over the underlying file bytes. This is the case for [[BYML]]s and other simple resource types that don't require any additional memory allocation. Such a wrapper is (unofficially) called ResourceBase.
If you're not sure which tool to use on a file, search for its extension on this wiki to find what file format that extension is used for.


=== Resource factory ===
=== Yaz0 compression ===
Factories are C++ classes that create resource objects.
Many files in the game are compressed with [[Yaz0]] compression. Compressed files generally have the letter <code>s</code> prefixed to their file extension. To decompress and recompress these files, use '''wszst''' (download from https://szs.wiimm.de/download.html#vers)
* <code>wszst decompress INPUT_FILE.sbactorpack OUTPUT_FILE.bactorpack</code>
* <code>wszst compress INPUT_FILE.bfres OUTPUT_FILE.sbfres</code>


During application init, factory instances are created and registered with the resource system. When a resource is loaded, the system will look up the corresponding factory based on the file extension, load the entire file into memory, and pass the data to the factory, which then returns a Resource object on which <code>parse()</code> must be called.
=== SARC archives ===
[[SARC]] archives contain collections of other files and folders, like .zip folders. Unpack, edit and re-pack them with leoetlino's [https://pypi.org/project/sarc/ <code>sarc</code> tool]. This tool automatically decompresses Yaz0-encoded archives, and re-compresses them when you repack if the extension starts with an <code>s</code>.
* <code>sarc extract INPUT_FILE.pack</code>
* <code>sarc create [-b] INPUT_FOLDER OUTPUT_FILE.sbactorpack</code> (Use <code>-b</code>for Wii U only).


==== List of resource factories ====
=== BYML files ===
[https://github.com/leoetlino/rstb/blob/master/rstb/resource_factory_info.tsv A complete list of resource factories can be found in the <code>rstb</code> project].
[[BYML]] files contain game parameters. Convert them to an editable format (and back) with leoetlino's [https://pypi.org/project/byml/ <code>byml</code> tool]. This tool automatically decompresses Yaz0-encoded files, and re-compresses them if the extension starts with an <code>s</code>.
* <code>byml_to_yml INPUT_FILE.byml OUTPUT_FILE.yml</code>
* <code>yml_to_byml [-b] INPUT_FILE.yml OUTPUT_FILE.sbyml</code> (Use <code>-b</code> for Wii U only).


These values were extracted from the Switch and Wii U 1.5.0 executables.
=== AAMP files ===
[[AAMP]] files also contain game parameters. Convert them to an editable format (and back) with leoetlino's [https://pypi.org/project/aamp/ <code>aamp</code> tool]. Unlike most other files, AAMP files are exactly the same on both Wii U and Switch.
* <code>aamp_to_yml INPUT_FILE.bxml OUTPUT_FILE.yml</code>
* <code>yml_to_aamp INPUT_FILE.yml OUTPUT_FILE.bgparamlist</code>


Note that:
=== BFRES files ===
* For ResourceLoadArg3 (used in model/bfres related code), the factory is hardcoded to be the ResourceBase factory.
[[BFRES]] files contain the game's models and textures. Tutorials on how to edit these for Wii U can be found in [https://drive.google.com/drive/folders/1Z1_A3w0VvPHp22H_Yh_dSdzI5hr4OAv9 Fooni's Tutorials].
* For ResourceLoadArg2 (used for actor resources and physics stuff) and ResourceLoadArg (everything else), the factory is determined from the file extension.
* <code>.sbfres</code> files contain models.
* Any file for which there isn't any specific factory will use the ResourceBase factory. 
* <code>.Tex1.sbfres</code> files contain textures (Wii U).
* <code>.Tex2.sbfres</code> files contain mipmaps for textures (Wii U). Since we can't edit these on Wii U yet, they need to be disabled.
* <code>.Tex.sbfres</code> files contain both textures and mipmaps (Switch). Unlike on Wii U, mipmaps don't cause any problems here.


== Special cases ==
== Rules ==
=== Compressed files ===
Compressed files can be automatically decompressed and loaded if their extension starts with an 's'. The 's' prefix indicates that a file is [[Yaz0]] compressed.


When a resource load is requested, the resource system will in most cases automatically prepend 's' to the extension and use <code>sead::ResourceMgr::tryLoadWithDecomp</code> to try loading a compressed version first. This means that any code that interacts with the resource loading functions '''must''' not include the prefix in resource paths.
=== The RSTB file ===
The file [[ResourceSizeTable.product.rsizetable]] (referred to as the RSTB file) contains a list of size limits for almost every file in the game (calculated from the non-Yaz0-compressed file sizes). If you edit a file to make its filesize bigger, you'll need to edit this file to prevent errors. For details, see [[Help:Editing the RSTB]].


Exceptions: bfevfl, bcamanim and barslist are always loaded uncompressed (see 0x710120A9F4 in Switch 1.5.0).
=== File names and caching ===
 
The game assumes that any 2 files with the exact same filename also have the exact same contents. If you don't follow this rule when editing files, the game might load the wrong version of the file, leading to errors.
=== Archives ===
The resource system can be configured to try loading from an [[SARC]] archive first.
 
To load from an archive, the global resource pack pointer (<code>res::ResourceMgrTask::sInstance->packRes</code> @ this+0x9c06f0 in Switch 1.5.0) must be set to an archive resource handle. Files in the archive can then be accessed as if they were at the root of the romfs/content partition. For example, if <code>EventFlow/Test.bfevfl</code> is a file in <code>Pack/TitleBG.pack</code>, the path that should be used for loading is <code>EventFlow/Test.bfevfl</code>, not <code>Pack/TitleBG.pack/EventFlow/Test.bfevfl</code>.
 
Calls to any of the resource loading functions will automatically check whether the specified resource exists in the archive. If it does exist, it will be loaded from the archive. If the file cannot be found in the archive, the game will ignore the active resource pack and load from the usual file device.
 
=== Add-on content ===
In order to load add-on content, the file device '''must''' be set to the AoC file device when calling a resource loading function.
 
This can be done by calling various <code>aocManager</code> that return the AoC file device if the resource path matches specific AoC path patterns and assigning the result to field 0x48 in the ResourceLoadArg class.
 
If no file device is explicitly set, the resource system will use the default file device (romfs on Switch or content directory on Wii U).
 
If the file device is set to the AoC file device, <code>Aoc/0010/</code> will be prepended to the [[canonical resource path]]. This is done to avoid conflicts between the base and AoC versions of a resource. '''Warning:''' the AoC file device is not always set when loading resources from AoC archives. This is the case for the model and [[ActorParam]] code. Conversely, the AoC device is sometimes set even when it is unnecessary.
 
==== Cases where the AoC file device is used ====
If the AoC version is >= 0x200 (in most cases) or >= 0x300 (for DLC Pack 2 content), and if the canonical resource name (without the <code>Aoc/0010/</code> prefix) matches any of the below patterns:
* <code>Terrain/A/AocField*</code>
* Any dungeon pack (<code>Pack/%s.pack</code>): shrines, divine beasts, Final Trial (for shrines, the dungeon number must be > 119)
* <code>UI/StaffRollDLC/*</code>
* <code>Map/MainField/*</code> (if Pack/AocMainField.pack was loaded successfully)
* <code>Map/MainFieldDungeon/*</code>
* <code>Map/AocField/*</code>
* <code>Map/CDungeon/*</code> (for DungeonNNN with NNN > 119)
* <code>Physics/StaticCompound/AocField/*</code>
* <code>Physics/StaticCompound/MainFieldDungeon/*</code>
* <code>Physics/StaticCompound/CDungeon/*</code> (for DungeonNNN with NNN > 119)
* <code>Movie/Demo6*</code>
* <code>Game/AocField/*</code>
* <code>NavMesh/AocField/*</code>
* <code>NavMesh/MainFieldDungeon/*</code>
* <code>NavMesh/CDungeon/*</code> (for DungeonNNN with NNN > 119)
* <code>Physics/TeraMeshRigidBody/AocField/*</code>
* <code>Voice/*/Stream_Demo6*/*.bfstm</code>
There are two other situations where the <code>Aoc/0010/</code> prefix is supposed to be prepended:
* If the load file device is set to the <code>Pack/AocMainField.pack</code> archive file device explicitly
* If the global resource pack is an AoC dungeon pack
However, the first case is irrelevant because that archive only contains Map/MainField/ (a case that is already handled by the path checks) and the second check appears to never pass.
 
== Memory allocation ==
=== Heap size ===
The size of the resource loading heap the system allocates every time a resource is loaded depends on the value that is listed in the [[#Resource size table]] (RSTB). If lookup fails, the game will fall back to the following formula (Switch on 1.5.0):
 
<syntaxhighlight lang="C++">
alignedFileSize = (actualFileSize + 31) & -32;
factory->getResourceSize()
+ factory->constant
+ factory->getLoadDataAlignment()
+ (signed int)(float)(factory->sizeMultiplier * alignedFileSize)
+ (factory->sizeMultiplier * alignedFileSize >= 0.0 &&
  (float)(signed int)(float)(factory->sizeMultiplier * alignedFileSize) != (float)(sizeMultiplier * alignedFileSize))
+ 0x750
</syntaxhighlight>
 
'''Warning''': Failure to add entries to the RSTB for a large number of resources may result in system instability. The purpose of this fallback appears to be to allow loading resources during development even without an entry in the resource size table. As such, the system will often allocate way more memory than needed to account for any dynamic allocation that the resource class may do.
 
=== Parent heap ===
If the heap size is smaller than 0x80000 bytes, the heap is created with the "arena for resource S" (small) heap as parent.
 
Otherwise, the parent heap is the "arena for resource L" (large).
 
=== Resource size table ===
The ''Resource Size Table'' (RSTB) contains a list of canonical resource names and their sizes. It is loaded from [[ResourceSizeTable.product.rsizetable]] and used to determine how much memory should be allocated when loading a resource.
 
It is currently unknown how Nintendo has determined the values in the RSTB. However by REing the resource system it was found that the resource loading heap is, for most factories, only used to allocate the file loading buffer (which is as large as the file to load), the C++ resource class, extra allocations depending on the resource class and some extra bytes to ensure data is aligned correctly in memory.
 
Therefore, the following formula should give a correct size value for all resource types:
 
(size rounded up to multiple of 32) + CONSTANT + sizeof(ResourceClass) + PARSE_SIZE
 
CONSTANT is 0x168 in the Switch version and 0xe4 on Wii U.
 
PARSE_SIZE is the amount of memory allocated from the resource heap in the <code>Resource::parse</code> function. Determining the exact value of PARSE_SIZE requires reversing that function and tracking calls to <code>operator new()</code> because the amount of dynamically allocated memory depends on the resource.
 
[[Category:Internals]]
[[Category:Subsystems (BotW)]]

Revision as of 16:05, 15 October 2018

Welcome, modders! This page is intended to be a collection of useful information for anybody who wants to create mods. If you want to simply use other people's mods, see Help:Using mods.

Getting set up

To make mods, you'll need to start by dumping the unedited game files from your copy of BotW.

Files, Tools and Tutorials

Since BotW uses a lot of Nintendo's own file formats, you'll need tools to be able to edit them. To use leoetlino's tools, you need to install the latest version of Python (64 bit version), and tick the "Add to PATH" / "Add to environment variables" box in the installer. Then, to install the tool, run pip install TOOL_NAME in the command line.

If you're not sure which tool to use on a file, search for its extension on this wiki to find what file format that extension is used for.

Yaz0 compression

Many files in the game are compressed with Yaz0 compression. Compressed files generally have the letter s prefixed to their file extension. To decompress and recompress these files, use wszst (download from https://szs.wiimm.de/download.html#vers)

  • wszst decompress INPUT_FILE.sbactorpack OUTPUT_FILE.bactorpack
  • wszst compress INPUT_FILE.bfres OUTPUT_FILE.sbfres

SARC archives

SARC archives contain collections of other files and folders, like .zip folders. Unpack, edit and re-pack them with leoetlino's sarc tool. This tool automatically decompresses Yaz0-encoded archives, and re-compresses them when you repack if the extension starts with an s.

  • sarc extract INPUT_FILE.pack
  • sarc create [-b] INPUT_FOLDER OUTPUT_FILE.sbactorpack (Use -bfor Wii U only).

BYML files

BYML files contain game parameters. Convert them to an editable format (and back) with leoetlino's byml tool. This tool automatically decompresses Yaz0-encoded files, and re-compresses them if the extension starts with an s.

  • byml_to_yml INPUT_FILE.byml OUTPUT_FILE.yml
  • yml_to_byml [-b] INPUT_FILE.yml OUTPUT_FILE.sbyml (Use -b for Wii U only).

AAMP files

AAMP files also contain game parameters. Convert them to an editable format (and back) with leoetlino's aamp tool. Unlike most other files, AAMP files are exactly the same on both Wii U and Switch.

  • aamp_to_yml INPUT_FILE.bxml OUTPUT_FILE.yml
  • yml_to_aamp INPUT_FILE.yml OUTPUT_FILE.bgparamlist

BFRES files

BFRES files contain the game's models and textures. Tutorials on how to edit these for Wii U can be found in Fooni's Tutorials.

  • .sbfres files contain models.
  • .Tex1.sbfres files contain textures (Wii U).
  • .Tex2.sbfres files contain mipmaps for textures (Wii U). Since we can't edit these on Wii U yet, they need to be disabled.
  • .Tex.sbfres files contain both textures and mipmaps (Switch). Unlike on Wii U, mipmaps don't cause any problems here.

Rules

The RSTB file

The file ResourceSizeTable.product.rsizetable (referred to as the RSTB file) contains a list of size limits for almost every file in the game (calculated from the non-Yaz0-compressed file sizes). If you edit a file to make its filesize bigger, you'll need to edit this file to prevent errors. For details, see Help:Editing the RSTB.

File names and caching

The game assumes that any 2 files with the exact same filename also have the exact same contents. If you don't follow this rule when editing files, the game might load the wrong version of the file, leading to errors.