Project:Help wanted: Difference between revisions

From ZeldaMods (Breath of the Wild)
Jump to navigation Jump to search
imported>Leoetlino
imported>Leoetlino
(→‎Game logic: One-hit kill protection)
 
(42 intermediate revisions by the same user not shown)
Line 1: Line 1:
Many secrets still remain to be uncovered. If you have experience with reverse engineering or if you are looking for a fun project to start with RE (or both!), please help us investigate the following mysteries.
Many secrets still remain to be uncovered. If you have experience with reverse engineering or if you are looking for a fun project to start with RE (or both!), please investigate the following mysteries or help with the following tasks.
 
== Reverse engineering tips ==
{{expand section}}
=== GameData flags ===
GameData flags can be read and set from many different systems:
* '''Event flows'''. A very convenient way to determine if a flag is being set in an event flow (or queried, passed to the executable, ...) is to grep [https://github.com/MrCheeze/botw-tools/tree/master/event MrCheeze/botw-tools:event] for the flag name.
* '''Map units'''. LinkTags can be used to conditionally spawn map objects based on GameData flags (which are called SaveFlags in map units) and also to set them. It is extremely common for Area tags to be used for triggering LinkTags (which in turn write to GameData flags).
* '''Executable'''. The executable obviously also reads and writes to game data flags. Do note that the name of the flag is not always hardcoded, and may instead come from a configuration file (example: the Defeated counters for the [[difficulty scaling]] system).
 
The GameData system is configured by bgdata files in [[Content/GameData]]. These files determine flags' initial values and their minimum/maximum values.
 
=== Resource locations ===
The following lists of game resources can be used to find where a resource is stored in the [[content]] files:
* https://github.com/leoetlino/botw-re-notes/blob/master/game_files/wiiu_rstb_150.csv
* https://github.com/leoetlino/botw-re-notes/blob/master/game_files/switch_rstb_150.csv
 
=== Actor names ===
Actors have an accompanying descriptor ([[ActorLink]]) that contains the name of the actor in Japanese. This name is usually more descriptive and easier to understand than the actor name.
 
=== Cutscenes ===
A cutscene can be identified via several methods:
* If the cutscene has text dialogue, looking at [https://github.com/MrCheeze/botw-tools/blob/master/text localisation strings] is often the best way to find out what a cutscene is used for.
* Look at the [[Demo]] list.
* Look at the associated event flow with the [https://eventviewer.zeldamods.org/ Event Viewer], [[help:Tools/EventEditor|EventEditor]] or [https://github.com/MrCheeze/botw-tools/tree/master/event MrCheeze's text dump of events].
* Listen to the demo's soundtrack (in Sound).


== Formats ==
== Formats ==
''The following tasks require reversing [[:Category:file formats|file formats]]. It is strongly recommended to also examine the executable to have more reliable information about formats and figure out how they are used by Nintendo ''exactly''.''
''The following tasks require reversing [[:Category:file formats|file formats]]. It is strongly recommended to also examine the executable to gain more reliable knowledge about formats and figure out how they are used by Nintendo ''exactly''.''


=== bfevtm ===
=== XLink databases ===
[[bfevtm]] files (binary event timelines) are extremely similar to [[bfevfl]] files (binary event flowcharts), but have a few significant differences.  
To make the development process more efficient, Nintendo introduced a system called SoundLink (aka SLink)<ref>[https://game.watch.impress.co.jp/docs/news/1078827.html BotW CEDEC 2017 talk]</ref><ref>0x7101E271AE in Switch 1.5.0</ref> (and its counterpart for effects: EffectLink<ref>0x7101E3790B in Switch 1.5.0</ref>) during development. Both systems were at some point unified and are now handled by a single xlink2 library which handles both the SLink and ELink systems.


Because timelines are exclusively used by the event manager &mdash; unlike flowcharts which are also used by smaller subsystems like TipsMgr &mdash; for [[demo]]s, it has proven difficult to find the code that reads timelines.
Two files are used to map effect/sound names to the actual resources. In ''Breath of the Wild'', they are called [[ELink2DB.belnk]] and [[SLink2DB.bslnk]] (found in ELink2 and SLink2 respectively).


If you are willing to take up this task, please ping [[User:leoetlino|leoetlino]] for the Switch 1.5.0 IDC.
The format has not seen a lot of research as of September 2018, despite the executable being full of debug strings that include field names and the fact that other recent Nintendo games such as ''Splatoon 2'' ship the library with debugging symbols.


=== Havok packfiles and structures ===
=== Havok packfiles and structures ===
Nintendo uses Havok binary packfiles for all Havok resources. Thus, it is difficult to manipulate physics files.
Nintendo uses Havok binary packfiles for all Havok resources. Thus, it is difficult to manipulate physics files.


[[User:leoetlino|leoetlino]] discovered that the Havok library fortunately relies on reflection data to load resources. This makes it possible to parse the binaries and reconstruct a human readable representation with ''every single field name and even enum definitions''. Reflection information has been [https://github.com/leoetlino/botw-re-notes/blob/master/tools/havok_reflection_info.json dumped for the Switch version].
Fortunately the Havok library relies on reflection data to load resources. This makes it possible to parse the binaries and reconstruct a human readable representation with ''every single field name and even enum definitions''. Reflection information has been [https://github.com/leoetlino/botw-re-notes/blob/master/tools/havok_reflection_info.json dumped for the Switch version].


The following tasks remain to be done:
The following tasks remain to be done:
Line 20: Line 45:
* Dump reflection data for the Wii U version. Data offsets are different since the Switch uses a different compiler, is a 64 bit little-endian platform unlike the Wii U (32-bit big endian machine).
* Dump reflection data for the Wii U version. Data offsets are different since the Switch uses a different compiler, is a 64 bit little-endian platform unlike the Wii U (32-bit big endian machine).
* Write a library to manipulate binary packfiles: getting, modifying, adding and removing Havok objects from a packfile (and perhaps converting them to XML too?)
* Write a library to manipulate binary packfiles: getting, modifying, adding and removing Havok objects from a packfile (and perhaps converting them to XML too?)
* Figure out how the compression for <code>hkpBvCompressedMeshShape</code>s works. Nintendo uses compressed mesh shapes for all [[Content/Physics/StaticCompound|static compound]] collision files.
* Document how the compression for <code>hkpBvCompressedMeshShape</code>s works. Nintendo uses compressed mesh shapes for all [[Content/Physics/StaticCompound|static compound]] collision files.
 
=== <s>Event timelines</s> ===
[[bfevtm]] files (binary event timelines) are extremely similar to [[bfevfl]] files (binary event flowcharts), but have a few significant differences that make them incompatible with the [https://github.com/leoetlino/evfl evfl] library as of September 2018.
 
Because timelines are exclusively used by the event manager &mdash; unlike flowcharts which are also used by smaller subsystems like TipsMgr &mdash; for [[demo]]s, it has proven difficult to find the code that reads timelines.
 
If you are willing to take up this task, please ping [[User:leoetlino|leoetlino]] for the Switch 1.5.0 IDC.
 
'''Update''' (16/12/2018): All timeline-related structures are now known thanks to Nintendo shipping the EventFlow library with full symbols in another game. Some fields are still unknown at the moment, though (see [https://github.com/leoetlino/bfevfl-template/blob/master/timeline.bt leoetlino/bfevfl-template:timeline.bt] and [https://github.com/leoetlino/bfevfl-template/blob/master/actor.bt leoetlino/bfevfl-template:actor.bt] for more details).
 
'''Update''' (19/12/2018): [https://github.com/leoetlino/evfl evfl v0.13.0] now supports event timelines.


== Game logic ==
== Game logic ==
''The following tasks require reversing the executable to have reliable information on how the mechanics work.''
''The following tasks require reversing the executable or some serious experiments to have reliable information on how the mechanics work.''
 
=== EnvPalettes and areas ===
The [[WorldMgr]] configuration file, [[normal.bwinfo]], has a set of 207 environment-related settings called "env palettes". The link between palette numbers and maps/areas/shrines is currently unknown.
 
=== The Master Cycle Zero's speed ===
How is the motorcycle's speed configured? It's not in GeneralParamList, so it might be physics-based or even hardcoded.
 
=== AutoPlacement ===
How often does AutoPlacement code run? RailDistance? Does it check the camera field of view to decide whether to spawn dragons or not?
 
Which positions are being considered for spawning actors?
 
=== Glitches ===
==== Horse sliding ====
How does this glitch work? How is the sliding direction determined?
 
==== Apparatus Storage ====
How does this glitch work? Is there anything else that could trigger a similar state?
 
==== Lizalfos physics glitch ====
How does this glitch work? Why does the game crash?
 
=== <s>amiibo drops</s> ===
{{lta|link=amiibo drops}}


=== amiibo drops ===
* How scaling (weapon upgrades) works for amiibo weapons.
* How scaling (weapon upgrades) works for amiibo weapons.
*: amiibo weapons don't entirely follow the regular weapon scaling and bonus rules.
*: amiibo weapons don't entirely follow the regular weapon scaling and bonus rules.
Line 31: Line 90:
*: amiibo [[ActorParam/DropTable|DropTable]]s generally contain 8 tables: Normal, SmallHit, BigHit({Normal,Parasail,Remain}), and GreatHit({Normal,Parasail,Remain}). Parasail tables are presumably selected after the player receives the paraglider, and Remain after Link completes a divine beast; however this is pretty much only speculation. It is also currently unknown what determines which xxxxxHit table gets used.
*: amiibo [[ActorParam/DropTable|DropTable]]s generally contain 8 tables: Normal, SmallHit, BigHit({Normal,Parasail,Remain}), and GreatHit({Normal,Parasail,Remain}). Parasail tables are presumably selected after the player receives the paraglider, and Remain after Link completes a divine beast; however this is pretty much only speculation. It is also currently unknown what determines which xxxxxHit table gets used.


=== One-Hit kill protection ===
=== <s>The Blood Moon</s> ===
{{lta|link=Blood moon}}
 
How do Blood Moons work? It is now well known that scheduled Blood Moons are based on a timer, and that (surprise!) [[WorldMgr]] is the thing that is responsible for keeping track of the schedule. However, what are the exact conditions that stop this timer?
 
(Panic Blood Moons are well understood now.)
 
=== <s>Respawning logic</s> ===
{{lta|link=Object respawning}}
 
While the respawn logic for most objects is very simple, some entities do not ''appear'' to be affected by Blood Moons: fruits on trees, shop items. Investigate GameDataMgr as this is likely where the logic is.
 
=== <s>One-hit kill protection</s> ===
It is well known that Link cannot die in one-hit under specific cases. What are the exact conditions for this protection to apply? Interestingly, it appears this mechanic is gone in Master Mode.
It is well known that Link cannot die in one-hit under specific cases. What are the exact conditions for this protection to apply? Interestingly, it appears this mechanic is gone in Master Mode.


<code>NoDeathDamageBase</code> (in Link's [[ActorParam/GeneralParamList|GeneralParamList]]) might be related.
<code>NoDeathDamageBase</code> (in Link's [[ActorParam/GeneralParamList|GeneralParamList]]) might be related.
== Resource sizes ==
''Investigating this requires reversing the [[resource system]] (in the executable) and possibly patching it.''
The [[resource size table]] (RSTB) determines how much memory should be allocated to a given resource (i.e. the size of the resource heap). Unfortunately, the sizes that are listed in there don't quite match the actual file size.
In June 2018, it was discovered that the RSTB sizes include internal memory consumption by the C++ resource classes (see [[Resource system]] for an explanation about the concepts), some of which further dynamically allocate memory from the resource heap! Taking the resource object sizes into account is easy, since they are constant. What's not so easy is determining the amount of memory that will be dynamically allocated for ''complex'' resources such as [[bfres]] and most [[AAMP]] files.
A way to guarantee RSTB sizes are large enough is to reverse each resource class, tracking calls to memory allocation functions. However, this is ''extremely'' time consuming given the sheer amount of resource classes in this game and the fact that the same thing would have to be done for both Switch and Wii U.
The current theory is that either the RSTB is machine generated by special versions of the game, or Nintendo is only calculating approximate file sizes.
A possible way to check whether the listed size is exact or approximate is to dump the memory of the game process and investigate resource heap usage.
In any case, figuring out how to calculate sizes correctly for complex resources or rewriting parts of the resource system to remove the fixed heap size limitation would be extremely useful.
== Executable analysis ==
Everything in ''Breath of the Wild'' is implemented in the executable. Some aspects can be modified by editing assets; other things are hardcoded, meaning it's impossible to change how the game works by merely editing content files.
Analysing the executable is the only way to understand how things are implemented internally and how the game actually works. Eventually, after enough information has been collected, it becomes possible to predict how something will behave exactly and also to directly edit the game core. With a more complete understanding, more in-depth modifications can be done by hooking into functions and injecting custom code.
This is similar to the process of creating cheat codes, but different in that this relies on static analysis more and requires understanding how all the different subsystems interact.
Because of the sheer size of the executable, it is infeasible for this long-term project to be completed by a single person. If you are willing to help, please ping [[User:leoetlino|leoetlino]] for the Switch 1.5.0 IDC.  


== Debug leftovers ==
== Debug leftovers ==
Line 41: Line 136:
=== Demo mode ===
=== Demo mode ===
Setting the ROM type to demo ROM types (found in the [[executable]]) make the game crash. What is locked behind them?
Setting the ROM type to demo ROM types (found in the [[executable]]) make the game crash. What is locked behind them?
=== Stage select screen ===
The release version has a stage select screen left (see [[Executable#ErrorViewer_and_Stage_Select]] for more details). It should be possible to get it to work.


=== Actor debug utilities ===
=== Actor debug utilities ===
In a similar fashion, Nintendo has also left an entire [[ActorDebug]] subsystem in all release versions, which can print information to the screen and interact with many different actor system components and things like map units.
In a similar fashion, Nintendo has also left an entire [[ActorDebug]] subsystem in all release versions, which can print information to the screen and interact with many different actor system components and things like map units.
=== <s>Stage select screen</s> ===
{{lta|link=StageSelect}}
The release version has a stage select screen left (see [[Executable#ErrorViewer_and_Stage_Select]] for more details). <s>It should be possible to get it to work.</s>

Latest revision as of 00:48, 12 January 2020

Many secrets still remain to be uncovered. If you have experience with reverse engineering or if you are looking for a fun project to start with RE (or both!), please investigate the following mysteries or help with the following tasks.

Reverse engineering tips

GameData flags

GameData flags can be read and set from many different systems:

  • Event flows. A very convenient way to determine if a flag is being set in an event flow (or queried, passed to the executable, ...) is to grep MrCheeze/botw-tools:event for the flag name.
  • Map units. LinkTags can be used to conditionally spawn map objects based on GameData flags (which are called SaveFlags in map units) and also to set them. It is extremely common for Area tags to be used for triggering LinkTags (which in turn write to GameData flags).
  • Executable. The executable obviously also reads and writes to game data flags. Do note that the name of the flag is not always hardcoded, and may instead come from a configuration file (example: the Defeated counters for the difficulty scaling system).

The GameData system is configured by bgdata files in Content/GameData. These files determine flags' initial values and their minimum/maximum values.

Resource locations

The following lists of game resources can be used to find where a resource is stored in the content files:

Actor names

Actors have an accompanying descriptor (bxml) that contains the name of the actor in Japanese. This name is usually more descriptive and easier to understand than the actor name.

Cutscenes

A cutscene can be identified via several methods:

Formats

The following tasks require reversing file formats. It is strongly recommended to also examine the executable to gain more reliable knowledge about formats and figure out how they are used by Nintendo exactly.

XLink databases

To make the development process more efficient, Nintendo introduced a system called SoundLink (aka SLink)[1][2] (and its counterpart for effects: EffectLink[3]) during development. Both systems were at some point unified and are now handled by a single xlink2 library which handles both the SLink and ELink systems.

Two files are used to map effect/sound names to the actual resources. In Breath of the Wild, they are called ELink2DB.belnk and SLink2DB.bslnk (found in ELink2 and SLink2 respectively).

The format has not seen a lot of research as of September 2018, despite the executable being full of debug strings that include field names and the fact that other recent Nintendo games such as Splatoon 2 ship the library with debugging symbols.

Havok packfiles and structures

Nintendo uses Havok binary packfiles for all Havok resources. Thus, it is difficult to manipulate physics files.

Fortunately the Havok library relies on reflection data to load resources. This makes it possible to parse the binaries and reconstruct a human readable representation with every single field name and even enum definitions. Reflection information has been dumped for the Switch version.

The following tasks remain to be done:

  • Dump reflection data for the Wii U version. Data offsets are different since the Switch uses a different compiler, is a 64 bit little-endian platform unlike the Wii U (32-bit big endian machine).
  • Write a library to manipulate binary packfiles: getting, modifying, adding and removing Havok objects from a packfile (and perhaps converting them to XML too?)
  • Document how the compression for hkpBvCompressedMeshShapes works. Nintendo uses compressed mesh shapes for all static compound collision files.

Event timelines

bfevtm files (binary event timelines) are extremely similar to bfevfl files (binary event flowcharts), but have a few significant differences that make them incompatible with the evfl library as of September 2018.

Because timelines are exclusively used by the event manager — unlike flowcharts which are also used by smaller subsystems like TipsMgr — for demos, it has proven difficult to find the code that reads timelines.

If you are willing to take up this task, please ping leoetlino for the Switch 1.5.0 IDC.

Update (16/12/2018): All timeline-related structures are now known thanks to Nintendo shipping the EventFlow library with full symbols in another game. Some fields are still unknown at the moment, though (see leoetlino/bfevfl-template:timeline.bt and leoetlino/bfevfl-template:actor.bt for more details).

Update (19/12/2018): evfl v0.13.0 now supports event timelines.

Game logic

The following tasks require reversing the executable or some serious experiments to have reliable information on how the mechanics work.

EnvPalettes and areas

The WorldMgr configuration file, normal.bwinfo, has a set of 207 environment-related settings called "env palettes". The link between palette numbers and maps/areas/shrines is currently unknown.

The Master Cycle Zero's speed

How is the motorcycle's speed configured? It's not in GeneralParamList, so it might be physics-based or even hardcoded.

AutoPlacement

How often does AutoPlacement code run? RailDistance? Does it check the camera field of view to decide whether to spawn dragons or not?

Which positions are being considered for spawning actors?

Glitches

Horse sliding

How does this glitch work? How is the sliding direction determined?

Apparatus Storage

How does this glitch work? Is there anything else that could trigger a similar state?

Lizalfos physics glitch

How does this glitch work? Why does the game crash?

amiibo drops

Full article: amiibo drops

  • How scaling (weapon upgrades) works for amiibo weapons.
    amiibo weapons don't entirely follow the regular weapon scaling and bonus rules.
  • What determines the drop table list that gets used?
    amiibo DropTables generally contain 8 tables: Normal, SmallHit, BigHit({Normal,Parasail,Remain}), and GreatHit({Normal,Parasail,Remain}). Parasail tables are presumably selected after the player receives the paraglider, and Remain after Link completes a divine beast; however this is pretty much only speculation. It is also currently unknown what determines which xxxxxHit table gets used.

The Blood Moon

Full article: Blood moon

How do Blood Moons work? It is now well known that scheduled Blood Moons are based on a timer, and that (surprise!) WorldMgr is the thing that is responsible for keeping track of the schedule. However, what are the exact conditions that stop this timer?

(Panic Blood Moons are well understood now.)

Respawning logic

Full article: Object respawning

While the respawn logic for most objects is very simple, some entities do not appear to be affected by Blood Moons: fruits on trees, shop items. Investigate GameDataMgr as this is likely where the logic is.

One-hit kill protection

It is well known that Link cannot die in one-hit under specific cases. What are the exact conditions for this protection to apply? Interestingly, it appears this mechanic is gone in Master Mode.

NoDeathDamageBase (in Link's bgparamlist) might be related.

Resource sizes

Investigating this requires reversing the resource system (in the executable) and possibly patching it.

The resource size table (RSTB) determines how much memory should be allocated to a given resource (i.e. the size of the resource heap). Unfortunately, the sizes that are listed in there don't quite match the actual file size.

In June 2018, it was discovered that the RSTB sizes include internal memory consumption by the C++ resource classes (see Resource system for an explanation about the concepts), some of which further dynamically allocate memory from the resource heap! Taking the resource object sizes into account is easy, since they are constant. What's not so easy is determining the amount of memory that will be dynamically allocated for complex resources such as bfres and most AAMP files.

A way to guarantee RSTB sizes are large enough is to reverse each resource class, tracking calls to memory allocation functions. However, this is extremely time consuming given the sheer amount of resource classes in this game and the fact that the same thing would have to be done for both Switch and Wii U.

The current theory is that either the RSTB is machine generated by special versions of the game, or Nintendo is only calculating approximate file sizes.

A possible way to check whether the listed size is exact or approximate is to dump the memory of the game process and investigate resource heap usage.

In any case, figuring out how to calculate sizes correctly for complex resources or rewriting parts of the resource system to remove the fixed heap size limitation would be extremely useful.

Executable analysis

Everything in Breath of the Wild is implemented in the executable. Some aspects can be modified by editing assets; other things are hardcoded, meaning it's impossible to change how the game works by merely editing content files.

Analysing the executable is the only way to understand how things are implemented internally and how the game actually works. Eventually, after enough information has been collected, it becomes possible to predict how something will behave exactly and also to directly edit the game core. With a more complete understanding, more in-depth modifications can be done by hooking into functions and injecting custom code.

This is similar to the process of creating cheat codes, but different in that this relies on static analysis more and requires understanding how all the different subsystems interact.

Because of the sheer size of the executable, it is infeasible for this long-term project to be completed by a single person. If you are willing to help, please ping leoetlino for the Switch 1.5.0 IDC.  

Debug leftovers

The following tasks require reversing the executable and possibly patching it to re-enable functionality.

Demo mode

Setting the ROM type to demo ROM types (found in the executable) make the game crash. What is locked behind them?

Actor debug utilities

In a similar fashion, Nintendo has also left an entire ActorDebug subsystem in all release versions, which can print information to the screen and interact with many different actor system components and things like map units.

Stage select screen

Full article: StageSelect

The release version has a stage select screen left (see Executable for more details). It should be possible to get it to work.

  1. BotW CEDEC 2017 talk
  2. 0x7101E271AE in Switch 1.5.0
  3. 0x7101E3790B in Switch 1.5.0