How do I force the installation to cancel if the .dll returns a unsuccessful return code?
The return value from launching a .dll is not returned to the Call Custom DLL from Installation action. The action uses a .dll that retrieves the value of the return code, but it is not passed back to the original action.
The return code could be a large range of numbers because there are many API and .dll files that can be called from this action. To prevent those otherwise harmless return codes to fail, the installation the code is not returned to the Call Custom DLL from Installation action.
To terminate the installation based on the return code the Call Custom DLL action should be located in the User Interface or Execute Immediate sequences. If you write the value of the return code to a file or registry key on the computer, you can use another custom action located directly after the Call Custom DLL action that could evaluate the return property and fail the installation using a terminate installation action.
If <MYPROPERTY> = <NUMBER> then
Note: In Execute Deferred, that return value is not accessible. If the action is required to be in Execute Deferred, it must be a standard Call DLL action in order to use the return code to terminate the installation.