Custom calling conventions don't affect caller clobbered register analysis

[WeiN76LQh]

Version and Platform (required):

• Binary Ninja Version: 5.3.9393-dev Ultimate (ae6ee6f5)
• Edition: Ultimate
• OS: macOS
• OS Version: 26.3
• CPU Architecture: M1

Bug Description:
When applying a custom calling convention that states certain registers are callee saved, like; x0, x1, x2, etc. (the ones that are usually clobbered or used as arguments in the default calling convention) I would expect the caller to assume its values for those registers persist across the call. Instead those registers seem to be being considered clobbered by the call because HLIL shows the call returning values into all registers that are used later on by the caller, that under the default calling convention would be considered clobbered.

18e977ac0    int32_t _glob0(uint64_t arg1, uint64_t arg2, uint64_t arg3, uint64_t arg4, uint64_t arg5)

18e977aec        int64_t* x1
18e977aec        void* x2
18e977aec        int64_t x3
18e977aec        void* x4
18e977aec        x1, x2, x3, x4 = ____chkstk_darwin(amountToIncreaseTheStackBy: 0x4010)
18e977b28        int64_t* x0 = ___gl_globtilde()
18e977b28        
18e977b2c        if (x0 == 0)
18e977d04            *j____error() = 7
18e977d08            return -1
...

I don't know the exact function type of this function but it clearly takes 5 arguments, which can be seen in the disassembly:

18e977ac0    int32_t _glob0(uint64_t arg1, uint64_t arg2, uint64_t arg3, uint64_t arg4, uint64_t arg5)

18e977ac0  7f2303d5   pacibsp 
18e977ac4  fc6fbba9   stp     x28, x27, [sp, #-0x50]! {__saved_x28} {__saved_x27}
18e977ac8  f85f01a9   stp     x24, x23, [sp, #0x10] {__saved_x24} {__saved_x23}
18e977acc  f65702a9   stp     x22, x21, [sp, #0x20] {__saved_x22} {__saved_x21}
18e977ad0  f44f03a9   stp     x20, x19, [sp, #0x30] {__saved_x20} {__saved_x19}
18e977ad4  fd7b04a9   stp     fp, lr, [sp, #0x40] {__saved_fp} {__saved_lr}
18e977ad8  fd030191   add     fp, sp, #0x40 {__saved_fp}
18e977adc  09028852   mov     w9, #0x4010
18e977ae0  b14030b0   adrp    x17, 0x1ef18c000
18e977ae4  31e21b91   add     x17, x17, #0x6f8
18e977ae8  300240f9   ldr     x16, [x17]  {____chkstk_darwin}  {data_1ef18c6f8}
18e977aec  110a3fd7   blraa   x16, x17  {____chkstk_darwin}
18e977af0  ff1340d1   sub     sp, sp, #0x4, lsl #0xc
18e977af4  ff4300d1   sub     sp, sp, #0x10
18e977af8  f30304aa   mov     x19, x4
18e977afc  f40303aa   mov     x20, x3
18e977b00  f50302aa   mov     x21, x2
18e977b04  f60301aa   mov     x22, x1
18e977b08  886c2f90   adrp    x8, 0x1ed707000
18e977b0c  08013091   add     x8, x8, #0xc00  {___stack_chk_guard}
18e977b10  080140f9   ldr     x8, [x8]  {___stack_chk_guard}
18e977b14  a8831bf8   stur    x8, [fp, #-0x48]
18e977b18  f8230091   add     x24, sp, #0x8 {var_4058}
18e977b1c  e1230091   add     x1, sp, #0x8 {var_4058}
18e977b20  02808052   mov     w2, #0x400
18e977b24  e30316aa   mov     x3, x22
18e977b28  95000094   bl      ___gl_globtilde
18e977b2c  800e00b4   cbz     x0, 0x18e977cfc
...

____chkstk_darwin only clobbers x9 so the values of the argument registers are exactly as they were when the function was called when they are assigned to the registers; x19, x20, x21, x22, after the ____chkstk_darwin call.

However you can see from the HLIL in the first snippet that Binary Ninja is getting confused and assuming the call to ____chkstk_darwin is returning values via x1, x2, x3, x4. Coincidentally these are 4 of the 5 argument registers that are used later on and are typically considered clobbered by the default calling convention.

Steps To Reproduce:
Please provide all steps required to reproduce the behavior:

1. Compile into a plugin the native code provided below for the custom calling convention (don't forget to call Register())
2. Open a copy of DYLD Shared Cache
3. Load the libsystem_pthread.dylib library
4. Wait for analysis to complete
5. Go to ____chkstk_darwin
6. First change its type so that its void ____chkstk_darwin(uint64_t amountToIncreaseTheStackBy @ x9) (this has to be done due to Using the UI to change a function's type removes the assigned custom calling convention #8052, see what happens if you skip this step if you're curious).
7. Now edit the function properties and change its calling convention to apple-chkstk
8. Go to _glob0
9. Observe the assignments to a bunch of registers as return values from the call to ____chkstk_darwin
10. Re-analyzing the function doesn't do anything

#pragma once

#include <vector>

#include "binaryninjaapi.h"

class AppleCheckStackCallingConvention : public BinaryNinja::CallingConvention {
private:
    AppleCheckStackCallingConvention(BinaryNinja::Architecture* arch);

public:
    static void Register();

    virtual std::vector<uint32_t> GetIntegerArgumentRegisters() override;
    virtual std::vector<uint32_t> GetFloatArgumentRegisters() override;
    virtual std::vector<uint32_t> GetCallerSavedRegisters() override;
    virtual std::vector<uint32_t> GetCalleeSavedRegisters() override;
    virtual std::vector<uint32_t> GetImplicitlyDefinedRegisters() override;
    virtual uint32_t GetIntegerReturnValueRegister() override;
    virtual std::vector<uint32_t> GetRequiredArgumentRegisters() override;
    virtual bool AreArgumentRegistersUsedForVarArgs() override;
    virtual bool IsEligibleForHeuristics() override;
};

#include "CheckStack.hpp"

#include <cassert>

#include "arch/arm64/disassembler/regs.h"
#include "binaryninjaapi.h"

using namespace BinaryNinja;

AppleCheckStackCallingConvention::AppleCheckStackCallingConvention(Architecture* arch)
    : CallingConvention(arch, "apple-chkstk")
{
}

void AppleCheckStackCallingConvention::Register()
{
    const auto arm64 = Architecture::GetByName("aarch64");
    assert(arm64);
    const auto conv = new AppleCheckStackCallingConvention(arm64);
    arm64->RegisterCallingConvention(conv);
}

std::vector<uint32_t> AppleCheckStackCallingConvention::GetIntegerArgumentRegisters()
{
    return { REG_X9 };
}

std::vector<uint32_t> AppleCheckStackCallingConvention::GetFloatArgumentRegisters()
{
    return {};
}

std::vector<uint32_t> AppleCheckStackCallingConvention::GetCallerSavedRegisters()
{
    return {};
}

std::vector<uint32_t> AppleCheckStackCallingConvention::GetCalleeSavedRegisters()
{
    return { REG_X0, REG_X1, REG_X2, REG_X3, REG_X4, REG_X5, REG_X6, REG_X7, REG_X8, REG_X10, REG_X11, REG_X12, REG_X13,
        REG_X14, REG_X15, REG_X16, REG_X17, REG_X18, REG_X19, REG_X20, REG_X21, REG_X22, REG_X23, REG_X24, REG_X25,
        REG_X26, REG_X27, REG_X28, REG_X29, REG_X30, REG_V0, REG_V1, REG_V2, REG_V3, REG_V4, REG_V5, REG_V6, REG_V7,
        REG_V8, REG_V9, REG_V10, REG_V11, REG_V12, REG_V13, REG_V14, REG_V15, REG_V16, REG_V17, REG_V18, REG_V19,
        REG_V20, REG_V21, REG_V22, REG_V23, REG_V24, REG_V25, REG_V26, REG_V27, REG_V28, REG_V29, REG_V30, REG_V31 };
}

std::vector<uint32_t> AppleCheckStackCallingConvention::GetImplicitlyDefinedRegisters()
{
    return {};
}

uint32_t AppleCheckStackCallingConvention::GetIntegerReturnValueRegister()
{
    return BN_INVALID_REGISTER;
}

std::vector<uint32_t> AppleCheckStackCallingConvention::GetRequiredArgumentRegisters()
{
    return { REG_X9 };
}

bool AppleCheckStackCallingConvention::AreArgumentRegistersUsedForVarArgs()
{
    return false;
}

bool AppleCheckStackCallingConvention::IsEligibleForHeuristics()
{
    // Don't let Binary Ninja guess other calls use this
    return false;
}

Expected Behavior:
Analysis should recognise that registers saved by the callee do not have their values modified and can therefore be assumed to be unchanged across calls. As far as I can tell Binary Ninja does this fine for default calling conventions. It just seems it ignores this for custom calling conventions. It feels like its hardcoded to only consider the default calling convention but I haven't done enough testing to confirm this theory.

Binary:
Tested against DYLD Shared Cache for iOS 26.2 for an iPhone 17 Pro Max.

Additional Information:
I've noticed before and can confirm its still the same, that the clobbered registers tick box list view in the edit function properties dialog doesn't seem to do anything. I'm not 100% sure what the exact purpose of these tick boxes are. I assumed it was to tell callers what register values they can assume have been modified/clobbered by calling the function but I've never seen any change in analysis when modifying this list.

[WeiN76LQh]

Interestingly this problem does not occur if you apply a call type adjustment to a call where the target is not loaded in the binary view.

So if you follow the instructions above to reproduce the issue but skip the parts about loading libsystem_pthread.dylib and setting the type for ____chkstk_darwin. Instead go directly to _glob0 where there will be the call to ____chkstk_darwin but the function won't be loaded in (although the symbol is defined for it). Then set a call type adjustment with the following type void __convention("apple-chkstk") calltarget(uint64_t amountToIncreaseTheStackBy). This will need to be done using the Python API due to #8052.

The following code works whilst the cursor is on the ____chkstk_darwin call line:

current_function.set_call_type_adjustment(here, Type.function(Type.void(), [FunctionParameter(Type.int(8), "amount")], bv.arch.calling_conventions["apple-chkstk"]))

Before:

After: