Writing to the in active bank in ATSAMD51P19A

Hi,
 
I'm trying to write a firmware update mechanism that doesnt use a bootloader. (I work on an ATSAMD51P19A chip)
For this purpose I have code in my current active bank that is supposed to write a newer code to the inactive bank and switch banks once it is done.
My problem is that I cant write to the inactive bank (starting at address 0x80000), when I try to do that I get a hard fault.
Here is how I'm trying to make it happen:
1. Unlock the region -
      hri_nvmctrl_write_ADDR_reg(device->hw, dst_addr); // dst_addr = 0x80000
      hri_nvmctrl_write_CTRLB_reg(device->hw, NVMCTRL_CTRLB_CMD_UR | NVMCTRL_CTRLB_CMDEX_KEY);
2. Write a page worth of data to the page buffer:
#define NVM_MEMORY ((volatile uint32_t *)FLASH_ADDR)
      static void _flash_program(void *const hw, const uint32_t dst_addr, const uint8_t *buffer, const uint16_t size)
{
    uint32_t *ptr_read    = (uint32_t *)buffer;
    uint32_t  nvm_address = dst_addr / 4;
    uint16_t  i;

    while (!hri_nvmctrl_get_STATUS_READY_bit(hw)) {
        /* Wait until this module isn't busy */
    }

    hri_nvmctrl_write_CTRLB_reg(hw, NVMCTRL_CTRLB_CMD_PBC | NVMCTRL_CTRLB_CMDEX_KEY);

    while (!hri_nvmctrl_get_STATUS_READY_bit(hw)) {
        /* Wait until this module isn't busy */
    }

    /* Writes to the page buffer must be 32 bits, perform manual copy
     * to ensure alignment */
    for (i = 0; i < size; i += 4) {
        NVM_MEMORY[nvm_address++] = *ptr_read;
        ptr_read++;
    }

    while (!hri_nvmctrl_get_STATUS_READY_bit(hw)) {
        /* Wait until this module isn't busy */
    }

    hri_nvmctrl_write_ADDR_reg(hw, dst_addr);
    hri_nvmctrl_write_CTRLB_reg(hw, NVMCTRL_CTRLB_CMD_WP | NVMCTRL_CTRLB_CMDEX_KEY);
}
 
Once I call _flash_program() with addr 0x80000 and a buffer of length 512 I get a hard fault when it reaches the line "NVM_MEMORY[nvm_address++] = *ptr_read;".
 
What am I doing wrong ?
How can I program the inactive bank ?
 
Thanks