Multi-Homed BGP - Shorewall & FRR

Greetings - Long time listener, first time caller.

I am running a physical server with shorewalll 5.2.8, Rocky Linux 9.1, and
FRR 8.5

I have 2 ISP and a Class C ASN that is peered to both ISP.

I am using shorewall to provide basic firewall for the router, with minimal
configuration.

I have a separate firewall that is handling NAT / DHCP and other rules.

For the router, I have 3 interfaces

ISP1
ISP2
BGPWAN

ISP1 & ISP2 are setup with the peering IP addresses for each provider, and
BGPWAN is setup with the ARIN assigned class C.

The issue I am having is that when I ping ISP1 externally, replies go out
ISP1 as expected. When pinging ISP2, replies are attempting to go out ISP1.

I don't know if this is something that I should attempt to configure within
shorewall (multi-ISP setup and conntrack) or if there is a better way to
handle this via FRR.

I have not installed conntrack yet, as I was trying to keep things
minimally installed. Appreciate the advice.