slow insert speeds with bytea

Is there anything I can to increase insert speeds for bytea? Currently
running postgres 9.6.15

I have a few tables without a bytea and a few with bytea. There is a large
performance difference with inserts between the two. I'm inserting a byte[]
that's usually less than 1MB on content. The content itself is actually
just utf8 string data.

For the non-bytea table, inserts can be as high as 40k rows/sec, whereas
the bytea table is closer to 4k/sec or less.

If this is just a limitation of postgres, then that's fine but the
performance delta is so significant that i feel like i'm missing something