Can there be two different math?

Can there be two different math?

As per usual, let PA denote Peano Arithmetic and ZFC denote
Zermelo-Fraenkel set theory with choice. These two theories 'validate'
each other, in the sense that ZFC proves that the PA axioms hold for the
standard model of the natural numbers.
My question is this. Can we find theories PA' and ZFC' such that:
PA and PA' agree on all $\Pi_1$ sentences and all $\Sigma_1$ sentences.
ZFC and ZFC' agree on all $\Pi_1$ sentences and all $\Sigma_1$ sentences
in the language of arithmetic.
PA and PA' contradict each other. That is, there is a sentence $\sigma$ in
the language of arithmetic such that either PA proves $\sigma$ and PA'
proves $\neg \sigma$, or vice versa.
ZFC' validates PA', in the sense that ZFC' proves that the PA' axioms hold
for the standard model of the natural numbers.
Is this possible?
And if so, how do we decide which is the correct, true math?
Can there be two different math, with neither more correct than the other?
And is it even important?