"According to Matthew, what is Joseph and Mary's hometown? Your natural reaction is to say 'Nazareth.' But only Luke says this. Matthew sats nothing of the sort. He first mentions Joseph and Mary not in connection with Nazareth but in connection with Bethlehem. [2:1-2] The wise men, who are following a star (presumably it took some time), come to worship Jesus in his house in Bethlehem. [2:11] Joseph and Mary evidently live there. There is nothing about an inn and a manger in Matthew. Moreover, when Herod slaughters the children, he instructs his soldiers to kill every male two years and under. [2:16] This must indicate that Jesus had been born some time before the wise men show up. Otherwise the instruction does not make much sense: surely even Roman soldiers could recognize that a toddler walking around the playground was not an infant born some time last week. So Joseph and Mary are still living in Bethlehem months or even a year after the birth of Jesus. So how can Luke be right when he says that they are from Nazareth and returned there just a month or so after Jesus' birth? Moreover, according to Matthew, after the family flees to Egypt and then returns upon the death of Herod, they initially plan to return to Judea, where Bethlehem is located. [2:22] They cannot do so, however, because now Archelaus is the ruler, so they relocate to Nazareth. In Matthew's account they are not originally from Nazareth but from Bethlehem."
Ehrman summarizes everything well, but I want to expand just a bit on that last part. Here's Matthew 2:22-23:
"But when [Joseph] heard that Archelaus was reigning over Judea instead of his father Herod, he was afraid to go there. And being warned by God in a dream, he turned aside into the region of Galilee. And he came and dwelt in a city called Nazareth, that it might be fulfilled which was spoken by the prophets, 'He shall be called a Nazarene.'"
I'm going to break down the language in this passage to make it entirely clear why it shows that, according to Matthew, Mary and Joseph did not originally live in Nazareth.
- "he was afraid to go there" – Joseph had planned to return to Bethlehem in Judea.
- "he turned aside" – Joseph did not originally intend to go into Galilee.
- "he came and dwelt in" – This perfectly describes moving to a new town. Had Matthew wanted to describe arriving home, he would have said "came back to" or "returned to."
- "a city called Nazareth" – Matthew introduces Nazareth as a completely novel location, as though nothing relevant has previously occurred there.
- "that it might be fulfilled" – The purpose is not to return home, but to fulfill a prophecy.
It's worth noting that there is no prophecy that "He shall be called a Nazarene" anywhere in the Old Testament. Apologists have tried to make sense of this in various ways, but the bottom line is that Matthew basically made it up. There's also a Bethlehem prophecy quoted in Matthew 2:6, a very rough paraphrase of Micah 5:2 (compare them here). Ehrman tells us why this is significant:
"[T]here is a prophecy in the Old Testament book of Micah that a savior would come from Bethlehem. What were these gospel writers to do with the fact that it was widely known that Jesus came from Nazareth? ... To get Jesus born in Bethlehem but raised in Nazareth, Matthew and Luke independently came up with solutions that no doubt struck each of them as plausible."
So, to review: Matthew heavily implies that Mary and Joseph were from Bethlehem, he gives every indication that they were not from Nazareth, and he has a prophetic motive for constructing the narrative the way he did. If we provisionally assume Luke's narrative is correct that Nazareth was their hometown, then Matthew's account on this point is thoroughly deceptive at best and irreconcilably conflicting at worst.