If matter is expanding, what is it expanding relative to? If we're going to posit that mass is expanding, then we're implying a hypothetical measurement could exist of it's expansion. If we could thought-experiment our way to a hypothetical measurement, what would that measurement look like?
This discussion gets to the heart of probably the biggest unsolved question in physics, what is mass? We measure mass, we can manipulate it, and we can infer it's existence where we cannot see it. But we don't really know what it is. We can define mass as that which exists in the presence of matter, but this is wholly unsatisfying. This question is what prompts particle physicsts to look for "god" particles, and particles responsible for gravity, because we haven't really solved the problem of what mass is.
Ask yourself a very hard question. This is a very difficult question to mentally wrestle with.
At what speed does information travel? What is the maximum allowable speed of information? This would have to be limited by physics if it we are supposing it is limited, right? I mean, if you could hypothetically grab Jupiter and move it around the earth at whatever speed you wanted, how long would it take for any effect of Jupiter's close proximity and fast orbit of earths position to reach and affect Earth? That's the kind of maximum speed we're talking about. Try to imagine that the universe has a maximum speed at which things can affect each other, and then try and figure out what that speed is. If the universe does not have a maximum speed limit for information, then everywhere we point our telescopes, we should be looking at present-time. However, we know this is not the case, becuase the further we look into the universe, the stranger things look, the less planet/star/galaxy-like things are.
When you start playing with that question, you start to realize a lot of things.