Is the so-called evolution of man directly observed?
Yep. The evolution of alleles for high-altitude life in Tibetans, and the evolution of an allele against plague, to name two. The recent evolution of HbC against malaria, without the sickle cell consequences for another.
Common descent of all human species has never been directly observed. We haven't even observed the common descent of all H. sapiens. We do, however, have abundant evidence for this, just as we have abundant evidence that a giant redwood tree can grow to maturity, even if no one has directly seen it happen. Creationists are willing to accept the evidence for growth of giant redwoods, but are unwilling to accept the much stronger evidence for human evolution.
Contradicting isn't it?
Yes, it is. But giant redwoods don't contradict the man-made doctrines of YE creationism. And that makes all the difference for them.
You said "Evolution is a fact because it's a directly-observed phenomenon."
Yep. We see it every day.
Evolution, as to the degree that one directly observes it to happen, may well be a fact in that sense.
Just as some growth of a redwood seedling is a fact in that sense.
But with regards the evolution of man, which obviously is not and cannot be directly observed, it remains to be not a fact.
In the sense that the growth of a seedling into a giant redwood remains to be not a fact. In each case, the evidence is overwhelming for it, but if you think facts are only things you've directly observed, then there you are. Again, the difference is that giant redwoods don't contradict your new doctrine men made for you.
What it is, is plain guess and make believe.
As you now realize, it's merely the only reasonable inference from the evidence. We can confidently admit to the fact of human evolution and the growth of seedlings to giant redwoods, even if we've never observed it.
It can't even be considered a theory.
A theory is an idea or group of ideas that has been repeatedly confirmed by testing. As you have learned, there are many, many predictions of evolutionary theory that have been later confirmed. Why not just accept it God's way, and be done with it?