Regardless of whether or not you call the men whom God worked through part of the Catholic Church, the fact remains that actually God did work through men, and those men therefore, by the power of God, were infallible. That is, if one actually understands what infallible means.
To be infallible is a charism of the Holy Spirit that ensures that truth with prevail on Earth. If you think that a book of scripture is inspired and inerrant, then by the definition of infallible, you have no choice to admit that the author of that book, whether it be Matthew, Mark, Luke, John, Peter, Paul, or whomever, is infallible, because at least for that one act of writing that book, his act was protected by the Holy Spirit.
Those men I just listed were part of the Church, which is the people of God, and thus it was absolutely necessary for infallibility to come into play for the Bible to be inspired in the first place.
Likewise it is also necessary to have another infallible act to define the canon of scripture, because while a book might itself be inspired and inerrant, the men deciding if that book is inspired and inerrant are by nature very prone to error. They could easily have said that an inerrant, inspired book was not canonical while a errant, non-inspired book was canonical. UNLESS they were infallible: that is, unless the Holy Spirit protected their decree. Only then do we know for sure that the Canon of scripture contains ALL the inspired books and ONLY the inspired books.
In the absence of these infallible persons, we would never know whether the canon is correct and we may be using a Bible that is part inspired and part not inspired while calling the whole thing inspired.