it is impossible to identify a university, because it is not a physical object
That's where I see you as going wrong. What do you mean by 'identify'? You are saying that only physical objects, according to materialists, can be 'identified.' I can talk about unicorns and not think that they exist. If that is 'identifying' a unicorn, so be it. Your conclusion that in order to speak of unicorns, I must accept a separate realm of existence (universals) is the position of one of my favorite philosophers, Plato, and the issue has been debated for 2500 years.
I think Ryle was saying that such attribution of 'existence' to universal concepts constitutes a confusion of applying adjectives and attributes of one category to a separate category where such adjectives are improper - grammatically, semantically, and perhaps metaphysically if you believe that metaphysics is meaningful. To be a consistent nominalist, I think one has to maintain metaphysical neutrality ('ontological relativity' per Quine) all the way down to the fundamental particles of physics. In other words, the buildings of the university do not enjoy a special 'reality' that concepts do not, for instance. That we often call physical objects 'things' or 'objects' that 'exist' is because that is the grammar that has developed naturally over time. A building, of course, can be thought of as a collection of bricks and mortar, then types of molecules, then atoms, and so forth. We speak of each 'level' differently and to attribute characteristics of 'physical objects' to quarks is as wrong as attributing such to 'concepts' - both are category mistakes.
You see such nominal relativity, I think, as fundamentally absurd. Perhaps you are right that it is 'all or nothing' - that is, if universals don't exist, then neither do buildings. But then we'll spend a long time arguing over what the word 'exists' means. I think you're using the word 'identify' as something like 'conferring existence onto.'