It would not. Ag tuiscint is the Present progressive "understanding" and it makes no sense after in ann.
Táim ag tuiscint an leabhair - "I am understanding the book" (inelegant, but grammatically feasible).
Note leabhair, the genitive singular after the progressive, not the nominative plural.
Hum. I believe the lentition because it's the preposition a, whereas ag is some other part of speech (but I know not what). My grammar book gives examples of using a where it is not "to verb" eg tá sé tar éis an bád a dhíol, rendered in Hiberno-English as "he is after selling the boat."
Probably not the best example, because "selling the boat" isn't an infinitive construction ("to do something"), but you're on the right track.
In English, where sentences are subject-verb-object, an infinitive phrase is verb-subject - "to sell a boat", "to read a book", "to break the glass".
In Irish, where sentences are verb-subject-object, an infinitive phrase is subject-verb - bád a dhíol, leabhar a léamh, an ghloine a bhriseadh.
The a is indeed a preposition, and here are some further examples from the FGB:
(Used to connect a preceding noun or pron. with vn.) (a)Síol a chur, to sow seed. Uisce a ól, to drink water. Ba mhaith liom iad a bheith ann, I would like them to be there. (b) (In relative clause) An rud atá sé a scríobh, what he is writing. An fear atáthar a dhaoradh, the man who is being condemned. (c) (Denoting purpose) D’éirigh sé a chaint, he rose to speak. Téigh a chodladh, go to sleep. Tháinig sé a iarraidh iasachta orm, he came to ask me for a loan.
But to the initial question, a verb doesn't always have a subject in these situations, and it won't be lenited in that case, because it is the preposition that causes the lenition, and you don't get a preposition without a subject - Is maith liom snámh - "I like to swim", caithfidh mé imeacht - "I have to leave", etc
GRMH. Just to add to my previous: ag is, of course, also a preposition but without lentition eg ag an doras.
Most languages have an infuriating fuzzy logic. We just have to live with it. I don't think it is possible to learn by rote all the correct usages of verbal nouns in Irish.
From your example, I would add Is maith liom snámh and Is maith liom a bheith ag snámh.