"Excuse me, is the bathroom here?"
It seems you are confusing nàlǐ 那里 (over there) and nǎlǐ 哪里 (where?). The first syllable in 哪里 is in third tone rather than fourth (which just goes to show that tones are indeed as important as vowels and consonants).
哪里, being a question word, usually doesn’t combine with 吗. I say “usually” because you can have both in the same sentence under certain circumstances, but you won’t encounter those until you get to subordinate clauses a bit later in the course. But for the moment you can memorize the rule that 哪里/哪儿 doesn’t go with 吗, and only worry about the exception later.
That would be “Excuse me, is here in/at the bathroom?“
在 is (in this case) a verb meaning “to be (located) at/in (a place)”, so 在洗手间 “to be in the bathroom”. Whereas 洗手间在这里 “the bathroom is here/at this place” (literally “at here”, but English doesn’t use a preposition with “here/there/where”). Add 吗 and you get the corresponding yes-no question: “Is the bathroom here?”
An alternative would be 这里是洗手间吗 “is this the bathroom”. Here you have to use 是 instead of 在 because you’re not asking where “here” is located. You’re basically asking “this place = bathroom?” That being said, this way of asking sounds to me like both you and the addressee are already in the bathroom when you’re asking the question (it’s the kind of formulation you’d use when you’re unsure of where you are, a bit like English “what is this place?”), whereas with 洗手间在这里吗 you’re probably standing outside the door pointing at it.