Which one and why?
Ich habe versucht Deutsch zu sprechen.
Ich habe Deutsch sprechen versucht.
Ich habe Deutsch zu sprechen versucht.
The first one. "Ich habe versucht, Deutsch zu sprechen." Technically, the comma between the clauses is required/not optional. Why?
There are two clauses here. "I tried/I have tried". That's a complete clause which is followed up with a comma to separate it from the next clause. The second clause is an infinitive clause, meaning it uses "zu (verb)" at the end of the sentence AND has no subject (re)stated... as in, it doesn't say "ich" again. It's just "...to speak German." with no actor specifically (re)stated. The "zu (verb)" is always at the end of a clause... always. Therefore, "Ich habe versucht, Deutsch zu sprechen."
I can explain why #3 is (also) correct. Here's a bit of grammar. (1) The conjugated verb always takes the 2nd place in word order. (2) All additional verb information (in your case, the past participle) gets shoved to the end, or final position, of the clause. (3) Therefore, the past participle of your sentence (versucht) bumps the infinitive clause (zu sprechen) up a spot from the last position. (4) Finally, the direct or indirect object (in this case, Deutsch) comes after the conjugated verb. So you get: ich - habe - Deutsch - zu sprechen - versucht.
I see where you're coming from, but I don't entirely agree. As far as (1) goes, I agree. However, insofar as (2) and (3) are concerned, you're treating "Deutsch zu sprechen" as you would, for example, objects or adverbs, and it's neither: it's an infinitive clause. Infinitive clauses can (at least in my experience, so someone please correct me if I don't know this yet) only go in their own separate clause after a comma, or essentially replace the subject in first position. For example: Deutsch zu sprechen ist mein Ziel (however, that's not what's happening here).
It makes sense that "to speak German" is an independent thing that one can "try", perhaps with an unspecific pronoun to refer to it like "it", for example: Ich habe es versucht. Here, you'd be right, and that'd be fine. But as an infinitive clause, no, it doesn't work.