I am not a native speaker but I think your sentence is correct, with the only difference being slight abbreviation:

"Give to him, not me" instead of "Give IT to him, not me."

उसे is him/her/that similar to the objective form of he/she/it Whereas उससे means by him/her/that or from him/her/that or with him/her/that (से is more like a preposition)