Both Norwegian words can mean to try. ''Jeg skal prøve\forsøke å lage middag'' (I will try to make dinner). Sometimes you use prøve where forsøke is impossible. In a shop you say: ''Kan jeg prøve denne jakken?'' (Can I try this jacket?) In this sentence the word forsøke is impossible. But you can say to your friend: ''Jeg tror disse buksene er for trange, men jeg skal prøve\forsøke å få dem på meg.''(I think these pants are to tight, but I'll try to put them on.)
My translation of 'Can I attempt it?' was not accepted, however, one of the suggested correct answers shown is 'Can I attempt?', which is not a correct English sentence as there is no object of the verb. I knew my answer would be wrong as I added 'it' to the sentence, but I'm surprised by the answer suggestions.
It's not listed as the best translation, but is accepted. I imagine what happened is that Duo showed you the accepted solution closest to what you wrote, rather than the best solution. We're a bit more generous when translating to English as our goal is not to teach English but Norwegian.
As with lots of the discussions concerning translations in Duo, there is no context given and, as a result, there are several "best" answers possible. In this case, I find the recommended answer of "Can I make an attempt?" to be unusual, and only fitting in rather special circumstances.
The most likely context for "Kan jeg forsøke?" that I can imagine is that a colleague has been trying to do something difficult, maybe trying to get his car to start, and you think that you might be able to do it better. If that were the context, then the recommended answer should simply be: "Can I try?" or "Can I have a go?". In real life, though, I would most probably say: "Let me have a go", but that's probably something else in Norwegian.
When immediately following a modal verb, e.g. 'kan' in the sentence, the second verb is in its infinitive form (what you find in the dictionary) but without its infinitive marker 'å'. This is why forsøke is used.
If a third verb is used it will have 'å' in front of it. Look at this example I have taken from LINHARS above: Jeg skal forsøke å lage middag (I will try to make dinner). skal - modal, forsøke - 2nd verb, å lage - 3rd verb