"I like sitting by the window."
Translation:Ich sitze gerne am Fenster.
I'm just a learner, but I would say the difference is the same as in English between "at" and "by".
For example, look at the different meanings of "Stay by me" and "Stay at my home".
That's how it is with prepositions, you just have to learn them from scratch with every new language.
Also, in this context "an Fenster sitzen" (at) means that you are literally at the window, looking out at the beautiful world around. On the other hand, "bei Fenster sitzen" mean that you are sitting somewhere by the window, like next to it or somwhere near it, and this expression defines just your location, not an activity.
Just like sitting at the table ("an") means eating a meal, and sitting by ("bei") the table is just being somewhere near the table. At least that's how I understand it :)