I have shit memory as well and I know it bothers my boyfriend to the nines. Actually, your relationship sounds like how ours used to be.
I asked him to help me remember things, such as when a bill was due or to remind me several times if we had made plans before the day of said plans. We both write on our calendar that hangs beside the front door and he will even remind me to check it.
As far as listening issues go... if he is speaking to you or would like to speak to you, stop what you are doing and give him your undivided attention. That should go without saying. It is a sign of respect from you to him that you think what he's saying is more important than anything else going on at that point in time. This will also help your memory - no distractions means you are less likely to forget what was said two minutes prior. If it's important, write it down. He needs to be aware that your memory is an issue for you and he needs to not be mean about it. But you owe it to him to show him just as much attention. Ask him to have patience with you and if he can't... well, there's nothing to be done to fix it.
As far as affection goes... you'll need to CALMLY have a serious conversation about it if it really bothers you that much. Tell him that sometimes you feel put out/sad/whatever when he doesn't respond to cuddling or whatever it is. DO NOT MAKE IT SOUND LIKE AN ACCUSATION. Ask him if there is any reason specifically why he doesn't always reciprocate. LISTEN to what he says. Work with him. You don't have to be joined at the hip to be in a happy, healthy relationship.
If his sarcasm is meant as humor, you're going to need to lighten up and learn to laugh with him. If he's being rude and mean to you on purpose, you need to tell him that when he does it that it really hurts your feelings and ask him to stop. You deserve to be treated with respect, too.
If you ever need someone to talk to, feel free to PM me :) Best of luck!