New boy is driving me crazy!?

I am a 23 y/o f. About a month ago I met this guy at a bar and we really hit it off. He texted me a few times here and there over the next month, and we tried to make plans but it never worked out because we had opposite schedules. Then last week (maybe Thurs) he called me to get together the following week. Long story short, he ended up meeting me out at a bar on Sat. night. We had a great time, and he even came back to my apartment afterwards (nothing much sexually happened...I wanted to wait). He left very early Sun. morning because he had something the next day. He kissed me goodbye and that was it. I texted him on Sun to say I had a great time, and he answered saying he had fun too. But that was it and now it's Wednesday morning already and I have not heard from him. When we were together on Sat we joked about how he is a terrible texter, and he told me that he hears this all the time. But I am worried that he is not interested. I don't want to be annoying and text him again, but I REALLY want to see him again. I have dated a lot over the past year, and have never been this excited about a guy the next day. PLEASE HELP!