It's almost 3am here, so bear with me if this is fuzzy.

I have struggled with an analogy for this for awhile.  You can think of this
like a phone conversation.  I call you using your phone number, but for us to
talk you have to pick up the phone.  Since I called you, you can look on
caller ID and then call me back... but you can't force me to pick up the
phone, either.

I know, it's a weak analogy.  Let's try a set of doors.

Imagine the front door to my house.  You are fine with telling me how to
lock the door, but there's no way you are giving me the keys to your house to
unlock the door.  And vice-versa, I'd be more than happy to tell you how to
lock the front door to my house, but there's no way I am giving you the key
to unlock it.

You are the client, I am the service in this example.

You want to talk to me securely.  To do so, you will encrypt the message
using my public key.  You can't decrypt the message using my key, I won't
give you my keys (not to my car, my house, or anything else).  But you can
encrypt the message just fine.  In other words, I can tell you how to lock
the door, but since I am the one with the key, I am the only one that can
unlock it.

You can sign the message with your private key and send the public key along
with the message.  When I get the message, I use the public key to re-sign
the message and compare the results... if it matches, I know it has not been
tampered with, and I know it came from you.  It's your signature, I am just
comparing it to the signature I have on file for you.

Now I want to send you a response.  I can't encrypt it with my public key,
because remember that I am not about to give you my key (not to my house, not
to my car).  If I did that, you couldn't unlock it.  I need to encrypt the
message with something that only you can decrypt it with... but what do I
have laying around that could do that?  Oh yeah... you sent me your public
key when you signed the message.  I can use that.  Now I can't decrypt the
response, because I don't have your private key (I am betting you have the
same policy about not handing out the key to your front door).  In other
words, you came to my house and left a note that told me how to lock your
front door.

The whole scheme works because I can't unlock your stuff, and you can't
unlock mine.