I'd prefer to/I'd rather/ I'd rather you didn't

 would prefer + infinitive

After it's time / would prefer we use the infinitive:
Examples:
A: It's time to get the dinner ready.
B: I'd prefer to go to a restaurant for a change, wouldn't you?

A: It's time to clear out the spare room. Your parents are coming next month.
B: I'm too tired just now. I'd prefer to do it at the weekend.

A: Isn't it about time to pack up and go home?
B: I'd prefer to stay just a bit longer. The beach is so nice today, isn't it?

Would rather + infinitive without to
After would rather we use the infinitive without to:
Examples:

A: It's time to cut the grass.
B: I'm at a great bit in my book. I'd rather do it later.

A: It's time to go now.
B: Can't we stay a bit longer? I'd rather get a later train.

A: Come on, it's time to do the dishes.
B: I'd rather leave them till the morning. Let's just go to bed and forget about them.

would rather + past tense
We can use a past tense after it's time or would rather but it has a present meaning:
Examples:

A: Do you mind if I smoke?
B: I'd rather you didn't.

A: Can I go round to Bobby's after school tomorrow?
B: I'd rather you came straight home actually. Remember your grandma's coming to visit.

A: Where are we going this summer?
B: I don't know yet. I guess it's time we were planning our holidays.