The look on his face when he asks her this shows he already knows her answer.
She looks straight at him and tells him not to travel alone, but before he can respond, he looks up and to the side, blinking, and opening his mouth like he’s almost going to speak before he finally just asks her to travel with him. When he does, he looks abruptly down failing to force a smile rather than looking her in the eye because he knows she’s going to return to her life eventually like she always does. And, as he discovered at the beginning of the episode, she’s older than the last time he saw her and he doesn’t know how much longer she has left.
When he looks down after speaking, you can see how much he just opened up by asking her that. He’s not just asking her to come on fun adventures and he doesn’t want to go out and find new friends to replace Amy and Rory.
He can’t look her in the eye because he’s putting himself in a vulnerable position by asking her, specifically and only her, to travel with him.
He first asked her to travel with him a very long time ago and she’s never taken him up on it, but this time she’s all he has left and she means more to him now than she did then. If she says no to traveling with him, it’s going to hurt so much more. This is it. Once she’s gone to the Library, he falls hard.