c ++ – Selection of objects using the ray tracer in opengl using inverses

I have a scene with many objects and I would like to be able to select them using the ray tracing. I used to have only model transformations happening in vertex shading and I was able to select objects by doing

                                Vector4f p_screen (xpos, height-1-ypos, 0,1);
Vector4f p_canonical ((p_screen[0]/ width) * 2-1, (screen_p[1]/ height) * 2-1.0,1);
p_world = model.inverse () * p_canonical;
// Convert the position of the screen into world coordinates.
float xworld = p_world[0];
float yworld = p_world[1];

Vector4f ray_origin = Vector4f (xworld, yworld, 1, 1);
Vector4f ray_direction = Vector4f (0, 0, -1, 0);

All that worked perfectly.

However, I just implemented a projection and see the transformation in vertex shading as such

                "#version 150 core  n"
"in position vec3;"
"in normal vec3";

"out vec3 FragPos";
"out vec3 Normal";

"uniform mat4 model";
"uniform mat4 view;"
"uniform mat4 proj";

"main vacuum ()"
"{"
"FragPos = vec3 (model * vec4 (position, 1.0));"
"Normal = mat3 (transpose (inverse (model))) * normal;"
"gl_Position = proj * view * vec4 (FragPos, 1.0);"
"}";

I tried to update my ray tracer to include the inverse of the view and projection matrices to be

    p_world = model.inverse () * view.inverse () * projection.inverse () * p_canonical;

but now it's totally broken and I'm not quite sure what I'm doing wrong. I know that the view and projection matrices work, since it is shown in the projected window with perspective projection.