physics – Player clips into the ground when I press the restart button

The title says it all. I have a subway surfers like game prototype, where I just added a restart button that can reload the scene. However whenever I press restart the charaters gets into a slightly lower position. After a few deaths the player is clipped into the ground and cannot jump at all.
I looked up so many similar questions, all about checking colliders and rigidbodies but I personally couldn’t find anything. It also has to do something with my gravity and stuff. Because I have a gravity modifier that if I set to 1, this issue never happens. The problem is, that then the player jumps around like he is on the moon.
I also tried an experiment where I started my player from the air, not from the ground. And with every single restart the player fell faster and faster eventually falling trough the ground once again. So something can be with gravity.
This is what I have in the character controller. I wasn’t satisfied with my physics and I wanted it to make a bit more interesting.

I tried readding colliders, putting the same scripts on brand new objects, checking on and off rigidbody options.

void Start()
    {
        playerRb = GetComponent<Rigidbody>();
        Physics.gravity *= gravityModifier;
    }

    // Update is called once per frame
    void Update()
    {
        Vector3 xPos = new Vector3(xRange, 0, 0);
        if (Input.GetKeyDown(KeyCode.Space) && isOnGround)
        {
            playerRb.AddForce(Vector3.up * jumpForce, ForceMode.Impulse);
            isOnGround = false;
        }