Manacles require a DC 20 strength check to break. However in this case a character is not trying to break out of manacles, they are trying to break out of manacles tied with 100ft of rope. How high a DC is required to break out of that is up to the DM, but clearly it should be higher than 20.
Considering the target is bound tightly and gagged, and spending multiple days in this condition, I would expect this constitutes disadvantage if not exhaustion.
Retries are purely up to the DM
While there are no rules preventing making unlimited retries as fast as you can (once ever 6 seconds in combat, more or less often outside of combat depending on the task), there are various factors you should consider.
In many situations there is only 1 opportunity for attempts. If you are trying to spot a lie and fail, you can hardly ask the person to repeat what they just said. If you are trying to pickpocket someone and fail, then they might be on to you.
In some other situations attempts may prevent subsequent retries. If you try to pick a lock and fail your tools may foul the lock or become stuck inside. If you try to forge a signature and fail, you can’t just erase the ink.
In this situation the target is sitting there, manacled and tied up. They have plenty of opportunities to attempt to break out. Failing to break out does not prevent future break outs.
But, After minutes if not hours of attempts the target is surely exhausted, the DM may start stacking on exhaustion levels. How long can you stand there trying to tear a piece of rope apart while bound?
Being tied up doesn’t exist in a vacuum
So how can you keep someone tied up if they can retry many many times when the only penalty is perhaps getting tired?
Well, there’s no need to be narrow minded. As you experienced, those are not the only possible penalties. When the target did manage to partially succeed you managed to restrain them further. You could also threaten to beat them, harass them, or threaten someone or something they care about. These threats are what keeps the target from making attempts.
The DM may decide that the target may also be conserving strength or thinking of another plan. NPCs are smart after all.