How I handled this as a DM
I was aware of how difficult it is in swords and sorcery genre games to keep a spell caster under control/in prison, and not just in the current edition of D&D. This 2011 vintage Q&A illustrates the consistent problem that this has presented. But the NPCs and villains do have some tools with reasonable effectiveness.
From your comment, the party is: Bard, Ranger, Warlock, Priest, all at 4th level.
(it’s 5e D&D, they are an Imp familiar and a Ranger’s Primal Beast) that they can summon to help
OK, the Bard’s awake and the other three start at 0 HP. Simply having allies amplifies the difficulty of keeping them in custody.
I had a similar situation where the party (me DM) captured an evil druid (CR 2) and wanted to turn him over to authorities (eventually, a very high level druid in another settlement). They were worried that the druid would, for example, wild shape into a spider and crawl out of his cell.
Keep the spell caster knocked out and under 24/7 guard.
Take away the components pouch, take away the material components (if any) and if need be strip the caster down to their undies or completely. That’s step one of the precaution. Bind, and if need be, gag the spell caster (Bind and gag is a temporary inconvenienceif the PC is even modestly resourceful, and/or has compatriots with them).
The initial “keep him knocked out” method was that a guard or PC, when the NPC Druid would wake up (he’d been reduced to 0 HP, knocked out, bound and taken to a ship) would whack him with a club to reduce him to 0 HP again. Dreamland for 1-4 hours at this point. This required an investment of resources, in terms of a 24/7 guard and the willingness to keep whacking this prisoner on the head. They would then stabilize him to prevent death saving throw fails. Yeah, narratively, he ended up with a lot of bruise marks on his head.
Use a potion / poison that induces sleep/unconsciousness.
You have control of the NPCs, so either sufficient alcohol or something like it fed to the prisoner keeps them incapacitated or unconscious (Conditions, Appendix A). For example:
Essence of Ether (Inhaled). A creature subjected to this poison must
succeed on a DC 15 Constitution saving throw or become poisoned for 8
hours. The poisoned creature is unconscious. The creature wakes up if
it takes damage or if a other creature takes an action to shake it
awake. (Ref = DMG, also in the SRD page 204)
Or, just rub this on them if they won’t drink or breath in the ether
Oil of Taggit (Contact). A creature subjected to this poison must
succeed on a DC 13 Constitution saving throw or become poisoned for 24
hours. The poisoned creature is unconscious. The creature wakes up if
it takes damage.
Or homebrew a similar poison/drug using these for basic guidelines.
That prevents their being able to cast spells, since they can’t take actions. While the party didn’t resort to that, the first town that had this NPC druid in jail eventually did.
- The fighting chance
This risks of the “knock out drug” include a few potential loop holes:
- PC makes the saving throw and then fakes being knocked out – if/when this happens, the guards think that they are disabled when they are not
- When the “knock out juice” wears off, the PCs wake up and, unless the Bad Guy’s minions are very alert (perception check for the bad guys), the PCs have a chance to try to sleight-of-hand or stealth based action to get help, or have the imp or primal companion do something to open the chance for escape.
- Use of the feign death spell in a situation like this (party has a cleric) may provide the party with the misdirection that they need to get an escape attempt underway.
- Inflicting the Poisoned Condition is another method with loopholes
There are other poisons that induce the poisoned condition, such as pale tincture or assassin’s blood (same place in SRD/DMG. Those two last for 24 hours if the save isn’t made. While this leaves the PC with disadvantage on ability checks and attacks (Appendix A, Conditions, Poisoned) there is a saving throw which leaves a loophole. If you keep lacing their food and drink with this stuff, and thus keep the PCs poisoned, their efforts to escape will be hampered, but are not utterly voided as being unconscious does or incapacitated does.
How my first D&D 5e DM handled it
We were fed poisoned soup (DC 20 save) tried to fight our way out, got beat down to 0 HP, and were bound, gagged, and kept in a dark room; all equipment was confiscated.
So there we were, a barbarian human, a dwarf paladin, and a cleric human(me), level 6, bound, gagged and naked. We got on with our various squrmings around and try to undo the bonds working after we woke up from being at 0 HP; becoming unbound was not all that much trouble, but we were in an unfamiliar place and tried to sneak out. Multiple guards with pole arms (and with polearm mastery) were able to overcome us (My attempt at a command spell met with three successful saving throws – the dice hated me!) and knocked us back to nearly 0 HP before the NPC wizard showed up and put us to sleep. We woke up bound and gagged again, with two armed half elves with a serious case of negative attitude, guarding us in the dark room. When I began to talk to them my cleric got knocked out again. The other three party members were the ones who had to break us three out.
How I almost handled it for a different group: transporting statues
After two escape attempts, after which they were captured while trying to retrieve their gear, my Evil Boss lady threatened to turn the PCs to stone. She had some pet cockatrices. Once the PC is turned to stone, they won’t be able to act until the Bigger Bad Guy unstones them at the destination.
The problem with this approach, turning the PCs to stone, is that it closes the door on any PC attempts at the escape. In my experience, escape scenarios can be a lot of fun. This is ultimately why I chose not to have the Evil Lady make good on that threat. If it fits your situation, though, turning the PCs to stone and shipping them the 600 miles (per your comment) by boat or cart reduces the overhead for the Evil Party considerably as the try to make the delivery of these slaves to the Bigger and Badder Evil party.
With the Imp and the Primal Companion, I as a DM would prefer to watch and see how they put those class features to work.
The Primal companion from Tasha’s, which your party’s Ranger has access to, does not require V/S/M to summon. If the Ranger is conscious, that source of assistance seems to be unsuppressable.
A word on the scope of your challenge if you are unwilling to render them unconscious or otherwise incapacitated. If you remove all of their material components and spell casting foci as a foundation for the challenge to your PCs as they try to escape, there are still a variety of spells available to them.
From this question (using PHB, Xanathar’s, and Tasha’s) the entire list of spells from all classes that are ‘verbal component only’ are:
Cantrips: Lightning Lure, Mind Sliver, Sword Burst, Thaumaturgy, Vicious Mockery
Lvl 1: Cause Fear, Command, (Compelled Duel), Dissonant Whispers,
Ensnaring Strike, Faerie Fire, Hail of Thorns, Healing Word, Hunter’s
Mark, (Searing Smite, Thunderous Smite, Wrathful Smite) Zephyr Strike
Lvl 2: Blindness/Deafness, Blur, (Branding Smite), Earthbind, Knock,
Misty Step, Prayer of Healing, Tasha’s Mind Whip, Warding Wind
Since your party has no paladin, the compelled duel and smite spells are not available (in parens above); depending on your party’s spell selections, some of the above are likely available for them to use.
For “somatic components only spells” for the four classes in the party:
Cantrip: Thunderclap, True Strike
First Level: Absorb Elements
Second Level: Beast Sense, Mind Spike
If they are able to use both verbal and somatic components, but no material components, there are an additional 45 spells that they have available among cantrips, first level, and second level spells. D&DBeyond Search is here. I am not going to list them all since I have no idea what your characters selected, but the number of spells that can help them escape is considerable.
The bottom line is that the point I mentioned at the top of this post – the sheer difficulty in keeping captive four spell casters who are not incapacitated or rendered unconscious – becomes layered and quite complex even if verbal and somatic components are their only options for spell casting.
The Heavy Armor Rules Mechanic approach
One way to deal with the bard, sorcerer and warlock is to make them don heavy armor that they can’t take off per this fine answer – (which I suggest you take a good look at; good experienced-based input). Why does this work? If one is not proficient with heavy armor one cannot cast spells while wearing it. PHB CH 5, Armor and Shields
If you wear armor that you lack proficiency with, you have disadvantage on any ability check, saving throw, or attack roll that involves Strength or Dexterity, and you can’t cast spells. (Basic Rule, p. 46)
An additional benefit to this is that attempts at stealth to get away will be at disadvantage if they are stuck in heavy armor.