dnd 5e – How can I maintain control over a Dybbuk for over an hour?

Just stop it from speaking

The best solution is a magic circle for the sake of planar binding (see this Q&A for solutions on casting planar binding on a creature conjured by a spell with a 1-hour duration if the Dybbuk was conjured by summon greater demon).

As you mention, dimension door is a problem since it can try to escape this way on a save (and can do so at-will):

If the creature tries to use teleportation or interplanar travel to (escape the cylinder), it must first succeed on a Charisma saving throw.

However, keep in mind that dimension door has a verbal component:

Most spells require the chanting of mystic words. The words themselves aren’t the source of the spell’s power; rather, the particular combination of sounds, with specific pitch and resonance, sets the threads of magic in motion.

…as such, any method you might use to restrict verbal components would prevent the fiend from escaping. The Basic Rules provide at least a few examples:

a character who is gagged or in an area of silence, such as one created by the silence spell, can’t cast a spell with a verbal component.

If silence is your best solution (which it might be since a Dybbuk is an incorporeal spirit when not possessing a corpse), then you will need enough level 3+ spellcasters (bards, clerics, desert druids, and some warlocks are eligible) or a level 5+ rangers to cast it six times (since silence lasts 10 minutes).

For a more level efficient option, you might be able to have a druid, ranger, or trickery cleric use pass without trace in the vicinity of the Dybbuk:

A veil of shadows and silence radiates from you, masking you and your companions from detection. For the duration, each creature you choose within 30 feet of you (is affected by the spell effect)

It is not abundantly clear if the silence from this veil prevents verbal components in the same way as the silence spell but a generous DM might agree that it does. If so, you only need one casting since pass without trace lasts an hour.

What if I don’t have silence or enough spell slots to cast it 6 times

In this case, PixelMaster, in the comments offers some other solutions:

  • a level 9 cleric or fiend warlock could cast hallow with Extradimensional Interference:

Affected creatures can’t move or travel using teleportation or by extradimensional or interplanar means.

This option has the added benefit that you could cast the spell any time before summoning the Dybbuk since it lasts until dispelled (simply exclude fiends so that they can enter the area of effect)

  • you could use Dimensional Shackles (a rare magic item):

In addition to serving as mundane manacles, the shackles prevent a creature bound by them from using any method of extradimensional movement, including teleportation or travel to a different plane of existence. 

Innate Spellcasting

Restricting verbal components works because the Dybbuk’s stat block only removes the need for material components; Verbal components still must be used:

It can innately cast the following spells, requiring no material components