I stole most of the answer from Andreas's book 🙂
The bitcoin developer's documents provide this link here, which goes from P2SH to multisig. The idea is to create a so-called redemption script, which is a ripemd160 (sha256 (sigscript)).
Example of the book:
Redeem script: 2 PubKey1 PubKey2 PubKey3 PubKey4 PubKey5 5 CHECKMULTISIG Block script: HASH160 <20-byte hash of redeem script> SAME Unlock script: Sig1 Sig2
see Andreas' book "Masterig Bitcoin, 2nd edition", page 152 and following or in the GitHub version here.
The actual book at http://chimera.labs.oreilly.com/books/1234000001802/index.html
how to create a multiple signature address and redeem a script without using
bitcoind? What algorithm is used?
You can use any hashing tool or libraries, show Andreas with the "bx" library, and also openssl.