I need to design and write a program (which I have already written) to act as an intermediary between our CMS and an Internet SMS sending / receiving platform. I have created a functional prototype, but unfortunately it is very fragile and based on bad practices, such as different modules that write / read from text files, and so on. I have requirements that seem to create more problems when I implement them. Every time I review the code, I feel that it can be improved in many ways and I am beginning to question my general design options.
Some basic concepts:
This program needs & # 39; listen & # 39; a table in the database for any new row inserted. When a new row is inserted, you must extract the relevant data and send it to the SMS platform API through the Python request library. Similarly, you must do the same on the other side to receive.
I chose to do this with a couple of modules: a sender module, a receiver module, a jobHandler module (queue jobs) and a main module. The main module simply exists to start the other 3 modules with (as it is currently) multiprocessing.
The challenge comes from designing this so as to unite everything correctly. As it currently works, the sending module does not queue the jobs, it simply checks the database every 1 second and stops verifying when it sends a job, so I don't know what will happen if 2 jobs are sent at the same time.
For this reason, I decided it was better if I implemented multiprocessing or threads and a dedicated queue module … something to listen to new rows and queue them.
When I try to implement this, I face the challenge of sharing a queue object between separate processes … and so I began to question my general design choices.
Then finally my question.
How would you design something like this? How many modules would you use and implement multiprocessing or just make it work linearly?
I feel that I must be doing something fundamentally wrong with the amount of alternative solutions that I have been implementing for this to work. I have even considered putting all the functions / code in a single module, so I don't need to import anything between modules and I only have a large unreadable module hahaha.
If I used multiprocessing, I would choose to have 3 separate processes, one to send, receive and queue jobs … but then the challenge of sharing objects between these processes.
Any suggestions are appreciated.