docker – Chromedriver stop running in a celery beam scheduler after one time of execution

I am just trying to upgrade my local web scraper in python to a celery periodic task, so I won’t have to manually execute it every day.

I’ve set up a celery image with chrome/chromedriver installed through dockerfile.
The beam works fine except that it directly jumped to the end the moment it finished processing the first chrome workload.

My scraper function is like (scraping info and pack them into a dict) :

def scrape(code):
    …
    chrome_options.add('--headless')
    chrome_options.add('--no-sandbox')
    driver = webdriver.Chrome(chrome_options=chrome_options)
    driver.close()
    driver.quit()
    print(dict)
    return dict

my task is defined like (master_tasks.py):

@app.task
def taskSender():
    print('<<<----Task Start---->>>')
    (scrape(code) for code in codes)
    print('<<<----Task Complete---->>>')

Yet docker-compose up gave:

<<<----Task Start---->>>
{'pe_ttm': '17.09', 'total_mv':'320.41'}
Scheduler: Sending due task master-beat
Received task: tasks_proj.master_tasks.taskSender

So, this the weirdest part, with that dict printed out, chrome works fine but only executed once. Rest of the task seems to get ignored. What did I miss?

Environment: Azure Centos 7 docker container