You have a misconception of what a site map is.
The site map is used to audit the site crawl by the search engine bot. The site map and the crawling of a site are two different and independent things. Google will continue to track your site regardless of any sitemap. The site map will be used to audit / see if Google can properly track your site. For example, if pages are found on your site map and Google has not seen the page, Google can add the page to the search queue to include it.
The opposite is not true. If a page is not found on the site map, Google will not remove it from its index. Why? Because Google found it crawling the site.
What he seems to believe is that the site map is the total authority that Google uses to know which pages exist on a particular site. This is not the case. The tracking is. The site map only helps Google know Yes they can correctly track your site and, otherwise, what pages are missing from Google that should be added to the search queue.
Hope Google no longer attempts to access the pages because these pages are no longer on your site map is incorrect. Sitemaps are cached and only checked periodically. Why? Because it is an audit process.
You have a real problem that you must solve.
It is returning a 500 error for pages that are not found. This is bad. Your site should return a 404 error not found. Error 500 is a system error and Google will treat the condition as temporary. If your site returned a 404 error, Google will continue testing the page for several attempts over a period of time until you decide that the page no longer exists. If possible, you want to issue a 410 Deleted error for the pages you have deleted. If this is too much work or is not possible, 404 will equal the same over time.
You need to correct your 500 error.