Which git workflow for micro-services and testing

I would like to know what best practices exists out there with regard to micro services, CI/CD, testing and Git workflow, especially the way of working given the constraints that

a) nothing should block a prodfix, and

b) some commits/features need manual testing.

c) PR process

A little more detail.

a) Given the need to do a prodfix, nothing should block that due to testing of other features already in the pipe (merged), or potentially deploying the previously merged untested code in order to get the fix out quick.

b) Often, a change in one service have to be manually tested. Ideally, that would be in an isolated environment with all the services we depend on and enough data to do meaningful testing.

c) For several reasons, other developers need to sign off on a feature before merged.

I see two potential solutions, listed below. I’m interested to see if there are alternative solutions used by others. Also, I’m looking for a somewhat short-term solution to remedy the immediate problem.

  1. Test a feature branch in an isolated environment. Drawbacks is that it requires a lot from an on-prem solution to get dependencies (services) and good data up in a quick way.
  2. Have a development branch and a prod branch. A prodfix would go to the prod branch where normal features are tested on the development branch in a shared test environment (no need to create an environment per feature branch).

I fail to see a solution with the following:

  • Using one master branch that’s deployed in prod without having the possibility to test the feature branches in an isolated env before being merged.
  • Given the contract (API) of a micro-service, keep all tests automatic or completely remove the manual testing and merge once the tests are green. There’s a good coverage of unit and integration tests, but manual testing is stil needed. There’s some GUI code, and manual testing also verifies the logic that might have been misunderstood by the developer.
  • Use feature-flags when developing and deploy “dead” code which later can be released by toggling the flag. Even though the intent is to deploy code that’s not finished, we need testing the ensure that the code isn’t run, that the feature flag is working, etc. and that removes the point with feature flags. They would be useful if our feature branches were kept for weeks and we experience merge issues, but that’s not the case.

What do you use?