8 – How can I invalidate cached field access grants for anonymous users?

We have a Drupal 8 application that uses an event subscriber class to validate IP addresses against a table of white-listed addresses on kernel requests, and adds that value to a session variable. Basically, anonymous users with authenticated IPs can access privileged content on the site. I can use hook_node_access and hook_block_access to control access when appropriate, but hook_entity_field_access only fires once after rebuilding the caches, and not for each session, so field access seems more difficult to control this way. I’ve thought about injecting a service into my event subscriber and adding the necessary logic there for field access control, but I’m wondering if there’s something easier that I’m missing about invalidating cache tags that would cause that hook to fire for each session.