Is it possible at all to define member permissions to certain custom content & member fields?

Here’s a demo situation with 4 member groups:

1 Regular Members
2 Content Submitters (submitted items become pending)
3 Content Moderators (pending items will be declined or approved by this group)
4 Administrators

Let’s pretend we have a custom ‘remarks’ text field for both content & members.

Regular Members should not be able to see the content & member remarks.
Content Submitters should only be able to fill out content during content submissions and are not able to see member remarks.
Content Moderators should only be able to read both remarks.
Admins should have read & edit access to the remarks.

EE does have a separate member field permission option “Is field visible in registration page? If your choose no, it will only be visible to admins” but that does only work for admins, and doesn’t exist at all at the custom content field options.

Are there any workarounds possible to avoid these limitations?

Have you tried conditional variables? You could assign each type of member to a group and then check with a variable to show/hide content.