One way would be to define purchase groups by each individual buyer. That way each buyer gets the authorisation to view/ modify/ create purchase orders for his purchase group only
(You will have to create one role per purchase group)
The other way is to define the authorisation check per user ID in BADI BBP_AUTHORITY_CHECK, method AUTHORITY_CHECK_PRODOC