Hi Dmitriy,
it's possible to create a password protected storefront. To accomplish this you have to modify your theme (using liquid and the customer object).
The invitation part is not standard Shopify functionality but it should be possible to create an external App which takes care of inviting friends (and limit it to 5 invitations). See API docs on creating customers.
HJ