After some reading and asking I have come up to this:
Yes, if you want your users to sign up with their Facebook or Google account, you call the API, get the e-mail address (it is even easier with Google麓s AccountManager on Android), send it to your server that will save the e-mail address, associate a userID and generate it麓s own access code. The access code will be sent back to your client app to store it for later use. Whenever the user wants to do some operations, the server API will be called with the user麓s e-mail address and access code and you can be sure it really is the user. It is much harder to call the API from outside and guess correctly both the e-mail address and access code so it is somewhat safe.
Since the Facebook login is only used to authenticate the user, which means to only verify that the user exists and has an account, we don麓t actually need the FB accessToken. We would need the FB accessToken only for API calls for Facebook server, so for example when we want to retrieve a list of user麓s friends and so on. In this case, you can get the active session that is provided by Facebook SDK and get the accessToken from there.
This case is again pretty simple.
If you only use Facebook login to authenticate the user, you don麓t care if the user deletes his account in the future. After the first login, you save his email address, possibly a picture and no longer care about his facebook profile.
If you use the Facebook login for getting a friends list and so on, you do not keep this type of data in your local storage anyway, so as soon as the user deletes his account, he loses his friendlist as well. Or, you can keep his friendlist and try to update it everytime he uses your app and once he deletes his account, the friendlist stops being updated and it is, again, up to your user not to use the Facebook account. The last idea would rather suit game apps use cases....just an idea, not anything officially accepted as the best.