The problem, of course, is who builds/maintains this, how do they get paid, and how do you pay the server fees.

The app would run on the user's local PC or phone, and it would access data directly from their mailbox (and create a local DB from that data). So no backend server to worry about.

Without a third-party managing associations, how do you maintain that "friendship" is consensual? In other words, how do you prevent "I have your email address" from meaning "you're my friend now."

When you send a friend request to someone, their stuff is shared with you only if they accept the request on their end. If they reject the friend request, then anything coming from you that they don't want can be handled as spam (the client side software would just filter the content itself in that case).

Almost all email clients have some sort of filtering. And the email messages would all have a keyword in the subject line, so filtering could be accomplished by that (or via an x-header). The app itself could automatically create the filters for various web hosted email systems too.