Many apps nowadays use phone numbers for verifying new accounts. Obviously, it is far from impossible for a malicious user to get a new phone number if they care enough, but it does usually have a monetary cost, which will keep a significant portion of malicious users out. This also can help prevent some forms of fraud.

This is somewhat obvious, but take care in designing social interactions on your app, and introduce easy reporting systems and user-controlled blocking systems (e.g., even if you don't respond to a user's report in a timely manner, a user should still be able to block communications from another user using a button in the app). You may want to allow users to choose to only allow messages from users who have been on the site for a certain amount of time, or have some kind of additional level of verification. Also consider introducing active, paid moderators who actively respond to reports as they come in, if your budget allows.

Allowing users access to open text fields on the internet is inherently dangerous. I have tried to avoid it in all of my apps (for example, I've made games that use randomly-generated user names rather than allow users to input their own, and I only added user comments to an app I was building _after_ I'd built a user reporting system and ensured I got an immediate alert on my phone if a user sent a report). It is your responsibility to try to design your app, and moderate your community, to mitigate these risks.

> a user should still be able to block communications from another user using a button in the app

Adding to this - when you do act, your ban should be that all users are effectively blocking this user, ie a shadow ban.

That way the bad user, to themselves, still seems to be interacting with the site, just nobody else seems to care.

One nice moderation trick I've done for my company is building a slack integration that posts new reports into a dedicated channel, each of which has a "ban user" button. This way onboarding new moderators is as simple as adding them to the channel, and you automatically get push notifications and access from all of your devices with configurable controls (i.e. you can mute the channel when you're busy w/o having to build this functionality yourself).

We do the same for moderating user photos (they get posted into slack with a "remove" button) as well as for other moderation features. I highly recommend this approach for when the volume of reports is manageable.

The “burner” app makes it trivially easy to get a new phone number. Many other apps do the same.

That’s not really a barrier for anyone who cares to give a moments thought to how they can get around that limitation.

This is the correct answer. User-controlled blocking systems will work well.

I'm not sure it's applicable for you since it's a mobile app but I encountered the same issue with a single specific user, which would cause chaos everywhere.

The right answer for us has been to make the website utterly annoying for him, slowing down pages, disconnecting him randomly, removing the styling randomly, throwing a 500 page randomly, his experience was truly miserable.

There was a plugin to do this on Vbulletin forums.

It was somethink like a shadow-ban but where you would give random timeouts/slow loads to the user

"Tachy goes to Coventry" was the first implementation I know of. And I can't believe I remembered that off the top of my head.

I did the exact same thing on a website I used to own ~2011-2012. I went one step further and occasionally redirected the user to sites people generally shouldn't visit through a url shortner. The user disappeared within a few hours.

Fraud and abuse prevention really boils down to something like this.

You don't need to be perfect with blocking these folks, but you need to be better than your competitors so that the cost/benefit balance shifts to encourage the user to go elsewhere. Unfortunately trolls don't disappear, they just go somewhere else.

Similar to the saying about the hungry lion. If a hungry lion approaches your group while hiking, you don't need to be faster than the lion, you just have to be faster than at least 1 of the other people in your group.

That's pretty passive-agressive...

Genius

Here are some strategies for IOS:

1) use device identifer / advertising identifier to identify the user and block them. problem: user can reinstall app and these identifiers are recreated

2) store a random identifier on the disk or in nsuserdefaults or some other app storage. problem: user can reinstall app and these locations are wiped

3) store a random identifier in icloud keychain. my understanding is this doesn't have the same problem as above but its possible for user to wipe this.

4) use apple device check API. https://developer.apple.com/documentation/devicecheck this allows you to associate 2 bits of information with a users device. you can use 1 bit to control whether the device should be banned.

5) use a third party device fingerprinting solution or roll your own device fingerprinting. my understanding is there is enough entropy on the phone to uniquely identify it. for example you can find out what time the original OS was installed in the Foxconn factory. i think apple should ban all apps that do this but a lot of apps do this. :(

Possibly off-topic ...

Can you provide a reference for “find out what time the original OS was installed in the Foxconn factory”?

You can stat files in the filesystem. We found some apps were stat'ing some files and these had different dates on different devices and they seemed to be around the time the device would have been manufactured. I can't remember off the top of my head the what the names of the files were.

You can also determine when the last update was installed by stat'ing files in the filesystem. But this is less useful for fingerprinting a device because it is prone to change.

App developers are come creepy people themselves.

So, a slightly sketch thing to do is to shadowban the user, so that they don’t know that nobody can see their posts. They might get bored. You might need to modify some of your application code though. You can do this by their IP if they haven’t caught on.

Something I've seen people do in the past on older forums is to degrade the experience for the offending user as well, adding things like load speed delays etc. The idea being that they get frustrated and eventually move on. It's also somewhat sketchy but it was quite popular in forums / bulletin boards.

I'm not sure it's that sketchy if the behaviour is abusive. The problem with shadowbanning, although equally true of other options such as degrading experience, is that it's open to abuse by site owners.

I was going to suggest the same thing. If you ban malicious users, they will come crawling back, almost as if breaking your system is a challenge.

Are IPs really that effective for isolating users? VPNs, Tor, and even just Dynamic IPs issued by local ISPs made me always feel like that's a poor way to ban malicious users.

Someone regularly posting lewd content might be used to getting ignored. The user might not even notice. An even more sadistic trick would be to make it random, just enough that they think you're censoring certain words in messages but not the user.

Shadowbanning in its different forms would be the best option, don't let the user know they have been noticed.

Let them think they sent messages and comments to other users, but don't actually deliver them.

I can't count the number of times I clicked on a dead comment's user profile only to see a shocking number of dead comments, sometimes going back months (years, even?). Out of some morbid curiosity, I suppose, I'd just keep clicking on 'more' and regularly the number of comments would be in the hundreds.

One the one hand, this approach seems very effective, but I can't help but feel pity for this person and even some sense of cruelty if I were to implement such a feature myself (although I probably would).

Of course, I'm also the person in real life who will eventually 'engage' with that "one person" in a group who everyone has been tuning out for a while. I don't get the impression doing so matters to said person, because clearly they're oblivious, but at least it makes me feel better or less guilty.

Yes, shadowbanning is the best idea.

If you really want to put some work in, generate some automatic replies you show them, so they think they are getting brief, boring replies.

Shadowbanning is creepy. Might cost you dedicated users over time.

Captchas. Do not block him, but make him solve a captcha if you detect a bad word. He'll lose the fun.

Took idea from this talk https://www.youtube.com/watch?v=0kBhvM9OIaI

Don't block them, degrade their performance to annoy them out of your platform. Or treat them the way you would any spambot,add registration difficulty,cumulative but minimal interaction cost(performance,financial,points,etc...) and as a last resort ban or set restrictions on their connectivity network (subnet,isp,email provider, vpn provider,asn,country,etc...)

Shadowbanning is also an option: they'll think they are sending messages to other users, but those other users will never see them.

We need a thing whereby a person can only sign up to a service once - without disturbing the on-boarding process (too much). Maybe a contradiction in terms but I'm putting it out there since there are some mighty clever people in here, and someone may just have a solution.

Login with Apple, Facebook, Google, Microsoft etc...

Let them handle account creation and detect misuse.

So, when a user only gets as many tries to be a jerk as there are oauth services? Could work some extent, but not optimal imo.

Is it that hard to create an additional Google account?

no you're right. The idea doesn't work at all.

How about creating rules?

  A talks to B
  B blocks A
  A registers as C
  *not too much time passes*
  C talks to B

At this point C is considered to be a harasser and therefore it will be hidden for B. B can of course lift the ban if it's just a false positive. To further specify who's who in this graph, the logic can find the smallest time between ban and the creation of a new user with the same behavior (writes to B) as the reported one.

I think this is way more easier than handling those who abuse the actual service you're selling.

Make it very easy for people to report these messages, then put in a temporary block on a user after the first report, until it can be checked by a human.

Related question for folk in similar situations: What are your experiences bringing these sorts of abuse to the attention of the ISPs?

What if they are just annoying enough to want them off your app, but not quite bad enough for their ISP to turn off their account? My guess is that most cases would fall into this category, and only a rare few would be so bad that an ISP would actually take action.

On your app, you want them off if they are a jerk. The ISP doesn't care about that level misbehavior.

I'm all for brushing off a transgression, but once a user starts ban evading then I would side with reporting them.

On top of what many users have stated here, shadow blocking! If the malicious user doesn't know that they're blocked, they'll keep using their account. If you shadow ban them, you just don't show their comments and posts to anyone else than themselves and they can't harm anyone.

Slightly related, does anyone know any solutions to this for a public slack group? I'm a member of one that has a public invite link (temporarily disabled) and it's constantly plagued by one troll account that comes back whenever the invite link is enabled again.

Additional to this, can some one help with this -

* Under GDPR, user can request for it's account deletion where we have to delete all the user's reference. If a user signup again there would be no way whether it is the same malicious user, how to overcome this?

You can claim legitimate interest and keep certain information that would allow you to reidentify them for a reasonable amount of time (2 years?).

I haven't got a solution here, but it's extra friction for them to do this, you have a time window for processing and not all trolls will bother, so you would still have a better experience for other users by implementing it

You can still keep information.

Lie about it. Let them try to prove you did what they asked.

User ‘about’ checks out

GDPR penalties are steep, I'd highly recommend not doing this. You only need one lawyer with too much free time to test you and ruin your life.

No one has ever been ruined by GDPR.

So, you're banking on not being the first person to get caught?

To be fair I don't think it'd be a bad idea to have some wordfilters for sexual language etc, and if detected, not necessarily autoban but flag for a moderator to review.

Depending on what the app is for there's probably some nuance needed to that approach - you want to catch people harassing other users early, but you also absolutely don't want to be snooping through legitimate conversations of that nature between two consenting users. I guess its less of an issue if you're at the sort of scale where you have external moderators, but most things like this start off with a few staff members and maybe some community members doing the moderating.

rate limiting by ip, force recaptcha, shadow banning the user

Are there any SDK or APIs which can help do this?