Twitter: do not backfeed from blocked accounts

[kylewm]

Discussed yesterday in IRC, we should avoid backfeeding tweets from blocked users. I confirmed that the Twitter search API returns tweets from blocked users with no indication that they are blocked.

The blocks/ids endpoint gives up to 5000 ids at a time, and is rate-limited to 15 calls per user per 15 minutes. (the largest shared blocklist I know of has 10k accounts on it)

• proactively filter out blocked users when searching for replies
• on individual tweets, serve 410 tombstone so that mentions from blocked accounts will be deleted
• periodically check for old responses from accounts that are now blocked, and re-propagate them.

[snarfed]

interesting! i did indeed see the IRC discussion. thanks for writing it up!

i have to admit, i'm a bit skeptical of the ROI on this one. i definitely do see the value and @kevinmarks's request, but i'm not quite sure the overall impact will be worth the work and added complexity. i'm open to discussion though!

(i'd also be curious to see what the comparable features in other silos are, and how much work it would take to support them too.)

the IRC discussion also covered the key point that webmention recipients (both CMSes and services like @voxpelli's great https://webmention.herokuapp.com/) also need to support at least manually deleting webmentions too, if not blocklists as well. i expect a common case is that you get a bad wm from a new person, so you then add them to your blocklist and also need to remove the bad wm. yes, bridgy could detect blocklist changes, go back and search for all wms from that user, and resend and 410 them, but i'm definitely skeptical of the ROI on that. (not to mention i suspect 410 support among wm recipients is far from widespread.)

sorry for the negativity. definitely still worth discussing more!

[snarfed]

btw, i'm painfully aware that i'm privileged and don't get harassed online - and deliberately don't participate in [redacted controversial topic] - so i'm probably biased. i'm trying hard to balance that against my usual judgment process, but that's tough. feedback is obviously welcome.

[voxpelli]

An alternative solution would perhaps be to translate the Twitter blocklist into an IndieWeb blocklist that could be consumed by eg. my endpoint to auto-block certain content (preferably with Pubsubhubbub-support to ensure it stays up to date in realtime). That way the solution will work even if a user starts to maintain its own Silo-independent blocklist.

True IndieWeb WebMentions can never be hidden from the source – they always have to be hidden through the target – so for Brid.gy to bridge also block-data to the IndieWeb it should probably expose that data in a way that non-silos can do as well.

Closest related ticket at my endpoint is: voxpelli/webpage-webmentions#28

Will add one for consuming block lists.

Related wiki-pages:

• http://indiewebcamp.com/block#Brainstorming
• http://microformats.org/wiki/block-list

[kevinmarks]

I like that idea, voxpelli - also translating those you follow to a whitelist may make sense. the challenge is that these are supposed to be non-public, so we'd have to do a confirmation between the services that it is the end-user requesting it, which is delegated auth etc.

[voxpelli]

If one wants a block list to be possible to be both public and private (Twitters are currently private so any block lists from twitters should probably be private when exposed through Bridgy) then using the brainstorm documented here would be useful: https://indiewebcamp.com/private_posts#Public_Page_Upgrading

Expose a page that can be upgraded to private access through an IndieAuth token and then possibly allow the client to subscribe to it using Pubsubhubbub (or else expect a client to poll it every now and then).

The exact markup and discoverability of a block list is still a bit of an unknown but I would say that as its a blacklist it's the opposite of the markup of the people one follows, which one typically mark up with XFN, so it would look like XFN, but have other relations.

On the discoverability side I don't know how to really do it – typically XFN-data is discovered in a users profile and one traverses rel-me relations to find more profiles and on them one finds more XFN-data. But as rel-me data and the identity graph can grow rather large such a solution would require a crwaler like relspider to manage – so one would probably have to add some specific relation data do make such a Bridgy-block page discoverable – not sure what that would be though – some property of a user's h-card probably – u-blocks (and perhaps a reverse u-follows if one wants to make a whitelist as discoverable)?.

[voxpelli]

After some brief discussion with tantek I would say u-block-list and u-follow-list, prefixed with x-, would be the more right way to start experimenting with this: http://indiewebcamp.com/irc/2015-09-15/line/1442345907245

If @kevinmarks could add such a link to his h-card then I could discover that on my side and ask him if he wants to enforce that block list on my endpoint and then start consuming and enforcing it.

[snarfed]

thanks for the brainstorming @voxpelli!

if i understand all this right, the asks for bridgy are:

1. fetch a twitter user's block list and translate it to mf2
2. serve that mf2 on a private, access controlled page
3. grant access based on... indieauth token?
4. bonus: let consumers subscribe to changes via PuSH

I'd happily consider adding #1 to granary. the rest are unlikely at best, at least for me. if someone else wants to add them to bridgy, I'm open to that.

thanks again!

[voxpelli]

@snarfed: That sums it up pretty well.

As block lists are private on Twitter, exporting them to something public seems like something one would like to avoid. Are you thinking that Granary would expose it privately or publicly?

The IndieAuth-token check should be pretty simple to implement – not sure about the relation between Granary and Bridgy and where one would add it though.

[snarfed]

agreed, granary definitely can't expose private block lists (or anything else private) publicly.

the first step is just to add it to the granary library. the next step is exposing it via REST and the UI in exchange for a twitter access token, which is nice but not enough for you, since you don't have twitter credentials for your users.

the problem is that granary doesn't store twitter creds either (or anything at all). we'd need a service that does store twitter creds and check IndieAuth tokens (or whatever). I'd love for that to be something else, but right now the only candidate i know of is bridgy...unless you or webmention.io want to start collecting and storing twitter creds? :P

[snarfed]

i thought about this a bit more and filed snarfed/granary#40. a good first step could be to do just public block lists. we'd add granary support, then serve them from bridgy since it has users' silo credentials. probably no UI (or minimal) in bridgy, just a per user endpoint.

this isn't a priority for me personally right now, but I'd love to see it happen!

[snarfed]

re the last comment: twitter doesn't actually support public block or mute lists. they're all private. so the MVP use case for this is bigger and has more open questions than we (i) thought.

[snarfed]

more importantly, if you block someone on twitter, they can't see your tweets at all, so they can't reply, favorite, or retweet them. which leaves just @-mentions that bridgy would still need to block.

(apologies if all this was already obvious. i hadn't paid much attention to blocking before this. privilege check, etc.)

[tantek]

This is currently a blocker (no pun intended) for me implementing showing any kinds of comments (including from Bridgy via Twitter), and likely even reposts/likes (because I'd really like to avoid having a direct link from my site/permalinks, even in a responses display section, to e.g. a hate/abusive profile on Twitter) - and this happens with many (most?) POSSE tweets of mine (sometimes "just" sensitive images rather than outright hate but still :/ )

Upstream I could make a feature request to webmention.io to filter out Twitter responses (for now), or barring that the barrier is raised (for me) even higher to "just" directly handle receiving webmentions myself, and write my own filtering code.

However I'd prefer to help push to make these services usable by more devs, which is going to require blocking support for all but the smallest / most privileged accounts that see little abuse on Twitter.

[aaronpk]

I just tested this via bridgy to confirm, tho I guess I should have just taken @kylewm's word for it. Bridgy currently does send webmentions even if I've blocked an account. https://brid.gy/log?start_time=1500407904&key=aglzfmJyaWQtZ3lyNQsSCFJlc3BvbnNlIid0YWc6dHdpdHRlci5jb20sMjAxMzo4ODc0MDA0MTc5MzU1MDMzNjEM

[snarfed]

ok, so the task here is at least the first two items in @kylewm's original issue. (he was right all along! :P)

• proactively filter out blocked users when searching for replies [..., links, @-mentions, etc]
• on individual tweets, serve 410 tombstone so that mentions from blocked accounts will be deleted

we're very unlikely to do the third:

• periodically check for old responses from accounts that are now blocked, and re-propagate them.

...but fortunately aaronpk/webmention.io#85 should have the same effect, and is maybe more likely to actually happen.

[tantek]

Note: there's the historical aspect also. E.g.:

1 receive an abusive @-reply via Bridgy backfeed
2 block that user on Twitter
3 "refresh" (automatically?) Bridgy to update its cache of my Twitter blocklist
4 Bridgy notices any new blocked users, and 410s all previous (historical) webmentions from them, preferably resending webmentions (webmention deletes) to cause those webmentions to automatically get deleted by the receiver (e.g. a webmention proxy like webmention.io)

If that's different / harder enough to merit a separate issue, ok with splitting that out. From a "UX of the service" perspective, it feels like it should "just work", as this is how for example blocking a user on Flickr works (it removes their past comments and favorites from all your photos).

[snarfed]

ok, this is mostly done. bridgy now fetches twitter block lists (up to hourly, cached beyond that), ignores responses from blocked users, and 410s their translated tweets and responses. example: https://brid.gy/post/twitter/schnarfed/510676816084754432

resending all past webmentions from newly blocked users (#473 (comment)) is a worthy goal, but nontrivial, and has open UX questions. i've split it out into #762.