twitter: /statuses/retweets.json returns 403 Forbidden

[snarfed]

for https://brid.gy/twitter/iwontsignuphere , every poll. other API calls beforehand succeed. not sure why. we're incorrectly (maybe?) interpreting it as rate limiting.

example log:

Params: NestedMultiDict([('last_polled', u'1970-01-01-00-00-00'), ('source_key', u'aglzfmJy...')])
Source: iwontsignuphere (Twitter) iwontsignuphere, http://brid.gy/twitter/iwontsignuphere
Last poll: http://brid.gy/log?start_time=1469094672&key=aglzfmJyaWQtZ3lyHAsSB1R3aXR0ZXIiD2l3b250c2lnbnVwaGVyZQw
Updating iwontsignuphere (Twitter) /twitter/iwontsignuphere : {'poll_status': 'polling', 'last_poll_attempt': datetime.datetime(2016, 7, 21, 10, 52, 18, 474590)}
urlopen GET https://api.twitter.com/1.1/search/tweets.json?q=%22julieannenoying.com%22&include_entities=true&result_type=recent&count=50 {}
urlopen GET https://api.twitter.com/1.1/favorites/list.json?screen_name=&include_entities=true {}
urlopen GET https://api.twitter.com/1.1/account/verify_credentials.json {}
urlopen GET https://api.twitter.com/1.1/statuses/user_timeline.json?include_entities=true&count=50&screen_name= {}
urlopen GET https://api.twitter.com/1.1/statuses/retweets.json?id=744651175199383552 {}
Error 403, response body: {"errors":[{"code":"200", "message":"Forbidden."}]}
Error 403, response body: {"errors":[{"code":"200", "message":"Forbidden."}]}
Error 403, response body: {"errors":[{"code":"200", "message":"Forbidden."}]}
Error 403, response body: {"errors":[{"code":"200", "message":"Forbidden."}]}
Rate limited. Marking as error and finishing. HTTP Error 403: Forbidden
Updating iwontsignuphere (Twitter) /twitter/iwontsignuphere : {'poll_status': 'error', 'rate_limited': True}
Added poll task 62356658033700949421 with args {'countdown': 80642.525166884909}

[snarfed]

error code 200 isn't listed in https://dev.twitter.com/overview/api/response-codes , and https://dev.twitter.com/rest/reference/get/statuses/retweets/%3Aid doesn't mention this error.

however, https://twittercommunity.com/t/protected-tweets-with-statuses-retweets-id/12903/4 has lots of other people reporting the same thing: /statuses/retweets.json returns the same error for a protected account.

my best guess is this is intentional for protected accounts, so i should catch it and handle it.

[snarfed]

confirmed, same things happens with another protected account, snarfed_org: /statuses/retweets/701173347606880256.json returns 403 {"errors":[{"code":"200", "message":"Forbidden."}]}.

[snarfed]

fixed by snarfed/granary@34a7bcb

[snarfed]

happening again, but an odd variant, for ianvellosa. example log below.

we fetch his tweets with /statuses/user_timeline.json, as usual, and then we fetch retweets for tweet id 262967438101069825, which returns HTTP 403 with error code 200, same as above. oddly though, that tweet isn't in his timeline! i don't understand where we're getting it.

no matter i guess, fix is probably to just handle HTTP 403 + error code 200.

2019-08-12 20:25:06.373158 I Source: ianvellosa (Twitter) ianvellosa, https://brid.gy/twitter/ianvellosa
...
2019-08-12 20:25:06.789362 I urlopen GET https://api.twitter.com/1.1/statuses/user_timeline.json?include_entities=true&tweet_mode=extended&count=50&screen_name= {}
2019-08-12 20:25:07.059948 I urlopen GET https://api.twitter.com/1.1/statuses/retweets.json?id=262967438101069825&tweet_mode=extended&since_id=976198456070037504_favorited_by_20429282 {}
2019-08-12 20:25:07.127041 W Error 403, response body: u'{"errors":[{"code":..."message":"Forbidden."}]}'
2019-08-12 20:25:07.145029 I Updating ianvellosa (Twitter) /twitter/ianvellosa : {u'poll_status': u'error'}
2019-08-12 20:25:07.226816 E HTTP Error 403: Forbidden
Traceback (most recent call last):
...
  File "/base/data/home/apps/s~brid-gy/background:8.420268296435611139/tasks.py", line 95, in post
    self.poll(source)
  File "/base/data/home/apps/s~brid-gy/background:8.420268296435611139/tasks.py", line 161, in poll
    min_id=source.last_activity_id, cache=cache)
  File "/base/data/home/apps/s~brid-gy/background:8.420268296435611139/models.py", line 348, in get_activities_response
    resp = self.gr_source.get_activities_response(**kwargs)
  File "/base/data/home/apps/s~brid-gy/background:8.420268296435611139/local/lib/python2.7/site-packages/granary/twitter.py", line 351, in get_activities_response
    tweet['retweets'] = self.urlopen(url)
...
HTTPError: HTTP Error 403: Forbidden

[bright1993ff66]

Does anyone have the same problem? I am using Tweepy 3.5.0 and having exactly the same problem...

I just wrote the codes to save the forbidden tweet ids...

# The dataframe contains the tweet id you are interested in
for index, row in dataframe.iterrows():
    tweet_id = int(row['id'])
    
    if tweet_id in processed_tweet_network:
        print('The tweet {} has been processed'.format(tweet_id))
        pass
    else:
        print('Coping with the {}th tweet: {}'.format(index+1, tweet_id))

        while not tweet_id in processed_tweet_network:
            try:
                results = api.retweets(tweet_id)
                name_list = []
                id_list = []
                for result in results:
                    retweeter_name = result.author.name
                    retweeter_id = result.author.id
                    name_list.append(retweeter_name)
                    id_list.append(retweeter_id)
                retweeters_id_list.append(id_list)
                retweeters_name_list.append(name_list)
                processed_tweet_id_list.append(tweet_id)
                processed_tweet_network[tweet_id] = 'Done'
                # sleep for 5 seconds to reduce the probability of encountering the rate limit issue
                time.sleep(5)
            except tweepy.error.TweepError as e:
                print(e.response.text)
                if e.api_code == 88:
                    print('The rate limit has reached!')
                    current_time = datetime.today()
                    print('The current time is: {}'.format(datetime.today()))
                    future_time = datetime(current_time.year, current_time.month, current_time.day+1, 0, 0, 1)
                    print('Hence we will sleep for {} seconds'.format((future_time - current_time).seconds))
                    time.sleep((future_time - current_time).seconds)
                    print('We could start again!')
                elif e.api_code == 200:
                    print('The access action is forbiddened...')
                    retweeters_id_list.append(['Forbiddened'])
                    retweeters_name_list.append(['Forbiddened'])
                    processed_tweet_id_list.append(tweet_id)
                    processed_tweet_network[tweet_id] = 'Forbiddened'
                    time.sleep(5)
                else:
                    print('This tweet could not be accessed!')
                    print('The api code is: {}'.format(e.api_code))
                    retweeters_id_list.append(['Not authorized'])
                    retweeters_name_list.append(['Not authorized'])
                    processed_tweet_id_list.append(tweet_id)
                    processed_tweet_network[tweet_id] = 'Not authorized'
                    time.sleep(5)