optimize reddit/PRAW

[snarfed]

Our use of the Reddit API, via PRAW, is pretty badly inefficient right now. Not usually a big deal, but got a lot more noticeable recently due to https://brid.gy/reddit/lgats, which backfeeds a few big sites that get linked and mentioned on Reddit a lot: https://sec.report/, https://uspto.report/, https://lei.report/, https://fccid.io/.

Two obvious problems right now. First, PRAW repeats user lookups a lot. Example poll log:

D 2021-03-10 15:46:13.897607+00:00 Params: [('source_key', 'agdicmlk...'), ('last_polled', '2021-03-10-09-24-49')]
I 2021-03-10 15:46:13.915966+00:00 Source: schnarfed (Reddit) schnarfed, https://brid.gy/reddit/schnarfed
I 2021-03-10 15:46:13.916355+00:00 Last poll: https://brid.gy/log?start_time=1615368289&key=agdicmlkLWd5chULEgZSZWRkaXQiCXNjaG5hcmZlZAw
I 2021-03-10 15:46:13.931726+00:00 Updating schnarfed (Reddit) /reddit/schnarfed : {'poll_status': 'polling', 'last_poll_attempt': datetime.datetime(2021, 3, 10, 15, 46, 13, 916446), 'rate_limited': False}
D 2021-03-10 15:46:14.007187+00:00 Using ETag None, last activity id uouxi
D 2021-03-10 15:46:14.128411+00:00 Fetching: GET https://oauth.reddit.com/r/all/search/
D 2021-03-10 15:46:14.128634+00:00 Data: None
D 2021-03-10 15:46:14.128711+00:00 Params: {'q': 'site:"snarfed.org" OR selftext:"snarfed.org"', 'restrict_sr': False, 'sort': 'new', 'syntax': 'lucene', 't': 'all', 'limit': 100, 'raw_json': 1}
...
D 2021-03-10 15:46:14.861554+00:00 Fetching: GET https://oauth.reddit.com/user/milkbone20/about/
D 2021-03-10 15:46:14.861680+00:00 Data: None
D 2021-03-10 15:46:14.861738+00:00 Params: {'raw_json': 1}
D 2021-03-10 15:46:14.950963+00:00 Response: 200 (656 bytes)
D 2021-03-10 15:46:14.951954+00:00 Fetching: GET https://oauth.reddit.com/user/gst/about/
D 2021-03-10 15:46:14.952096+00:00 Data: None
D 2021-03-10 15:46:14.952163+00:00 Params: {'raw_json': 1}
D 2021-03-10 15:46:15.057583+00:00 Response: 200 (655 bytes)
D 2021-03-10 15:46:15.058516+00:00 Fetching: GET https://oauth.reddit.com/user/gst/about/
D 2021-03-10 15:46:15.058632+00:00 Data: None
D 2021-03-10 15:46:15.058688+00:00 Params: {'raw_json': 1}
D 2021-03-10 15:46:15.152530+00:00 Response: 200 (655 bytes)

eg in that poll, https://oauth.reddit.com/user/gst/about/ was fetched five times.

Second, we're fetching every comment in big threads, even in source mf2 handlers for individual comments, which seems excessive. Example log from /comment/reddit/lgats/m74e3g/gr9rmye:

2021-03-17 13:36:07.204 PDT Fetching: GET https://oauth.reddit.com/comments/m74e3g/
2021-03-17 13:36:07.204 PDT Data: None
2021-03-17 13:36:07.204 PDT Params: {'limit': 2048, 'sort': 'confidence', 'raw_json': 1}
2021-03-17 13:36:10.915 PDT Response: 200 (138240 bytes)
2021-03-17 13:36:11.192 PDT Fetching: GET https://oauth.reddit.com/user/Pesa2w/about/
2021-03-17 13:36:11.192 PDT Data: None
2021-03-17 13:36:11.192 PDT Params: {'raw_json': 1}
2021-03-17 13:36:11.280 PDT Response: 200 (868 bytes)
2021-03-17 13:36:11.282 PDT Fetching: GET https://oauth.reddit.com/comments/m74e3g/
2021-03-17 13:36:11.283 PDT Data: None
2021-03-17 13:36:11.283 PDT Params: {'limit': 2048, 'sort': 'confidence', 'raw_json': 1}
2021-03-17 13:36:14.329 PDT Response: 200 (138198 bytes)
2021-03-17 13:36:15.197 PDT Fetching: POST https://oauth.reddit.com/api/morechildren/
2021-03-17 13:36:15.197 PDT Data: [('api_type', 'json'), ('children', 'gr9vsp0,gr9vtqu,gr9diip,gr9l349,gr9cx6y,gr9i6tp,gr9wa32,gr9zcpc,gr9r6s4,gr9et5k,gr9klqy,gra0n1r,gr9fwny,gr9lkhx,gr9tc80,gr9ruht,gr97kfa,gr9pauh,gr9sq3u,gr9p113,gr9ttm4,gra03la,gr9e5gv,gr9izao,gra0d2p,gr9mjar,gr9oiew,gr9f2nw,gr9vubl,gr9tqh3,gr9sle3,gr9xdn0,gr9uixa,gr9obso,gr9xljq,gr9qb88,gr9tduk,gr9h84d,gr9gx29,gra0hur,gr9emw4,gr9tili,gr9j8tn,gr9aagg,gr9sbxu,gr9i5bn,gr9n71x,gr9f5va,gr9h4zj,gr9s2gw,gr9zm9w,gr9ia2t,gr9vr7q,gr9xifc,gr9t7kc,gr9faml,gr9gscx,gr9q1sn,gr9igex,gr9hcwt,gr9m9we,gr9ox51,gr9ynj5,gr9fdly,gr9o90p,gr9x12j,gr9qct9,gr9psbz,gr9x2nk,gr9x12q,gr9suyb,gr9rbnc,gr9jbv9,gr9w2bl,gr9sjwi,gr9z7pn,gr9ni6g,gr9tozt,gr9fikq,gr9m21a,gr9zj67,gr9u1n5,gr9x903,gr9yh91,gr9x49j,gr9vv38,gr9zgen,gr9wezg,gr9ejdi,gr9cvry,gr9gqgv,gr9csm9,gr9jm7a,gr9v8bq,gr9n0tj,gr9l3ag,gr9a12h,gr9db2p,gr9zcvg,gr9lnu6,gr9mrcf,gr9oqgm,gr9xlo9,gr9v0fv,gr9fsgl,gr9euwx,gr9di4b,gr9bnk3,gr9vuh5,gr9irx3,gr9i8lp,gr9hnmg,gr9pfrm,gr9g6c3,gr9mwq7,gr9sqal,gra0273,gr9t4io,gr9zvvl,gr9cxer,gra0a3q,gr9pe0h,gr9ni96,gr9wt9o,gr9kffw,gr9nacz,gr9j5tk,gr9esdv,gr9hpof,gr9ml34,gr9uu6c,gr9cmdh,gr9i2bq,gr9sq27,gr9vgaw,gr9ynnt,gr9j03b,gr9l6j0,gr9o18t,gr9j13k,gr9e0yc,gr9uxcl,gr9euze,gr9l4yj,gr9n42m,gr9zuch,gr9ufz6,gr9h3kz,gr9sk15,gr9qlh9,gr9wf3m,gr9qjcj,gr9zek6,gr9v248,gr9xyex,gr9qp8i,gr9legv,gr9cg3l,gr9tymd,gr9ye86,gr9djm2,gr9x2ts,gr9wbyr,gr9yx7i,gra0k79,gr9vw6c,gr9ccyt,gr9kvjb,gr97hk8,gr9ii7e,gr9ef8t,gr9e46n,gr9yhf4,gr9z6pe,gr9ttx0,gra0ddk,gr9zp4k,gr9uk8l,gr9p4uf,gr9nxvw,gr9r8g0,gr9wv7b,gr9e11o,gr9lbd9,gr9yqxb,gr9mfw8,gr9pknm,gr9o637,gr9eifo,gr9u09i,gr9v3rt,gr9da6y,gr9tivw,gr9sdst,gr9k9gh,gr9da78,gr9kqud,gr9j5ye,gr9khd5,gr9r0tm,gr9z9wr,gr9vung,gr9o1d9,gr9rmye,gr9nqbc,gr9u516,gr9o4qn,gr9y39c,gr9bx8m,gr9ny87,gr9wah4,gr9ql1l,gr9t9fu,gr9z56z,gr9lngi,gr9pnv3,gr9nqc1,gr9ubdf,gr9s93p,gr9pxcu,gr9hd7j,gr9r2rx,gr9rjtw,gr9fkfb,gr9l00t,gr9p9o0,gr9h5bu,gr9onjv,gra0dh9,gr9sh1d,gr9mrbp,gr9pr2e,gr9qsm9,gr9he6k,gr9nsdb,gr9z0i5,gra05lc,gr9yhjl,gr9s191,gr9ffq2,gr9lz6j,gr9mrmq,gr9qt0j,gr9swv9,gr9hd9i,gr9x65j,gr9va7e,gr9xso7,gr9swvh,gr9f76s,gr9za0i,gr9vrli,gr9z6ux,gr9kqyi,gr9yb98,gr9ur9a,gr9kfwh,gr9tqwy,gr9qetg,gr9z8g6,gr9sozt,gr9tklm,gr9gmfh,gr9z0jy,gr9nfql,gr9f7v6,gr9rjtd,gr9xlzq,gr9qw7t,gr9hjmj,gr9hvf2,gr9faca,gr9ql5q,gr9h4g5,gr9xyn7,gr9avfp,gr9kpev,gr9e17t,gr9bp0y,gr9nfe6,gr9tpdb,gr9cer7,gr9epsu,gr9q27s,gr9wtkp,gr9vwe5,gr9ynya,gr9z26d,gr9nfyd,gr9iguh,gr9hrka,gr9i47f,gr9kma4,gr9rtfx,gr9v8pa,gr9ndn1,gr9f5jo,gr9nc9h,gr9y507,gr9l595,gr9gkwq,gr9rdni,gr9yr4z,gr9y86f,gr9vq3t,gr9wqga,gr9z0mj,gr9txbg,gr9xx4e,gr9lhwz,gr9ugad,gr9xx4n,gra0gsa,gr9fm67,gr9ykcg,gr9q3us,gr9lukm,gr9iakn,gr9donf,gr9qqjq,gr9slxv,gr9vgno,gr9ocnp,gr9i2ft,gr9ik2g,gr9tvs0,gr9v0v6,gr9mq78,gr9zu6e,gr9znbw,gr9wv8r,gr9f5t8,gr9avjm,gr9z4mr,gr9schk,gr9w94q,gr9opbw,gr9v8j0,gr98y0w,gr9mrsy,gr9tefd,gr9ws3u,gr9ox8r,gr9nxxp,gr9tyz4,gr9rc5x,gr9xmwp,gr9vq6y,gr9jhb0,gr9dy72,gr9igyn,gra0as0,gr9y3j6,gr9y89w,gr9oyjq,gr9x6co,gr9e99l,gr9zmuk,gr9p3la,gr9j0fc,gr9tu9e,gr9ljlf,gr9m2k4,gr9jsdx,gr9oa21,gr9gd4q,gr9jhc2,gr9dc3n,gr9iao5,gr9rtl3,gr9vdl5,gr9xsi5,gr9qi6m,gr9sx3j,gr9nybo,gr9jb16,gr9vgrl,gr9uun6,gr9lew5,gr9v0yx,gr9tmej,gr9p8tu,gr9yo4x,gr9l8l0,gr9naut,gr9la63,gr9wwxp,gr9e9by,gr9kbud,gr9eke8,gr9okox,gr9wwxw,gr9nqo5,gr9m3kx,gr9eqq1,gr9wcej,gr9b1zh,gr9htce,gr9u8k0,gr9xvpt,gr9m11v,gr9ggd1,gr9llo9,gr9iaqj,gr9f84d,gra0an4,gr9afvm,gr9vopv,gr9gt0k,gr9s4pz,gr9iine,gr9em0d,gr9vlke,gr9t3hr,gr9n1fa,gr9vjzu,gr9jkl7,gr9yrhm,gr9xb7j,gr9mk1t,gr9svm4,gr9e68d,gr9honj,gr9kd1s,gr9jqxh,gr9x4ww,gr9l58t,gr9uxwk,gr9sheu,gr9y6uf,gr9rs4o,gr9shey,gr9pxc7,gr9rs4v,gr9hdma,gr9y243,gr9pz6m,gr9lrnb,gra02t4,gr9pedm,gr9vakg,gr9x83o,gr9vy9x,gra0llt,gr9txl8,gr9fubx,gr9yzbo,gr9mc7q,gr9ydjj,gr9mqii,gr9spcu,gr9z2hr,gr9r7mo,gr9votf,gr9gup1,gr9hief,gr9vfcd,gr9kxny,gr9z8ts,gr9vn8x,gr9te6r,gr9amb8,gr9x1tj,gr9jb7d,gr9dau2,gr9ympn,gr9oj98,gr9wqrh,gr9u2da,gr9lgyd,gr9g70j,gr9d7k4,gr9hn5x,gr9sb61,gr9u8p6,gr9rp1n,gr9shhr,gr9t8cw,gr9jnv5,gr9kd5e,gr9xuac,gr9vyc5,gr9hora,gr9u2dz,gr9scre,gr9c5y2,gr9xcwv,gr9o0bk,gr9tswy,gr9r4ir,gr9h6by,gr9pvon,gr99h9i,gr9wkh6,gr9pzfv,gr9lzwv,gr9i65t,gr9milw,gr9uuuv,gr9ko8p,gr9p0oq,gr9fc4x,gr9d613,gr9z8w8,gra0cen,gr9rykx,gr9wixe,gr9bt1w,gr9kevx,gr9pa6v,gr9pof0,gr9htk0,gr9l0wy,gr9k0s9,gr9vilg,gr9ltd3,gr9yt48,gra099n,gr9n6cw,gr9x3ht,gr9grn7,gr9vdvg,gr9b1ed,gr9sphc,gr9l0xt,gr9ketg,gr9hqym,gr9g2dl,gr9uy2z,gr9plak,gr9dfkg,gr9ymtt,gr9gjrm,gr9z47o,gr9xxjp,gr9z8yg,gr9tjiy,gr9ickd,gr9x89s,gr9neap,gr9vwvb,gr9qt8n,gr9trg1,gr9w4s2,gr9w6cz,gr9sqrr,gr9cndd,gr9llze,gr9t8hq,gr9fsxt,gr9z2o3,gr9uwjo,gr9qybp,gr9r4ng,gr9mgvb,gr9f9zm,gr9jxhw,gr9tw7q,gr9x0ex,gr9zfbx,gr9s1ut,gr9ti08,gr9lai3,gr9k29b,gr9rm28,gr9jcz5,gr9zn8y,gr9rm2d,gr9iu0o,gr9n3ay,gr9hapo,gr9x9xd,gr9bbsy,gr9koa3,gr9uq9x,gr9poju,gr9zfd7,gr9iu18,gr9f5aq,gr9jmmb,gr9e3dh,gr9uahk,gr9yzkp,gr9sja2,gr9qbcf,gr9ms9o,gr9zfdp,gr9qv8f,gr9obky,gr99rce,gr9pcp1,gr9riey,gr9v99q,gra0hbj,gr9sud4,gr9zc8r,gr9xl1h,gra0m2u,gr9ryrt,gr9nxt9,gr9udos,gr9shql,gr9co5p,gr9ez4e,gr9yok9,gr9ztng,gr9nyzp,gr9zox2,gra0cmf,gr9u8yw,gr9e69b,gr9wpgu,gr9q19r,gra09gw,gr9ve2a,gr9l90z,gr9qs51,gr9vqpi,gr9n3ef,gr9suex,gr9tzi2,gr9p3mz,gr9e1vp,gr9tzie,gr9ti4n,gr9xmof,gr9kbv6,gr9pic3,gr9wcus,gr9w6j9,gr9yzoa,gr9sg7z,gr9xo9v,gr9xjj9,gr9gvch,gr9zwvd,gra0j8q,gr9n0a7,gr9r5e1,gr9w08o,gr9so57,gra0fuq,gr9dtsb,gr9omrw,gr9vp72,gr9qtsf,gr9wmds,gr9xgep,gr9xetv,gr9eudw,gr9e1xy,gr9k0uy,gr9kmzf,gr9fi59,gr9f5gj,gr9m975,gr9qpuv,gr9lqi4,gr9u7hp,gr9mvul,gr9wsq8,gr9o1dn,gr9pvhv,gr9ggxf,gr9xt2u,gr9ugb8,gr9yago,gr9l5p0,gr9lfga,gr9wmez,gr9argd,gr9qkce,gr9utmp,gr9spsc,gr9b79r,gr9qtjs,gr9z992,gr9ltp4,gr9mdok,gr9owbg,gra00hy,gr9gk2n,gr9r069,gr9kzix,gr9jmoz,gr9f0rp,gr9t5m1,gr9f4xz,gr9u5yl,gr9pjq5,gr9kn1t,gr9nhrn,gr9c7n8,gr9mm60,gr9rjmn,gr9qv4w,gr9n6nq,gr9wudn,gr9ccek,gr9yaj1,gr9xzgv,gr9tjv8,gr9dh9q,gr9v36g,gra0b89,gr9kyw7,gr9wjbz,gr9sd7k,gr9qtwh,gr9dtj4,gr9g8o4,gr9fdgo,gr9aewa,gr9hdt3,gr9wr8t,gr9x8ms,gr9vx85,gr9q1an,gr9ultl,gr9r5ey,gr9h69r,gr9i3gg,gr9njep,gr9u1aa,gra0kqs,gr9m1oo,gr9qbx3,gr9s5cc,gr9go04,gr9y49m,gr9ticb,gr9ze3i,gr9wz6s,gr9st2j,gr9bkpi,gr9ty5s,gr9ptma,gr9ii43,gr9rz1w,gr9sxth,gr9iud2,gr9u2wn,gr9rz20,gr9toop,gr9mevx,gr9lcfz,gr9zqrk,gr9la74,gr9ttfm,gr9m81v,gr9pydg,gr9n0hy,gr9n0cs,gr9ucel,gr9k47i,gr9nt0p,gr9t45u,gr9f2hf,gr9gz2m,gr9tv13,gr94hnc,gr9octg,gr9ist7,gr9ejj4,gr9tahz,gr9sgh6,gr9hiav,gr9mpgv,gr9jddf,gr9oxaf,gr9k2ns,gr9ywsb,gr9ei23,gr9o8sk,gr9y939,gr9vr0n,gr9hklk,gr9wgay,gr9ibgv,gr9kosw,gr9k5dv,gr9vrvy,gr9nmon,gr9ey4k,gr9emqj,gr9zsf4,gr9pga3,gr9zvdo,gr9i0fh,gra052m,gr9uylr,gr9vj5c,gr9vuiw,gr9rz3o,gr9y5k2,gr9ytoe,gr9z003,gr9wl39,gr9lcjr,gr9j8p5,gr9wd70,gr9hu4y,gr9id3n,gr9bqhh,gr99d5i,gr9wpug,gr9qqxm,gr9homy,gr9x0x1,gr9h1pi,gr9t49z,gr9edbe,gr9qa3r,gr9y19p,gr9rsvj,gr9qz4k,gr9gh6k,gr9rpke,gr9nolx'), ('link_id', 't3_m74e3g'), ('sort', 'confidence')]
2021-03-17 13:36:15.197 PDT Params: {'raw_json': 1}
2021-03-17 13:36:15.609 PDT Response: 200 (23434 bytes)
2021-03-17 13:36:15.677 PDT Fetching: POST https://oauth.reddit.com/api/morechildren/
2021-03-17 13:36:15.678 PDT Data: [('api_type', 'json'), ('children', 'gr9vsp0,gr9lkhx,gr9diip,gr9l349,gr9cx6y,gr9i6tp,gr9wa32,gr9zcpc,gr9r6s4,gr9et5k,gr9klqy,gra0n1r,gr9fwny,gr9tc80,gr9ruht,gr9kvjb,gr97kfa,gr9pauh,gr9sq3u,gr9kffw,gr9sgh6,gr9e5gv,gr9izao,gr9fc4x,gr9mjar,gr9oiew,gr9f2nw,gr9vubl,gr9tqh3,gr9sle3,gr9nxvw,gr9uixa,gr9xljq,gr9tduk,gr9l00t,gr9gx29,gra0hur,gr9emw4,gr9tili,gr9j8tn,gr9aagg,gr9bp0y,gr9n71x,gr9f5va,gr9vtqu,gr9h4zj,gr9ykcg,gr9zm9w,gr9ia2t,gr9vr7q,gr9xifc,gr9faml,gr9gscx,gr9q1sn,gr9igex,gr9hcwt,gr9m9we,gr9z26d,gr9m3kx,gr9o90p,gr9x12j,gr9x2nk,gr9x12q,gr9suyb,gr9rbnc,gr9w2bl,gr9sjwi,gr9wudn,gr9tozt,gr9fikq,gr9m21a,gr9l58t,gr9u1n5,gr9x903,gr9mrbp,gr9x49j,gr9vv38,gr9wezg,gr9ejdi,gr9cvry,gr9csm9,gr9jm7a,gr9v8bq,gr9l3ag,gr9a12h,gr9db2p,gr9zcvg,gr9lnu6,gr9qbcf,gr9oqgm,gr9xlo9,gr9v0fv,gr9nc9h,gr9euwx,gr9bnk3,gr9f5t8,gr9irx3,gr9i8lp,gr9v36g,gr9pfrm,gr9g6c3,gr9e69b,gr9sqal,gr9nfyd,gr9t4io,gr9cxer,gra0a3q,gra0d2p,gr9r5ey,gr9wt9o,gr9ii43,gr9j5tk,gr9hpof,gr9ml34,gr9ey4k,gr9cmdh,gr9i2bq,gr9vgaw,gr9ynnt,gr9j03b,gr9l6j0,gr9o18t,gr9j13k,gr9e0yc,gr9uxcl,gr9euze,gr9l4yj,gr9r5e1,gr9n42m,gr9zuch,gr9h3kz,gr9sk15,gr9wf3m,gr9qjcj,gr9zek6,gr9xyex,gr9legv,gr9cg3l,gr9tymd,gr9ye86,gr9djm2,gr9x2ts,gr9yx7i,gr9vw6c,gr9ccyt,gr9pvon,gr97hk8,gr9ii7e,gr9ef8t,gr9e46n,gr9z6pe,gr9ttx0,gra0ddk,gr9x8ms,gr9vx85,gr9p4uf,gr9xdn0,gr9r8g0,gr9wv7b,gr9e11o,gr9lbd9,gr9yqxb,gr9pknm,gr9o637,gr9eifo,gr9v3rt,gr9da6y,gr9tivw,gr9sdst,gr9k9gh,gr9da78,gr9kqud,gr9j5ye,gr9khd5,gr9z9wr,gr9v8j0,gr9o1d9,gr9rmye,gr9u516,gr9o4qn,gr9y39c,gr9bx8m,gr9ny87,gr9wah4,gr9ql1l,gr9t9fu,gr9z56z,gr9u1aa,gr9pnv3,gr9nqc1,gr9ubdf,gr9s93p,gr9ez4e,gr9pxcu,gr9hd7j,gr9rjtw,gr9fkfb,gr9h84d,gr9p9o0,gr9h5bu,gr9onjv,gra0dh9,gr9sh1d,gr9pr2e,gr9qsm9,gr9s5cc,gr9co5p,gr9s191,gr9ffq2,gr9lz6j,gr9mrmq,gr9swv9,gr9hd9i,gr9qpuv,gr9va7e,gr9hief,gr9swvh,gr9f76s,gr9za0i,gr9vrli,gr9z6ux,gr9kqyi,gr9kfwh,gr9tqwy,gr9go04,gr9z8g6,gr9tklm,gr9gmfh,gr9gz2m,gr9nfql,gr9f7v6,gr9bkpi,gr9qw7t,gr9hjmj,gr9faca,gr9ql5q,gr9h4g5,gr9avfp,gr9qa3r,gr9e17t,gr9sbxu,gr9tpdb,gr9cer7,gr9epsu,gr9q27s,gr9wtkp,gr9vwe5,gr9ynya,gr9ox51,gr9iguh,gr9hrka,gr9i47f,gr9kma4,gr9rtfx,gr9f5jo,gr9fsgl,gr9y507,gr9l595,gr9gkwq,gr9rdni,gr9yr4z,gr9y86f,gr9vq3t,gr9wqga,gr9z0mj,gr9xx4e,gr9lhwz,gr9ugad,gr9xx4n,gr9riey,gr9fm67,gr9s2gw,gr9q3us,gr9lukm,gr9iakn,gr9donf,gr99rce,gr9slxv,gr9ugb8,gr9ocnp,gr9i2ft,gr9ik2g,gr9tvs0,gr9v0v6,gr9mq78,gr9rs4v,gr9wv8r,gr9mevx,gr9avjm,gr9schk,gr9w94q,gr9opbw,gr9lzwv,gr9mrsy,gr9iud2,gr9tefd,gr9ws3u,gr9nxxp,gr9tyz4,gr9rc5x,gr9toop,gr9vq6y,gr9jhb0,gr9dy72,gr9igyn,gra0llt,gr9y3j6,gr9y89w,gr9oyjq,gr9x6co,gr9e99l,gr9zmuk,gr9p3la,gr9j0fc,gr9tu9e,gr9ljlf,gr9m2k4,gr9jsdx,gr9gd4q,gr9jhc2,gr9dc3n,gr9iao5,gr9qt8n,gra0gsa,gr9vdl5,gr9xsi5,gr9qi6m,gr9nybo,gr9jb16,gr9vgrl,gr9uun6,gr9lew5,gr9v0yx,gr9pga3,gr9yo4x,gr9l8l0,gr9naut,gr9la63,gr9wwxp,gr9e9by,gr9kbud,gr9eke8,gr9mdok,gr9nqo5,gr9f4xz,gr9eqq1,gr9wcej,gr9b1zh,gr9htce,gr9u8k0,gr9xvpt,gr9m11v,gr9ggd1,gr9k0s9,gr9iaqj,gr9f84d,gr9la74,gr9afvm,gr9o1dn,gr9gt0k,gr9hiav,gr9iine,gr9em0d,gr9vrvy,gr9t3hr,gr9f2hf,gr9homy,gr9jkl7,gr9wkh6,gr9xb7j,gr9mk1t,gr9svm4,gr9e68d,gr9honj,gr9kd1s,gr9jqxh,gr9v99q,gr9x4ww,gr9ist7,gr9uxwk,gr9sheu,gr9y6uf,gr9rs4o,gr9shey,gr9pxc7,gr9mwq7,gr9hdma,gr9esdv,gr9pz6m,gr9lrnb,gra02t4,gr9qlh9,gr9vakg,gr9x83o,gr9vy9x,gr9txl8,gr9fubx,gr9mc7q,gr9xmof,gr9p0oq,gr9spcu,gr9z2hr,gr9votf,gr9gup1,gr9rjtd,gr9vfcd,gr9kxny,gr9z8ts,gr9vn8x,gr9te6r,gr9amb8,gr9x1tj,gr9jb7d,gr9ympn,gr9oj98,gr9wqrh,gr9u2da,gr9lgyd,gr9g70j,gr9d7k4,gr9hn5x,gr9sb61,gr9u8p6,gr9rp1n,gra0as0,gr9jnv5,gr9kd5e,gr9vyc5,gr9hora,gr9u2dz,gr9scre,gr9c5y2,gr9xcwv,gr9o0bk,gr9tswy,gr9h6by,gr99h9i,gr9yrhm,gr9pzfv,gr9xlzq,gr9i65t,gr9milw,gr9uuuv,gr9ko8p,gr9mqii,gr9z8w8,gra0cen,gr9rykx,gr9wixe,gr9bt1w,gr9kevx,gr9pa6v,gr9pof0,gr9htk0,gr9l0wy,gr9vilg,gr9ltd3,gr9yt48,gr9koa3,gr9n6cw,gr9x3ht,gr9grn7,gr9vdvg,gr9b1ed,gr9sphc,gr9l0xt,gr9yh91,gr9ketg,gr9g2dl,gr9dtsb,gr9plak,gr9dfkg,gr9ymtt,gr9gjrm,gr9l5p0,gr9xxjp,gr9z8yg,gr9tjiy,gr9ickd,gr9x89s,gr9hdt3,gr9oxaf,gr9kpev,gr9trg1,gr9w4s2,gr9w6cz,gr9cndd,gr9llze,gr9t8hq,gr9fsxt,gr9k5dv,gr9uwjo,gr9qybp,gr9r4ng,gr9mgvb,gr9f9zm,gr9jxhw,gr9tw7q,gr9x0ex,gr9zfbx,gr9s1ut,gr9ti08,gr9lai3,gr9bbsy,gr9k29b,gr9rm28,gr9jcz5,gr9zn8y,gr9rm2d,gr9iu0o,gr9bqhh,gr9hapo,gr9x9xd,gr9pedm,gra099n,gr9uq9x,gr9zfd7,gr9iu18,gr9f5aq,gr9e3dh,gr9uahk,gr9yzkp,gr9sja2,gr9mrcf,gr9qv8f,gr9obky,gr9h69r,gr9dh9q,gra0hbj,gr9zc8r,gr9xl1h,gra0m2u,gr9ryrt,gr9udos,gr9shql,gr9z0i5,gr9hvf2,gr9yok9,gr9ztng,gr9nyzp,gr9zox2,gra0cmf,gr9u8yw,gr9oa21,gr9wpgu,gra09gw,gr9ve2a,gr9l90z,gr9ydjj,gr9qs51,gr9vqpi,gr9llo9,gr9tzi2,gr9p3mz,gr9e1vp,gr9ti4n,gr9q1an,gr9kbv6,gr9pic3,gr9wcus,gr9w6j9,gr9yzoa,gr9uylr,gr9sg7z,gr9xo9v,gr9xjj9,gr9vuh5,gr9w08o,gr9so57,gr9jmmb,gr9omrw,gr9vp72,gr9wmds,gr9xgep,gr9xetv,gr9eudw,gr9k0uy,gr9kmzf,gr9f5gj,gr9m975,gr9x65j,gr9lqi4,gr9tmej,gr9wsq8,gr9nhrn,gr9n6nq,gr9p113,gr9yago,gr9z47o,gr9lfga,gr9wmez,gr9argd,gr9dtj4,gr9n0cs,gr9u7hp,gr9qtjs,gr9z992,gr9ltp4,gr9wwxw,gr9owbg,gr9dau2,gr9gk2n,gr9r069,gr9kzix,gr9jmoz,gr9f0rp,gr9t5m1,gr9u5yl,gr9pjq5,gr9kn1t,gr9ggxf,gr9c7n8,gr9mm60,gr9rjmn,gr9vung,gr9z7pn,gr9ccek,gr9xzgv,gr9tjv8,gr9hnmg,gra0b89,gr9wjbz,gr9sd7k,gr9qtwh,gr9g8o4,gr9fdgo,gr9aewa,gr9vwvb,gr9wr8t,gr9zp4k,gr9uk8l,gr9mfw8,gr9ultl,gr9ni96,gr9gqgv,gr9i3gg,gr9lngi,gra0kqs,gr9utmp,gr9m1oo,gr9t8cw,gr9qbx3,gr9he6k,gr9qetg,gr9y49m,gr9ticb,gr9ze3i,gr9wz6s,gr9rpke,gr9ty5s,gr9nacz,gr9rz1w,gr9sxth,gr9vgno,gr9u2wn,gr9uy2z,gr9xmwp,gr9lcfz,gr9zqrk,gr9ttfm,gr9qp8i,gr9m81v,gr9n0hy,gr9spsc,gr9y19p,gr9ucel,gr9k47i,gr9t45u,gr9vopv,gr9z0jy,gr9tv13,gr9octg,gr9b79r,gr9ejj4,gr9tahz,gra0j8q,gr9s4pz,gr9mpgv,gr9jddf,gr9k2ns,gr9ywsb,gr9ei23,gr9o8sk,gr9y939,gr9vr0n,gr9hklk,gr9wgay,gr9ibgv,gr9kosw,gr9z2o3,gr9vlke,gr9nmon,gr9uu6c,gr9emqj,gr9zsf4,gr9y243,gr9fdly,gr9i0fh,gra052m,gr9fi59,gr9gvch,gr9mvul,gr9hqym,gr9z003,gr9wl39,gr9lcjr,gr9j8p5,gr9wd70,gr9hu4y,gr9id3n,gr9jbv9,gr99d5i,gr9wpug,gr9vjzu,gr9x0x1,gr9h1pi,gr9t49z,gr9ynj5,gr9edbe,gr9kyw7,gr9nxt9,gr9rsvj,gr9rtl3,gr9qz4k,gr9gh6k,gr9di4b'), ('link_id', 't3_m74e3g'), ('sort', 'confidence')]
2021-03-17 13:36:15.678 PDT Params: {'raw_json': 1}
2021-03-17 13:36:16.051 PDT Response: 200 (17413 bytes)
2021-03-17 13:36:16.064 PDT Fetching: POST https://oauth.reddit.com/api/morechildren/
...

That poll includes over a dozen requests to https://oauth.reddit.com/api/morechildren/, many of which ask for >800 children each.

PRAW does some caching, but evidently not enough, or we haven't configured it correctly. Related issues: praw-dev/praw#1140, praw-dev/praw#131. And from praw-dev/praw#627: "Also the client side cache has been removed in PRAW4." (We're on PRAW 7.)

[snarfed]

Found the culprit for the comment handler. We fetch the post so we can do OPD, and we also fetch comments (with fetch_replies=True as an optimization for some sources, but that's clearly harmful for Reddit.

Relevant commits:

• 7bd3a6d: handlers: bug fix for f823e41: pass fetch_likes, etc when prefetching posts
• f823e41: handlers: use optimization in snarfed/granary@4d32457 to avoid double fetches
• snarfed/granary@4d32457: add optional activity kwarg to Source.get_*; move Facebook.get_rsvp to Source. the requested object in get_* can sometimes be extracted from the optional activity, which avoids an unnecessary API call.

[snarfed]

this last change, 241a4da, had a big impact. we shouldn't have been fetching replies to posts that just link to the user's site; now we don't. no more morechildren fetches in polls, since reddit currently only does link search.

i should probably also find someplace to put logic to drop those replies even if they do get fetched, since they shouldn't result in wms.

i still want to get rid of the duplicate user fetches though!

[snarfed]

added user profile API request caching in snarfed/granary@85ccb5a. pretty happy with reddit now, i'm going to close this.