BazQux Reader Community

Nitter.poast.org not updating

#1

It’s been three days since I’ve gotten updates. Here’s a feed example.

https://nitter.poast.org/KlasfeldReports/with_replies/rss

Raw feed looks fine to me and works fine in the Protopage.com RSS reader.

#2

For some reason they blocked BazQux. I’ve added a workaround (different user-agent) and it now works.

3 Likes
#4

Any chance you have time to look at this again? Same symptom as in Nov '23. Raw RSS feed looks current from a browser window, but I haven’t gotten updates since April 9th.

Thanks!

#5

Sorry for the delay, should be fixed now.

1 Like
#6

charging money to access someone’s hard work keeping a nitter instance alive

imagine thinking you’re the good guy in this equation, lol

2 Likes
#7

This. Unless you are giving half of your earnings to the Poast Nitter I don’t think this is a good service

2 Likes
#8

Enjoy being blocked forever.

1 Like
#9

>30 bux

lol

[pronographic image removed by moderator]

1 Like
#10

@vshabanov is what they’re saying on this post true? Is BazQux using multiple VPNs/IPs to evade their rate limits? Is BazQux taking into account response headers and limiting requests appropriately?

#11

In short:

  • There was a user-agent change in Nov’23 (2nd post in this thread, changed back few days ago), which could be counted as a rate-limit evasion (more on that below)
  • BazQux does use several proxy servers in a round-robin fashion. This could be counted as a rate-limit evasion, but it’s the only way to reliably update 0.5M feeds a day, and BazQux uses a very modest number of proxies. Any distributed fetcher uses more than one server to download content (read below why I switched from direct downloads from my German servers to using USA proxies)
  • BazQux doesn’t download from sites more frequently than once per second per a website host.
  • Most sites are downloaded much less frequently than once a second (including poast.org, where BazQux made a request around every 40s, now lowered to 67s)
  • There are no standard HTTP rate-limits headers to use, and there’s nothing about rate-limits in poast.org responses (there are rate-limit headers in API responses, and BazQux does use them when dealing with APIs)
  • I don’t know who was blocked, listed IPs are not mine.

More details:

BazQux uses 7 proxy servers (my own Linode nanonodes) in the USA to download most of the feeds. This was done due to

  • Hetzner servers were being blocked a lot by sites (I guess because there are a lot of bots running there thanks to Hetzner’s almost free internet traffic)
  • GDPR, when many sites started adding GDPR consents when downloading feeds (sic) from Europe.
  • Localization problems (sometimes you can get a German version of the feed even if you specify that you only accept English).

BazQux has never used VPN or 3rd party proxies as it’s impossible to guarantee that they are accessing a real website.

I also put a lot of work to limit parallel downloads from sites as much as possible (precisely not to be banned).

BazQux has strict rate limits (never download more than once per second from a site). Only a few sites are exceptions (YouTube and Facebook, where I’m using an API that allows a higher request rate, Google News and sites hosted by me).

I have changed the user agent for poast.org in November. Sometimes (4 sites currently) I do this as it’s faster than dealing with site admins (unfortunately they don’t always know how to configure their blocking and unblock BazQux).

Few days ago I actually changed User-Agent back to BazQux and doing downloads from my Germany (Hetzner) servers without any proxying. I’m not sure whom they blocked (the IPs listed in the post are not mine), but it’s not BazQux.

BazQux was making about 2k requests to nitter.poast.org per day, which doesn’t look like much (about one request every 40s). Yesterday, I added an additional rate limit for poast.org to not download more often than every 67s, which is about 1.3k downloads per day.

1 Like
#12

Now BazQux is actually banned from accessing nitter.poast.org. Next time I’ll ask users to ask site owners to unblock BazQux instead of taking shortcuts

#13

Well good to know that it wasnt Bazqux who was abusing the server.

When you say BazQux makes requests no more than one second per site. Is that restriction at the feed level or domain level?

For instance, if a mix of users are all requesting 10 different feeds on the same domain at the same time, BazQux will queue those requests so they are spread out over 10 seconds?

#14

It’s per IP. Sometimes blog1.example.com and blog2.example.com can be hosted on the same server (resolving to the same IP), and sometimes example.com can be a large site that resolves to multiple IPs.

BazQux runs per-IP queues. For some large sites, there are separate per-site queues to spread the downloads more evenly.

So yes, if 10 feeds hosted on the same server are requested at the same time, they will be queued and requests will be spread out over 10 seconds.

1 Like
#15

The blocks have been removed.

1 Like
#16

Thank you, and big thanks to the people of Poast.

#17

I have few RSS feeds that use nitter.privacydev.net. They load text and images in BazQux. As a test, I have the same feeds using nitter.poast.org, but the images do not load. Does anyone know why this is? Thank you.

#18

Images seem to show with the image proxy turned off (Settings => Appearance => Image proxy [off]). Perhaps some additional blocking.