Dariush

Feed update rate suddenly dropped from 30 minutes to 60 and isn't increasing (urgent)

Recommended Posts

See for yourself:

03mPbmZ.png

This feed is sufficiently high-volume that it immediately started hitting the update cap and thus losing items. My other similar (and in fact, smaller) feeds are still updating every 30 minutes.

Also, I probably requested this already, but notifying the user about hitting the 100-item cap would be really, really appreciated. I've been lucky that I happened to check crawler history only a few days after this occured over an unrelated question.

Share this post


Link to post
Share on other sites

Are you able to provide us the feed URL in order to give you more adequate answer?

How many articles this feed is posting per day?

Please note that the feed update interval is "throttled" to 1 hour if it publishes more than 1k articles per day. 

May be that's the case for you. 

The limitations have been updated couple of days ago because there were many similar feeds causing issues to our polling system performance. 
Sorry about the caused inconvenience but we are obliged to place limitations like this and regularly check the feeds behavior in order to keep that system running smoothly. 

Share this post


Link to post
Share on other sites

Yes, that's the case for me. Throttling the feeds that need the opposite, as I described in a nearby thread recently, with no warning, notification or even a changelog entry, was not a particularly user-friendly move, to be honest. If anything, the throttling should have been done for feeds that contain a lot of items at the moment of crawl, not cumulatively.

…And I cannot even boost the feed, instead receiving the error "Boosting is disabled for this URL. The most common reason for this is a publisher restriction.".

Could the throttling be lifted on this feed, please? I am a perfectionist and missing items really hurts my soul.

Share this post


Link to post
Share on other sites

I understand the frustration. We are trying to find the right balance between protecting our databases from malicious feeds and not disturbing the regular usage. However we cannot leave this unattended. Our databases are quick, but we are not a huge company and doesn't have unlimited funds to expand hardware to unrealistic scale. We are talking about tens of millions of articles entering our databases each day. We keep them all forever. There are feeds which post several thousands of articles per day, 99% of which no one ever reads, mostly because those feeds only have one or two subscribers. We already have 12 million feeds so micromanaging is just not possible. Most cloud-based RSS readers also only fetch 20 items at most at each poll. We've increased that limit to 100 a while ago and ever since we have a problem with feeds that post too much content, because they organically reach the shortest possible polling interval. The only way for us to keep things in order is to place some kind of thresholds and watch what happens. We will try to further optimize those thresholds in the future.

Share this post


Link to post
Share on other sites

IMO a good approach would be to do this:

1. Limit the permanent storage in the following way: first two months everything is stored, after the two months no more than, e.g., 5k items. Give an option to buy full permanent storage for selected feeds.

2. Don't throttle feeds with more than one user.

3. Actually detect which feeds aren't read - if, let's say, 50% of the items are never shown on screen when they hit the age of 1 month, the feed is considered dormant and can be throttled.

4. Have a regenerating quota per feed - for example, have it regenerate at a speed of 150/hour to a cap of 5k. Having this value low will cause the feed to be throttled. Basically, more soft caps, less hard ones.

I understand that you cannot micromanage everything, but I don't think so many people are sending requests for manual limit lifting that it's unfeasible to handle them on a case-by-case basis, plus it's not like people with malicious feeds will be pinging the developers anyway. I only have one throttled feed, so pretty please? ^_^

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now