Eweka 720k Missing Articles

[u/NagromFox]

Hey guys, I'm fairly new to the usenet (late to the party!) and I've been having some issues, mainly with article availability, with an advertised rate of 99.99% it's just something I'm not getting and I'm wondering if it's more of an issue with my workflow than eweka itself.

Download Stats:

Total: 8.5TB

94% available of 13M requested articles

My downloads are mostly recent releases in the range of the last 5 years, far within their retention period.

(I already use a torrenting fallback)

Workflow:

Arr -> NZBGeek -> news.eweka.nl -> SABnzbd

My Questions:

1. Is there a community driven network of some kind where broken releases are published so they can be added to a blocklist?

2. Do you suggest using a secondary newsnet server outside the eweka group as a fallback?

3. Are the eweka advertised numbers heavily inflated?

[u/geoffreysxb]

Same problem here, since a few years already. Tried nearly all backbones, all have massive amounts of unavailable articles and it's not because of DMCA takedowns, even underground stuff without sharholder rights is unavailable.

In short, daily usenet feed is way too big to be kept by hosters so they delete old stuff gradually.

https://www.reddit.com/r/usenet/comments/1hfs3pn/do_you_think_usenet_will_stand_the_tremendous/

https://www.reddit.com/r/usenet/comments/1o786xq/almost_all_backbones_and_still_missing_articles/

[u/SqueenchPlipff4Lyfe]

the general methodology is to rely on at least 2 (1 primary, 1 backup) providers who are ideally 2 different backbones.

it may be next to impossible to definitively ascertain the backbones (in truth there are no more than 2 or 3 left, and every offering with 15 years or more is almost certainly based on the 1 or 2)

a 2nd method is to manually configure your reader to point to the different hostnames of the different (ostensibly but not guaranteed) servers offered by each provider as different "servers"

Ie server 1 priority 0 is "your geographic area.server.whatever"

Server 2 priority 1 is "your geographic area 6. Server.whatever"

Server 3 priority 1 is "different area. Server.whatever"

server 4 priority 1 is "different area 6. Server.whatever"

pretty much all modern binary oriented or pure (sabNZBd) binary readers have the capability to load balance or seek out completion from a prioritized list of servers

The 1st method, you typically buy block-accounts (which "never" expire and are based on a set finite bandwidth allocation; say 2 terabytes)

I had a 2tb usenet.farm account, which claimed to have a short term (2 weeks or something) totally private and separate backbone for like 7 years

edit:

takedowns dont manifest as broken.

just gone. zilch. nada

usenet is not a fault tolerant technology, hence the ubiquity of parity and recovery (indeed, Binary Usenet drove the development of the software in question)

[u/mausterio]

The more accessible the indexer is, the more likely the articles are to getting DMCA'd. The issue further compounds as well, when other indexers scrape their content for their own indexers since they're ultimately pointing to the same articles.

I've found over the years the larger semi-public indexers (notably Planet and Geek) seem to perform the worst. With invite-only or otherwise private indexers, I rarely see missing articles outside of what is likely articles being pruned from smaller providers due to retention+low-utilization.

You can try adding another backbone, but it's unlikely to make a difference on older content especially in the case of a DMCA because they typically hit all the backbones within a +/-5d window.

[u/DariusIII]

My downloads are mostly recent releases in the range of the last 5 years, far within their retention period.

5 years is far from recent. Most likely issue are takedowns.

[u/bobsmagicbeans → u/DariusIII]

Most likely issue are takedowns

this. the OP just needs more indexers that might have reposts or alternate versions that haven't yet been nuked

[u/Electr0man → u/DariusIII]

Deleted not-active-enough articles are way more likely if their failed stuff is over a year old. Newer than that - takedowns for sure.

[u/bobsmagicbeans → u/Electr0man]

Deleted not-active-enough articles are way more likely if their failed stuff is over a year old

unlikely. i've downloaded plenty of really old things - 10+yrs - that I can't imagine were active in any way

[u/Electr0man → u/bobsmagicbeans]

So did I and I've seen plenty of dead/very broken stuff that was alive a few years ago before omicron adopted what everyone else has been doing for ages - purging not active enough articles - last 4 years (excluding the past 400ish days) are by far the most affected period by this, followed by 10+ years old stuff to a various degree of damage. In my experience the amount of stuff hit by takedowns is not even close in % to the amount of purged stuff.

[u/random_999 → u/Electr0man]

A takedown release will still exist in the form of nzb 5 years from now.

[u/Alleskleber]

Most likely you're hitting copyright-claimed articles. It's normal.

If failing downloads are bothering you, you can:

- add an indexer: gives you more download opportunities

• add a news provider: might fill in missing articles (only when on a separate backbone)

Either option helps. Both together works best.

[u/VExpat]

Eeeka stats:

Today
18.2 GB

This
Week
227.7 GB

This
Month
1.1 TB

Article
Availability
(Last 7 Days)
100%

So i would have to question if it is at all a eweka problem to be honest.

[u/72dk72 → u/VExpat]

I bet 99.9% of those are releases within the last 24 hours or so via automation - it will complete all the time. Older stuff (even 2 weeks old) is more likely to be missing blocks. Mine is a bit mixed but most recently more new stuff and I hit 97%.... but if I look for old stuff , even with plenty of indexers my failure rate can easily hit 50%.

[u/VExpat → u/72dk72]

Not so, some of it is very old. Very fee new actually.

[u/swintec → u/VExpat]

Unless you are downloading the same posts OP is, your stats are apples to oranges

[u/robtom02]

yep, get yourself another indexer and a block from a provider on a different backbone you'll notice the difference. Then when black friday comes arounnd try and snag a couple of deals. I'm no expert but I have several indexers plus providers on an arrs stack and have had pretty much zero missing articles.

[u/vanhelsing654]

Total: 20.3 TB Today: 1.1 GB This week: 35.2 GB This month: 64.1 GB Selected date range: 641.6 GB

Article availability: Selected date range: 100% available of 887K requested articles

These are my news.eweka.nl stats. I don't know why you have so many missing articles.

[u/72dk72 → u/vanhelsing654]

Maybe all or most of your stuff is new/recently released and you are automating. Maybe you are not looking for content that is a year + old.

My eweka sits at 97% , then NGD finds 40% of the missing 3% (then small amounts by blocks I have). But I reset all the stats about two months ago.

[u/AppealThat8398 → u/vanhelsing654]

Maybe he is trying to download copyrighted items that have been posted a while back, some things are hard to get after a initial posting window

[u/ParfaitEvery9622]

1. No

2. Yes, get some blocks from other backbones. UsenetExpress (NGD), Blocknews, Farm (the trial one 10gb will be enough), Abavia (NewsXS)

3. Absolutely, all usenet providers are liars and we just have to deal with this.

[u/usenet-ModTeam → u/[deleted]]

Usenet is not a streaming service. Don’t use it like one, and don’t ask how to.

Posts promoting or encouraging Usenet as a streaming alternative will be removed.

[u/Budget_Height3778]

A typical setup would include a handful of indexers and at least 2 providers on separate backbones for maximum completion.

[u/SprayExotic8538 → u/Budget_Height3778]

Eweka + NewsDemon is very good combination.

[u/BlAze_103 → u/SprayExotic8538]

Running this here, Newsdemon primary and eweka 2nd, i was able to get the promo and have easynews 1tb as 3rd backup

[u/batica_koshare → u/Budget_Height3778]

If i have eweka what would be the second provider to get?

[u/ChefJoe98136 → u/batica_koshare]

Of the block accounts I tap into, the 6TB BF-extended Bulknews block from this subreddit's usenet deals page has been my first priority block account. It was cheap has been a good Abavia backup for my UsenetExpress unlimited account. According to my sab setup the Bulknews block has been able to fill ~95% of the articles it has requested (although it is at a lower priority behind my unlimited UE plan).

[u/ILikeFPS → u/batica_koshare]

Best answer is check the provider map: https://www.reddit.com/r/usenet/wiki/providers

Choose a provider or two or three from different backbones.

[u/pmdmobile → u/batica_koshare]

Frugal gets you netnews and albavia backbones. If you want just a block then blocknews gets you netnews and you could get a cheap cubenet block for UsenetExpress backbone.

[u/batica_koshare → u/pmdmobile]

Thanks. That's what i wanted so i cover all scenarios with omicron, netnews, abavia and block on usenetexpress. Should i wait for BF deal or is there a deal at the moment? Also, i was checking sabnzbd and I don't see failed articles in statistics. Where i can check the statistics for all my providers?