Feed Spider - Part 1 vincode.io

@vincode I develop Unread. I am very interested in this.

I often wish for a way to find feeds for a particular site. Many sites have feeds but do not use link tags to make them discoverable. In some cases I can find them with simple heuristics but in other cases I cannot. A directory would be extremely useful.

I am eager to help if I can.

@johnbrayton I don't know if this project will help with sites that don't properly link their RSS feeds. The web crawler will end up following the same discovery rules that RSS Readers like Unread and NetNewsWire do. Here are the heuristics used in NetNewsWire, which you probably are aleady using. FeedFinder

I do think that Feed Spider will be interesting for you. Feed discovery is a difficult problem that most RSS readers don't have a solution for. Making it easier for users to discover content will help the open web and may help make RSS more accessible again. Wider distribution, including via commercial apps, will help with that.

I'll release everything under the MIT license and all the data inputs are publicly available information. The resultling database should be small enough to distribute as a build artifact. Assuming that the end product is high enough quality, I will try to get it bundled with NetNewsWire. You would have the same option to include it in Unread.

@vincode Sounds great!

@vincode Simplying echoing what @johnbrayton said. I develop Elytra and like you mentioned, almost every RSS Feed Reader faces this issue.

I went through some of your posts utilising ML for this. I'd love to know if I help with this in any way. You've made great progress so far.