The Future of Flatpak (lwn.net)

Defuddle is an open-source JS library I built to parse and extract the main content and metadata from web pages. It can also return the content as Markdown.

I built Defuddle while working on Obsidian Web Clipper[1] (also MIT-licensed) because Mozilla's Readability[2] appears to be mostly abandoned, and didn't work well for many sites.

It's still very much a work in progress, but I thought I'd share it today, in light of the announcement that Mozilla is shutting down Pocket. This library could be helpful to anyone building a read-it-later app.

Defuddle is also available as a CLI:

https://github.com/kepano/defuddle-cli

[1] https://github.com/obsidianmd/obsidian-clipper

[2] https://github.com/mozilla/readability

Comments (34)

Tsarp · 4m ago

Been using the obsidian clipper since it was out and this is a really neat. The per website profile based extraction is awesome.

Even if you are not a obsidian user, the markdown extraction quality is the most reliable Ive seen.

tmpfs · 4h ago

Interesting as I was researching this recently and certainly not impressed with the quality of the Readability implementations in various languages. Although Readability.js was clearly the best, it being Javascript didn't suit my project.

In the end I found the python trifatura library to extract the best quality content with accurate meta data.

You might want to compare your implementation to trifatura to see if there is room for improvement.

fabmilo · 2h ago

reference to the library: https://trafilatura.readthedocs.io/en/latest/

for the curious: Trafilatura means "extrusion" in Italian.

| This method creates a porous surface that distinguishes pasta trafilata for its extraordinary way of holding the sauce. search maccheroni trafilati vs maccheroni lisci :)

(btw I think you meant trafilatura not trifatura)

jeanlucas · 1h ago

Obsidian Web Clipper is a great tool to turn chatGPT conversations in markdown, or to just print it (believe me, it is a user case)

T0Bi · 1h ago

I just ask ChatGPT to provide the summary or whatever I need as a markdown file.

creakingstairs · 4h ago

I was just looking at obsidian web-clipper's source code because I've been quite impressed at its markdown conversion results and came across Defuddle in there. I'll be using for my bespoke read-it-later/ knowledge-base app, so thank you in advance :D

shrinks99 · 2h ago

I've been super happy with Obsidian Web Clipper! It's worked really well for me with the one exception of importing publish dates (which is more than forgivable !)

acrophobic · 59m ago

Is Mozilla's Readability really abandoned? The latest release (v0.6.0) is just 2 months ago, and its maintainer (Gijs) is pretty active on responding issues.

rcarmo · 5h ago

The Python analogues seem to be well maintained. I did my own implementation of the Readability algorithm years ago and dropped it in favor them, and I have a few scrapers going strong with regular updates.

kepano · 5h ago

Are there any in particular you can recommend?

khimaros · 5h ago

not parent, but this one looks maintained https://github.com/buriy/python-readability

inhumantsar · 3h ago

can confirm that readability seems to be on life support. I used it slurp, an obsidian plugin which serves the same basic purpose as web clipper, and always had a hard time getting PRs reviewed and merged.

i started working on my own alternative but life (and web clipper) derailed the work.

it's funny. somehow slurp keeps gaining new users even though web clipper exists. so i might have to refactor it to use your library sometime soon even though I don't use slurp myself anymore.

khaki54 · 2h ago

seems pretty much perfect including obsidian clipper. Thanks!

90s_dev · 3h ago

Neat. With ~3 more lines of code, you could get a URL and render it in simpler HTML and be a full fledged replacement.

billconan · 4h ago

Are you using ai models behind the scenes? I saw Gemini and others in the code. I am asking mainly to understand the cost of using yours vs. readability. Thank!

kepano · 4h ago

No it's all rules-based. I think the code you're referring to is "extractors", which are website-specific rules that I'm working on to standardize the output from sites with comments threads (e.g. HN, Reddit) and conversational chats (ChatGPT, Claude, Gemini).

pugio · 17m ago

I would love something which reliably extracted a markdown back/forth from all the main LLM providers. I tried `defuddle` on a shared Gemini URL and it returned nothing but the "Sign In" link. Maybe I'm using your extractor wrong? How are you managing to get the rendered conversation HTML?

busymom0 · 5h ago

In the playground, after I enter a url, I can't seem to figure out how to submit it to fetch the url? I tried pressing the return key on iOS keyboard but it didn't do anything. Am I missing something?

kepano · 5h ago

The input is there to test the url option — which I admit is a bit confusing, so I have removed it for now. I haven't found a good and free way to proxy requests from a GitHub page (yet).

input_sh · 4h ago

A bit off-topic, but I'm very excited to see the launch of Bases! I've obsessively followed the roadmap for like a year awaiting this day and have been frequently disappointed to still see it stuck somewhere under "planned".

Not that I didn't already implement a read-it-later solution with Obsidian+Dataview, but this definitely makes things simpler!

jeanlucas · 1h ago

Didn't it release just some days ago?

fkfyshroglk · 4h ago

For those not in the know: [Readability](https://github.com/mozilla/readability)

No comments yet