Fields where Native Americans farmed a thousand years ago discovered in Michigan (smithsonianmag.com)

I’ve been working on CocoIndex, an open-source data transformation framework for AI. it supports incremental processing and only processes what’s changed, for data freshness especially when resources cannot catch up.

With 100 lines of python you can easily set up a production grade pipeline from vector embeddings, knowledge graphs, structured extraction etc.

Once you bring the pipeline as live mode, it automatically detects source change, and keeps the target store updated, with minimal amount of processing.

I’m building this framework to be as simple as possible to develop with, yet production-ready from day one — combining the best practices and lessons I’ve learned over the past years.

Previously, I was a tech lead at Google on projects like search indexing and ETL infra for 8 years. After I left Google last year, I went through pivoting hell. In all the projects I’ve built, data still sits in the center of the problem. I need to write custom logic to prepare data for AI and I need the data to stay fresh for AI to generate effective decisions or recommendations. So I started CocoIndex.

Getting started: https://cocoindex.io/docs/getting_started/quickstart

Examples: https://github.com/cocoindex-io/cocoindex/tree/main/examples

If you prefer to read: https://cocoindex.io/blogs/tags/examples/

Would love to learn your thoughts. Thanks! Linghua

Comments (1)

badmonster · 18h ago

The core framework is written in Rust because it is performant, robust, and can bind to different languages. Currently it has a Python SDK. I’m thinking of supporting typescript soon. Would love to hear your thoughts, thanks

Start your own Internet Resiliency Club (bowshock.nl)

Show HN: Zeekstd – Rust Implementation of the ZSTD Seekable Format (github.com)

Solving LinkedIn Queens with APL (pitr.ca)

Jokes and Humour in the Public Android API (voxelmanip.se)

Why SSL was renamed to TLS in late 90s (2014) (tim.dierks.org)

Infracost (YC W21) is hiring software engineers (GMT+2 to GMT-6) (infracost.io)

Modifying an HDMI dummy plug's EDID using a Raspberry Pi (downtowndougbrown.com)

DARPA program sets distance record for power beaming (darpa.mil)

Childhood leukemia: how a deadly cancer became treatable (ourworldindata.org)

Real-time CO2 monitoring without batteries or external power (news.kaist.ac.kr)

Twin – A Textmode WINdow Environment (github.com)

Hyperspectral scans of historical pigments and painting reconstructions (github.com)

Datalog in Rust (github.com)

How to modify Starlink Mini to run without the built-in WiFi router (olegkutkov.me)

First 2D, non-silicon computer developed (psu.edu)

Chemical knowledge and reasoning of large language models vs. chemist expertise (nature.com)

Simplest C++ Callback, from SumatraPDF (blog.kowalczyk.info)

Datalog in miniKanren (deosjr.github.io)

The Hewlett-Packard Archive (hparchive.com)

Fields where Native Americans farmed a thousand years ago discovered in Michigan (smithsonianmag.com)

Foundations of Computer Vision (2024) (visionbook.mit.edu)

Telephone Exchanges in the UK (telephone-exchanges.org.uk)

Is Gravity Just Entropy Rising? Long-Shot Idea Gets Another Look (quantamagazine.org)

Maya Blue: Unlocking the Mysteries of an Ancient Pigment (mexicolore.co.uk)

Reinventing circuit breakers with supercritical CO2 (spectrum.ieee.org)

The long afterlife of a literary classic (thecritic.co.uk)

Lisp-stat: Lisp environment for statistical computing (lisp-stat.dev)

Unprecedented optical clock network lays groundwork for redefining the second (phys.org)

The Art of Lisp and Writing (2003) (dreamsongs.com)

Accumulation of cognitive debt when using an AI assistant for essay writing task (arxiv.org)

Canyon.mid (canyonmid.com)

Text-to-LoRA: Hypernetwork that generates task-specific LLM adapters (LoRAs) (github.com)

Ruby on Rails Audit Complete (ostif.org)

Cyborg Embryos Offer New Insights into Brain Growth (spectrum.ieee.org)

SQLite Date and Time Functions (2007) (www2.sqlite.org)

Cure Dolly's Japanese Grammar Lessons (kellenok.github.io)

The experience continues until you stop experiencing it (strangemachine.tv)

How the BIC Cristal Ballpoint Pen Became the Most Successful Product in History (openculture.com)

The Matrix (1999) Filming Locations – Shot-for-Shot – Sydney, Australia [video] (youtube.com)

A skyscraper that could have toppled over in the wind (1995) (newyorker.com)

Meta's Llama 3.1 can recall 42 percent of the first Harry Potter book (understandingai.org)

An Introduction to the Hieroglyphic Language of Early 1900s Train-Hoppers (openculture.com)

How fast can the RPython GC allocate? (pypy.org)

It’s nearly impossible to buy an original Bob Ross painting (2021) (thehustle.co)

Tiny-diffusion: A minimal implementation of probabilistic diffusion models (github.com)

An origin trial for a new HTML <permission> element (2024) (developer.chrome.com)

How easy is it for a developer to "sandbox" a program? (kristaps.bsd.lv)

Show HN: StellarSnap – Explore NASA APODs, simulate orbits, learn astronomy (stellarsnap.space)

GitHub CI/CD observability with OpenTelemetry step by step guide (signoz.io)

Social anxiety disorder-associated gut microbiota increases social fear (pnas.org)

Show HN: CocoIndex – Open-Source Data transformation for AI, only process delta

Comments (1)