New Order Sparks 'Martial Law' Concerns (newsweek.com)

All of these tools are insanely expensive (from my own experience at companies that have used them). I understand it, since building your own pipeline to handle the kind of throughput analytics takes is expensive and time consuming. Business leaders want the visibility but don't want to redirect dev resources to build and maintain these creaky data pipelines. It is the perfect market of high-value and low tolerance for build (on the build or buy spectrum).

But I am not going to pay $1000/month as a bootstrap startup. What open source alternatives exist that can be run on basic hardware?

mritchie712 · 27m ago

The best open source options are Airbyte and Meltano / Singer. But it's hard to keep them running. If you self-host them, you'll hit issues at least a few times a month which can each take a few hours to solve.

It's not like running Postgres which "just works". When you self-host Airbyte, you're still building a good bit.

I felt the same way about the cost of data tools. Paying $1,000 for Fivetran, $2,000 for Snowflake, $2,000 for Looker seemed crazy. We bundle all three for $500 / month at https://www.definite.app

caust1c · 52m ago

Check out redpanda connect / warpstream bento (depending on your license needs). Both came out of what was benthos.

https://github.com/redpanda-data/connect

https://github.com/warpstreamlabs/bento

zoogeny · 19m ago

Interesting, it looks like redpanda is a Kafka replacement and redpanda connect is a Kafka connect replacement but with a supported set of connectors (sources and sinks). I (once upon a time) had to write a Kafka connector myself so I get the general idea.

To be honest, I hadn't really given much thought about what event streaming I would use anyway. So I imagine using redpanda along with redpanda connect could be that layer (I was considering just using Redis streams or even PostgreSQL) and then there is just another redpanda connector for the db to add into that mix. If someone is starting from scratch that might be a good path. But I agree the MIT license of warpstream is a bit nicer if all you need is the connectors.

ssharp · 1h ago

I'm not sure about Census but Fivetran's free plan has met my needs to sync data from different ad platforms to BigQuery pretty well.

One of their pitfalls is charging by the row. If you're cost-conscious, you really need to watch what data you're syncing and you need to pare it down quite a bit during the 2-week period they give you when setting up a new connector. If you do all that though, you can get a lot of mileage out of the free plan for some use cases.

tomrod · 28m ago

Or batch massive rows? JSON structures in-database go a long way...

themanmaran · 1h ago

Airbyte is probably the best opensource tool in this space.

iflores12 · 24m ago

Airbyte gave us more headaches than it was worth. But if you can get it to work for you, it's probably the closest you'll get to Fivetran in the open-source tool space

zoogeny · 46m ago

Cheers, that is what I was thinking must exist but didn't know about.

loginx · 1h ago

Haven't used it personally, but I would suggest looking into Apache Hudi

zoogeny · 43m ago

Good to know about but looks more like an open source snowflake (e.g. data lake). Fivetran and Census are the in/out process layers that bookend the data lake. Although, Hudi does look like it has some of that functionality baked in.

paxys · 1h ago

A bootstrapped startup needs a MySQL database and a bunch of SELECT queries. Everything else is overkill.

zoogeny · 51m ago

Sure, SQL + something like metabase is a decent starting point (ideally running on a read-only replica). However, there is room to improve over that.

It's like logging. Yeah, there is sentry, papertrail, splunk, datadog and the like. But something better than greping sys logs is nice and totally reasonable for a startup to standup with Kibana/Elastic running on a tiny instance. That can provide significantly higher value.

There is a middle ground between stone tools and jet aircrafts. I was asking: what are the middle ground tools in this space.

morkalork · 1h ago

Ok if you're bootstrap it probably doesn't make sense but otherwise fivetran is fantastic for not having to deal with a boatload of third parties constant API updates and changes. If your core competency is something else entirely and not doing ETL, then it's worth paying for so you're not wasting time on doing that ETL work.

zoogeny · 37m ago

Yes, I've used Fivetran at VC funded startups that I worked at and I understand the value of not having to build this piece of common infrastructure. Although we did experience regular (probably once every couple of months) issues with our ETL getting out of sync. We even had to do a full re-sync on a couple of occasions (which to their credit they did for no charge).

As I said, I totally understand this market and why these companies are valuable. I respect the work they do. But while I am a tiny, tiny startup I don't want to lock in to anything and I know I can handle the amount of data myself with little effort if I have a basic open source alternative I can manage myself.

buremba · 2h ago

This indeed sounds like closing the loop, congrats to the team! Boris’s announcement is pretty interesting: https://www.getcensus.com/blog/census-joins-forces-with-five...

film42 · 1h ago

Congrats to the teams! Like others have said, your pricing ends up killing adoption for my company. We ended up self-hosting Airbyte. It ain't perfect but at least we're not paying $10/GB to replicate data within our own VPC.

barrrrald · 3h ago

Congrats to everyone. Some of the smartest and kindest people in data coming together!

_dark_matter_ · 4h ago

Seems like a no-brainer. I wonder if they ever started to build these capabilities in house; I'm sure they already had so much of the tooling available.

mritchie712 · 4h ago

Yeah, I was always curious why Fivetran didn't build this themselves when reverse-ETL started to take off.

I built a company[0], SeekWell, in this space (launched before Census), but was mostly focused on Sheets and Slack as destinations. SeekWell was acquired a few years ago too.

0 - https://seekwell.io/

skadamat · 4h ago

The challenge of syncing from stubborn SaaS tools to your data warehouse / database I suspect is different than syncing data from your data warehouse / database back to SaaS tools. Specifically, reverse ETL has to incorporate more context from the business I guess so the data that lands in the 3rd party tools is actually solid.

Once you have customers and a good network of integrations with a large number of tools, I suspect it's easier to just buy that company than build it all yourself?

georgewfraser · 3h ago

This is exactly right. We even went so far as to build a proof of concept internally, and the technical challenges are just very different. The simplest way to explain it is that Fivetran connects a skinny pipe (APIs) to a fat pipe (databases) while Census connects a fat pipe to a skinny pipe.

throwaway7783 · 3h ago

The data is only as solid as you make it to be. Ultimately reverse ETL is just a technology (basically from SQL to APIs). The quality/correctness of data is someone else's headache. I've been there and done that, and reverse ETL is a feature-product with huge churn. See how Hightouch pivoted hard from that into CDP.

educasean · 2h ago

Congrats to both Census and Fivetran. Census has an amazing product and very good people. Excited to see what's coming next from y'all

orangechairs · 4h ago

Anyone hear rumors of how much they were acquired for?

mritchie712 · 4h ago

I'd guess a little north of $500 million.

- Census last raised $60M Series B at a $630M valuation (upper bound)

- Census’s estimated annual revenue is $31.6 million with ~200 employees.

- Median private-SaaS EV/ARR multiple is 7× (7 * 31 = 217 = lower bound)

- Hightouch raises $80M on a $1.2B valuation(at ~60× ARR)

- Twilio completes $3.2B acquisition of Segment at ~21× ARR (upper multiple bound)

davidu · 3h ago

Congrats to the Census team and the Fivetran team!

bradleybuda · 3h ago

Thanks DU! You've had our back since the early days, thanks for taking a chance on us!

stalluri · 2h ago

Always wondered FT and Census might come into the each other's territory. Good to see both are merging forces together now!

tqi · 3h ago

What does this actually mean for customers? Is are we going to have to rebuild our Census syncs in Fivetran or will the product continue to run as-is? Will plans / pricing change?

throwaway7783 · 3h ago

Still missing a critical piece - ETL. If they acquire it instead of just asking people to use dbt, they have (kinda) the full stack solution.

mritchie712 · 3h ago

there's going to be more consolidation in data tooling this year. Many of the stand alone tools raised too much money and no one wants to buy 5 really expensive tools to assemble a "data stack" anymore.

if you want a data platform that's built to work as one cohesive unit, we got you: https://www.definite.app/

Definite has a data lake, ETL, and BI in one app.

tschellenbach · 4h ago

not a big fan of fivetran's pricing

arjie · 2h ago

I run a professional services org that helps you switch to an open source alternative. We'll host the solution for you if you want and aim to be drop-in Fivetran compatible in your workflows with a transition plan so you can run the thing if you'd like. Pricing is flexible. Personal email in profile.

throwaway7783 · 3h ago

Can you say more? Is it the unpredictability, or just too expensive?

rawgabbit · 3h ago

They charge based on the highest number of unique rows transferred for the month. It gets expensive quickly.