All of these tools are insanely expensive (from my own experience at companies that have used them). I understand it, since building your own pipeline to handle the kind of throughput analytics takes is expensive and time consuming. Business leaders want the visibility but don't want to redirect dev resources to build and maintain these creaky data pipelines. It is the perfect market of high-value and low tolerance for build (on the build or buy spectrum).
But I am not going to pay $1000/month as a bootstrap startup. What open source alternatives exist that can be run on basic hardware?
mritchie712 · 10h ago
The best open source options are Airbyte and Meltano / Singer. But it's hard to keep them running. If you self-host them, you'll hit issues at least a few times a month which can each take a few hours to solve.
It's not like running Postgres which "just works". When you self-host Airbyte, you're still building a good bit.
I felt the same way about the cost of data tools. Paying $1,000 for Fivetran, $2,000 for Snowflake, $2,000 for Looker seemed crazy. We bundle all three for $500 / month at https://www.definite.app
banditelol · 5h ago
Hi, I've been loking something like this!
Any of your custumer has success story migrating off bigquery to your platform?
And how do you compare to motherduck? (Looks like you built some of ypur stack on top of duckdb)
fblp · 6h ago
Your comment reads like a pitch but I checked definite and I've been looking for something like this. What we you using it for? Did you evaluate any other Ai analytics tools?
No comments yet
ssharp · 11h ago
I'm not sure about Census but Fivetran's free plan has met my needs to sync data from different ad platforms to BigQuery pretty well.
One of their pitfalls is charging by the row. If you're cost-conscious, you really need to watch what data you're syncing and you need to pare it down quite a bit during the 2-week period they give you when setting up a new connector. If you do all that though, you can get a lot of mileage out of the free plan for some use cases.
tomrod · 10h ago
Or batch massive rows? JSON structures in-database go a long way...
caust1c · 10h ago
Check out redpanda connect / warpstream bento (depending on your license needs). Both came out of what was benthos.
Interesting, it looks like redpanda is a Kafka replacement and redpanda connect is a Kafka connect replacement but with a supported set of connectors (sources and sinks). I (once upon a time) had to write a Kafka connector myself so I get the general idea.
To be honest, I hadn't really given much thought about what event streaming I would use anyway. So I imagine using redpanda along with redpanda connect could be that layer (I was considering just using Redis streams or even PostgreSQL) and then there is just another redpanda connector for the db to add into that mix. If someone is starting from scratch that might be a good path. But I agree the MIT license of warpstream is a bit nicer if all you need is the connectors.
banditelol · 5h ago
I've tried airbyte, sling, and dlt (besides building several tools from scratch)
My best bet for now will be dlt if you have dedicated DE team, but sling will get you a long way for moving data around your warehouse
loginx · 11h ago
Haven't used it personally, but I would suggest looking into Apache Hudi
zoogeny · 10h ago
Good to know about but looks more like an open source snowflake (e.g. data lake). Fivetran and Census are the in/out process layers that bookend the data lake. Although, Hudi does look like it has some of that functionality baked in.
themanmaran · 11h ago
Airbyte is probably the best opensource tool in this space.
iflores12 · 10h ago
Airbyte gave us more headaches than it was worth. But if you can get it to work for you, it's probably the closest you'll get to Fivetran in the open-source tool space
zoogeny · 10h ago
Cheers, that is what I was thinking must exist but didn't know about.
morkalork · 11h ago
Ok if you're bootstrap it probably doesn't make sense but otherwise fivetran is fantastic for not having to deal with a boatload of third parties constant API updates and changes. If your core competency is something else entirely and not doing ETL, then it's worth paying for so you're not wasting time on doing that ETL work.
zoogeny · 10h ago
Yes, I've used Fivetran at VC funded startups that I worked at and I understand the value of not having to build this piece of common infrastructure. Although we did experience regular (probably once every couple of months) issues with our ETL getting out of sync. We even had to do a full re-sync on a couple of occasions (which to their credit they did for no charge).
As I said, I totally understand this market and why these companies are valuable. I respect the work they do. But while I am a tiny, tiny startup I don't want to lock in to anything and I know I can handle the amount of data myself with little effort if I have a basic open source alternative I can manage myself.
paxys · 11h ago
A bootstrapped startup needs a MySQL database and a bunch of SELECT queries. Everything else is overkill.
zoogeny · 10h ago
Sure, SQL + something like metabase is a decent starting point (ideally running on a read-only replica). However, there is room to improve over that.
It's like logging. Yeah, there is sentry, papertrail, splunk, datadog and the like. But something better than greping sys logs is nice and totally reasonable for a startup to standup with Kibana/Elastic running on a tiny instance. That can provide significantly higher value.
There is a middle ground between stone tools and jet aircrafts. I was asking: what are the middle ground tools in this space.
doctorpangloss · 8h ago
Palantir's market cap is $274b and they make glorified dashboards. There's just too much money in it to spend cycles doing it for free.
Congrats to everyone. Some of the smartest and kindest people in data coming together!
r1290 · 4h ago
Fivetran has been great. But in this new ai world. Something like dragster + dlt and sling. You can have your own fivetran developed in house. I haven’t dove too much into reverse etl- but it would be awesome to see a dtl like open source tool for reverse etl.
educasean · 12h ago
Congrats to both Census and Fivetran. Census has an amazing product and very good people. Excited to see what's coming next from y'all
r1290 · 4h ago
Fivetran should’ve done this a long time ago. I think that both etl and reverse etl is going open source route. With this ai world we live in now. You just need dagster or temporal - and a few lines of python.
_dark_matter_ · 14h ago
Seems like a no-brainer. I wonder if they ever started to build these capabilities in house; I'm sure they already had so much of the tooling available.
mritchie712 · 14h ago
Yeah, I was always curious why Fivetran didn't build this themselves when reverse-ETL started to take off.
I built a company[0], SeekWell, in this space (launched before Census), but was mostly focused on Sheets and Slack as destinations. SeekWell was acquired a few years ago too.
The challenge of syncing from stubborn SaaS tools to your data warehouse / database I suspect is different than syncing data from your data warehouse / database back to SaaS tools. Specifically, reverse ETL has to incorporate more context from the business I guess so the data that lands in the 3rd party tools is actually solid.
Once you have customers and a good network of integrations with a large number of tools, I suspect it's easier to just buy that company than build it all yourself?
georgewfraser · 13h ago
This is exactly right. We even went so far as to build a proof of concept internally, and the technical challenges are just very different. The simplest way to explain it is that Fivetran connects a skinny pipe (APIs) to a fat pipe (databases) while Census connects a fat pipe to a skinny pipe.
throwaway7783 · 13h ago
The data is only as solid as you make it to be. Ultimately reverse ETL is just a technology (basically from SQL to APIs). The quality/correctness of data is someone else's headache. I've been there and done that, and reverse ETL is a feature-product with huge churn. See how Hightouch pivoted hard from that into CDP.
tpoacher · 9h ago
Is Fivetran to Fortran like what C++ is to C?
bicx · 2h ago
It’s like what Cantonese is to cantaloupe
orangechairs · 14h ago
Anyone hear rumors of how much they were acquired for?
mritchie712 · 14h ago
I'd guess a little north of $500 million.
- Census last raised $60M Series B at a $630M valuation (upper bound)
- Census’s estimated annual revenue is $31.6 million with ~200 employees.
- Median private-SaaS EV/ARR multiple is 7× (7 * 31 = 217 = lower bound)
- Hightouch raises $80M on a $1.2B valuation(at ~60× ARR)
- Twilio completes $3.2B acquisition of Segment at ~21× ARR (upper multiple bound)
tqi · 13h ago
What does this actually mean for customers? Is are we going to have to rebuild our Census syncs in Fivetran or will the product continue to run as-is? Will plans / pricing change?
davidu · 13h ago
Congrats to the Census team and the Fivetran team!
bradleybuda · 13h ago
Thanks DU! You've had our back since the early days, thanks for taking a chance on us!
stalluri · 12h ago
Always wondered FT and Census might come into the each other's territory. Good to see both are merging forces together now!
throwaway7783 · 13h ago
Still missing a critical piece - ETL. If they acquire it instead of just asking people to use dbt, they have (kinda) the full stack solution.
film42 · 11h ago
Congrats to the teams! Like others have said, your pricing ends up killing adoption for my company. We ended up self-hosting Airbyte. It ain't perfect but at least we're not paying $10/GB to replicate data within our own VPC.
mritchie712 · 13h ago
there's going to be more consolidation in data tooling this year. Many of the stand alone tools raised too much money and no one wants to buy 5 really expensive tools to assemble a "data stack" anymore.
if you want a data platform that's built to work as one cohesive unit, we got you: https://www.definite.app/
Definite has a data lake, ETL, and BI in one app.
tschellenbach · 14h ago
not a big fan of fivetran's pricing
arjie · 12h ago
I run a professional services org that helps you switch to an open source alternative. We'll host the solution for you if you want and aim to be drop-in Fivetran compatible in your workflows with a transition plan so you can run the thing if you'd like. Pricing is flexible. Personal email in profile.
throwaway7783 · 13h ago
Can you say more? Is it the unpredictability, or just too expensive?
rawgabbit · 13h ago
They charge based on the highest number of unique rows transferred for the month. It gets expensive quickly.
But I am not going to pay $1000/month as a bootstrap startup. What open source alternatives exist that can be run on basic hardware?
It's not like running Postgres which "just works". When you self-host Airbyte, you're still building a good bit.
I felt the same way about the cost of data tools. Paying $1,000 for Fivetran, $2,000 for Snowflake, $2,000 for Looker seemed crazy. We bundle all three for $500 / month at https://www.definite.app
No comments yet
One of their pitfalls is charging by the row. If you're cost-conscious, you really need to watch what data you're syncing and you need to pare it down quite a bit during the 2-week period they give you when setting up a new connector. If you do all that though, you can get a lot of mileage out of the free plan for some use cases.
https://github.com/redpanda-data/connect
https://github.com/warpstreamlabs/bento
To be honest, I hadn't really given much thought about what event streaming I would use anyway. So I imagine using redpanda along with redpanda connect could be that layer (I was considering just using Redis streams or even PostgreSQL) and then there is just another redpanda connector for the db to add into that mix. If someone is starting from scratch that might be a good path. But I agree the MIT license of warpstream is a bit nicer if all you need is the connectors.
My best bet for now will be dlt if you have dedicated DE team, but sling will get you a long way for moving data around your warehouse
As I said, I totally understand this market and why these companies are valuable. I respect the work they do. But while I am a tiny, tiny startup I don't want to lock in to anything and I know I can handle the amount of data myself with little effort if I have a basic open source alternative I can manage myself.
It's like logging. Yeah, there is sentry, papertrail, splunk, datadog and the like. But something better than greping sys logs is nice and totally reasonable for a startup to standup with Kibana/Elastic running on a tiny instance. That can provide significantly higher value.
There is a middle ground between stone tools and jet aircrafts. I was asking: what are the middle ground tools in this space.
I built a company[0], SeekWell, in this space (launched before Census), but was mostly focused on Sheets and Slack as destinations. SeekWell was acquired a few years ago too.
0 - https://seekwell.io/
Once you have customers and a good network of integrations with a large number of tools, I suspect it's easier to just buy that company than build it all yourself?
- Census last raised $60M Series B at a $630M valuation (upper bound)
- Census’s estimated annual revenue is $31.6 million with ~200 employees.
- Median private-SaaS EV/ARR multiple is 7× (7 * 31 = 217 = lower bound)
- Hightouch raises $80M on a $1.2B valuation(at ~60× ARR)
- Twilio completes $3.2B acquisition of Segment at ~21× ARR (upper multiple bound)
if you want a data platform that's built to work as one cohesive unit, we got you: https://www.definite.app/
Definite has a data lake, ETL, and BI in one app.