I know you can export directly to the backend, but the collector typically uses less than 50MB of RAM in my experience (even when handling lots of traces) and it's pretty easy to add sidecars to however you deploy your backends nowadays. Using Grafana SaaS metrics could look a little spiky or generally weirder without the collector, but normal with it, so I just default to using it now.
It's the shame the docs on it are still quite bad. The example config in the article here does look almost identical to the one we use everywhere, just without the redact, and should probably be pasted somewhere into the official docs.
Every provider seems to produce their own soft fork of the collector for branding (eg Alloy, ADOT, etc) and slightly changes the configuration, which doesn't help.
bfors · 2h ago
I happen to work with otel a lot so I'll offer a few of my thoughts:
- Consider decoupling your collector from whatever is consuming your traces with something like kafka. Traces can be pretty heavy and it can be tricky to scale collectors. If something goes down, it's probably a good idea to continue writing the traces to queue or topic.
I've been putting off a self-hosted observability setup for a long time.
Any recommendations on basis ease of setup and operation? (For something low-medium scale).
My ideal setup would be to just write SQL on telemetry data and plot dashboards / set alerts.
Also, thoughts on Vector vs otel agent?
srcreigh · 22m ago
HyperDX is really great. It is basically SQL on telemetry data in clickhouse.
Don’t use vector or otel-agent. Add a materialized view in clickhouse to transform data and swap HyperDX to load from your view (in the UI.)
ndhandala · 47m ago
OneUptime does this with otel. Happy to help! Feel free to reach out at nawazdhandala [at] oneuptime [dot] com
smarx007 · 47m ago
Seq?
oulipo2 · 58m ago
I've been looking at HyperDX (ClickStack) and SigNoz, but those indeed are coupled
srcreigh · 55s ago
I tried both. Signoz is pretty sloppily built. For ex the self hosted option starts a ZK instance with 1 clickhouse host. Their log transformation tool is broken and confusing.
HyperDX is just a lot better, sure a few papercuts but they got all the important stuff right imo.
k_bx · 36m ago
I'm evaluating Greptimedb in prod and while I hate to have a needless component like OTEL-Collector in general, it serves as a read-only gate between the database and the user, so that greptime keeps listening to localhost only, and OTEL-Collector guarantees nobody will write to the database directly.
If it were to give more fine-grained control over write-only access -- would probably just write directly and let it handle the load.
pewpewp · 34m ago
I did not like working with OpenTelemetry; made me miss the good old days (monolith).
It's the shame the docs on it are still quite bad. The example config in the article here does look almost identical to the one we use everywhere, just without the redact, and should probably be pasted somewhere into the official docs.
Every provider seems to produce their own soft fork of the collector for branding (eg Alloy, ADOT, etc) and slightly changes the configuration, which doesn't help.
- Consider decoupling your collector from whatever is consuming your traces with something like kafka. Traces can be pretty heavy and it can be tricky to scale collectors. If something goes down, it's probably a good idea to continue writing the traces to queue or topic.
- https://www.otelbin.io is a nice little tool to help with collector configuration
My ideal setup would be to just write SQL on telemetry data and plot dashboards / set alerts.
Also, thoughts on Vector vs otel agent?
Don’t use vector or otel-agent. Add a materialized view in clickhouse to transform data and swap HyperDX to load from your view (in the UI.)
HyperDX is just a lot better, sure a few papercuts but they got all the important stuff right imo.
If it were to give more fine-grained control over write-only access -- would probably just write directly and let it handle the load.