I know you can export directly to the backend, but the collector typically uses less than 50MB of RAM in my experience (even when handling lots of traces) and it's pretty easy to add sidecars to however you deploy your backends nowadays. Using Grafana SaaS metrics could look a little spiky or generally weirder without the collector, but normal with it, so I just default to using it now.
It's the shame the docs on it are still quite bad. The example config in the article here does look almost identical to the one we use everywhere, just without the redact, and should probably be pasted somewhere into the official docs.
bfors · 2h ago
I happen to work with otel a lot so I'll offer a few of my thoughts:
- Consider decoupling your collector from whatever is consuming your traces with something like kafka. Traces can be pretty heavy and it can be tricky to scale collectors. If something goes down, it's probably a good idea to continue writing the traces to queue or topic.
I've been putting off a self-hosted observability setup for a long time.
Any recommendations on basis ease of setup and operation? (For something low-medium scale).
My ideal setup would be to just write SQL on telemetry data and plot dashboards / set alerts.
Also, thoughts on Vector vs otel agent?
srcreigh · 15m ago
HyperDX is really great. It is basically SQL on telemetry data in clickhouse.
Don’t use vector or otel-agent. Add a materialized view in clickhouse to transform data and swap HyperDX to load from your view (in the UI.)
ndhandala · 40m ago
OneUptime does this with otel. Happy to help! Feel free to reach out at nawazdhandala [at] oneuptime [dot] com
smarx007 · 40m ago
Seq?
oulipo2 · 51m ago
I've been looking at HyperDX (ClickStack) and SigNoz, but those indeed are coupled
k_bx · 29m ago
I'm evaluating Greptimedb in prod and while I hate to have a needless component like OTEL-Collector in general, it serves as a read-only gate between the database and the user, so that greptime keeps listening to localhost only, and OTEL-Collector guarantees nobody will write to the database directly.
If it were to give more fine-grained control over write-only access -- would probably just write directly and let it handle the load.
pewpewp · 27m ago
I did not like working with OpenTelemetry; made me miss the good old days (monolith).
It's the shame the docs on it are still quite bad. The example config in the article here does look almost identical to the one we use everywhere, just without the redact, and should probably be pasted somewhere into the official docs.
- Consider decoupling your collector from whatever is consuming your traces with something like kafka. Traces can be pretty heavy and it can be tricky to scale collectors. If something goes down, it's probably a good idea to continue writing the traces to queue or topic.
- https://www.otelbin.io is a nice little tool to help with collector configuration
My ideal setup would be to just write SQL on telemetry data and plot dashboards / set alerts.
Also, thoughts on Vector vs otel agent?
Don’t use vector or otel-agent. Add a materialized view in clickhouse to transform data and swap HyperDX to load from your view (in the UI.)
If it were to give more fine-grained control over write-only access -- would probably just write directly and let it handle the load.