Discussion
Every time I look at OpenTelemetry in order to add instrumentation to @manyfold and @fedipub, I bounce off it because the server setup process, even for simple home experimenting, seems so overcomplicated.
I don't need a scalable stack of 4 different systems all with their own interfaces all doing different parts of the job, I just want somewhere simple to send my telemetry and then look at it.
"Setting up an OpenTelemetry server" isn't the problem I'm actually trying to solve here.
Advice?
@Slash909uk @phlash for me, no - I’m running Beszel and Dozzle at home and they’re perfectly fine for my needs. It’s more about adding instrumentation options for other users of the software, if they’re running commercial services for instance.
@Floppy @phlash ooh cool! I did not know about these tools for Docker 🙂 (I'm using Sylve on FreeBSD which is a similar idea, but less complete so far..)
Regards providing some API for operators to hook into your services and extract instrumentation, maybe a basic prometheus metrics endpoint would suffice? The interesting part is choosing what metrics to expose.. Experience running the services is probably the most valuable input to that 🙂 i.e. knowing what breaks and where the weak points are.
@Slash909uk @phlash Prometheus was the original plan, but with OLTP now able to do that and many more, it seems to be the way things are going. Beszel and Dozzle are great if you’re running containers; about as close to zero config as you can get…
@Floppy @manyfold @fedipub victoriatraces supports open telemetry ingestion and all the victoria services are easy to setup and trouble free once running: https://docs.victoriametrics.com/victoriatraces/data-ingestion/opentelemetry/
@johnl @Floppy @manyfold @fedipub
The wee two node cluster I run with my bro @Slash909uk settled on Victoria metrics stuff for storage and alerting, ingesting via alloy then otelcol. Having flipped from Linux to *BSD we find that all is well on freebsd, but nothing is available for netbsd or openbsd. I am now re-evaluating what we get out of this frankly monstrous pile of Go code (the monitoring stack is the biggest resource hog on the system 🤔...), and starting again with smaller tools...
@phlash @Floppy @manyfold @fedipub @Slash909uk I came to victoriametrics from a very large prometheus cluster and it was a breath of fresh air in simplicity and lowered resource usage, so I think I may be looking at this wrong! I've not used it yet, but I've been looking at https://www.jaegertracing.io as another solution - looks simpler.
@johnl Jaeger looks pretty straightforward, will have a go at that! Thanks!
@johnl thanks! I’ll check that out :)
@Floppy @manyfold @fedipub I went with an nhinx log basically... instances sending basic GET requests to a special endpoint, with data parameters.
That then (currently still) gets parsed into an ELK stack, but you could pull it into whatever you want or just leave it as a basic log.
Mid term I also want to get away from ELK...
See tracking.octoprint.org for some basic info.
@Floppy @manyfold @fedipub I think @fedify did integrate that just recently. Maybe they have some advice @hongminhee
@sabrinkmann @Floppy Two directions I know of. If a hosted service is fine, something like Sentry. For self-hosting, SigNoz is the only F/OSS option I know with easy OpenTelemetry ingestion; I don't know what else is out there. It has a paid cloud version too.
I think Sentry used to have an F/OSS edition. I'm not sure it still exists, or whether it accepts OTLP though.
@Floppy @manyfold @fedipub Do you really need the whole log/metric/trace/API thing, or just something to throw basic log text into?
As @phlash said, we've been around the loop here for our homelabs. Ended up with basic syslog for log streaming and VictoriaLogs for time series store and simple web UI.
If you want you can simply http post log lines (plain text or json) directly into VLog. Could not be simpler 🙂
Good luck!