Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

It sounds like you are doing ETL on your logs. Most people want to search them when something goes wrong, which means indexing.


No, what I'm doing is analysis on logs. That could be as simple as "find me the first N occurrences of this pattern" (which you might call search) but includes things like "compute the distribution of request latencies for requests affected by a certain bug" or "find all the tenants impacted by a certain bug, whose signature may be complex and span multiple services across a long timescale".

Good luck doing that in a timely manner with Kibana. Indexed search is completely useless in this case, and it solves a problem (retrieval latency) I don't (and, I claim, you don't) have.

EDIT: another way to look at this is the companies I've worked at where I've been able to actually do detailed analysis on the logs (they were stored sensibly such that I could run mapreduce jobs over them) I never reached a point where a problem was unsolvable. These days where we're often stuck with a restrictive "logs search solution as a service" I often run into situations where the answer simply isn't obtainable. Which situation is better for customers? I guess cynically you could say being unable to get to the bottom of an issue keeps me timeboxed and focused on feature development instead of fixing bugs.. I don't think anyone but the most craven get-rich-quick money grubber would actually believe that's better though.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: