We will log warn messages about hitting maxkbps as this will indirectly throttle how fast we are reading files. I would highly recommend that the customer split the monitoring workload between at least 2 instances.Īlso, you can check maxkbps in nf and make sure it is tuned to an acceptable value. (pre 6.3 - i will add more details about new features in 6.3 shortly)īased on experience, 100k being monitored by a single instance will always lead to these kind of issues and/or high indexing latency. You can use the tailing processor rest endpoint to determine which files are currently being read and in the queue. So, you can very easily run into a case where we are stuck reading a file for a long time and we fail to read other files before they are rotated or deleted. This is because, by default, both batch and tailingproc will read until EOF of file, and then wait 3 seconds (time before close) before switching to the next file in the queue. In this case, if all 100k files are being constantly updated, we can run into a case where tailingproc/batchreader will be stuck reading a file. Once they grow beyond 20MB they get defered to batch reader. If these files never grow past 20MB we will always use tailing processor to read them. That being said, the real world limiting factor is how fast we can read from disk and/or forward data to the indexing tier. We will read from disk as fast as the underlying storage will let us (assuming maxkbps=0 and a healthy unsaturated indexing tier). Technically there is no limitation on the amount of files that one splunk instance can monitor.
0 Comments
Leave a Reply. |