What is the problem you are trying to solve?
It’s not uncommon during a full reindex that a bulk document request to Elasticsearch is larger than the value http.max_content_length (by default 100 MB).
In these situations, the Liferay log will display an error trace like
ERROR [DocumentsConsumer--2][RestClientTransportFactory:71] Unable to connect to [host=http://<es-domain>:9200]
java.lang.Exception: Unable to connect to [host=http://<es-domain>:9200]
at com.liferay.portal.search.elasticsearch8.internal.connection.RestClientTransportFactory$1.onFailure(RestClientTransportFactory.java:71) ~[?:?]
at org.elasticsearch.client.RestClient.onFailure(RestClient.java:557) ~[?:?]
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:303) ~[?:?]
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:292) ~[?:?]
at co.elastic.clients.transport.rest_client.RestClientHttpClient.performRequest(RestClientHttpClient.java:92) ~[?:?]
at co.elastic.clients.transport.ElasticsearchTransportBase.performRequest(ElasticsearchTransportBase.java:138) ~[?:?]
at co.elastic.clients.elasticsearch.ElasticsearchClient.bulk(ElasticsearchClient.java:537) ~[?:?]
at com.liferay.portal.search.elasticsearch8.internal.search.engine.adapter.document.BulkDocumentRequestExecutor._getBulkResponse(BulkDocumentRequestExecutor.java:154) ~[?:?]
at com.liferay.portal.search.elasticsearch8.internal.search.engine.adapter.document.BulkDocumentRequestExecutor.execute(BulkDocumentRequestExecutor.java:53) ~[?:?]
at com.liferay.portal.search.elasticsearch8.internal.search.engine.adapter.document.ElasticsearchDocumentRequestExecutor.executeBulkDocumentRequest(ElasticsearchDocumentRequestExecutor.java:57) ~[?:?]
at com.liferay.portal.search.elasticsearch8.internal.search.engine.adapter.ElasticsearchSearchEngineAdapterImpl.lambda$execute$1(ElasticsearchSearchEngineAdapterImpl.java:173) ~[?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
at java.lang.Thread.run(Thread.java:1583) [?:?]
This trace is usually accompanied on the Elasticsearch side by a request from ES like HTTP/1.1 413 Request Entity Too Large.
The official documentation will point you to either adapt reduce the index batch sizes or even increasing the value of http.max_content_length.
The usual entities to consider are
- com.liferay.journal.model.JournalArticle
- com.liferay.document.library.kernel.model.DLFileEntry
but there could be other culprits. Unfortunately, it can be very difficult to know which ones because the complaint coming from ES is not easily correlated to a specific bulk request.
What is your project about? (e. g. Intranet, Partner Portal, Enterprise Website, etc)
Any project using Elasticsearch and a large amount of information to reindex.
What is your proposed solution? (optional)
A straightforward way to identify the exact bulk request to ES producing the problem would to add a DEBUG trace in com.liferay.portal.search.elasticsearch8.internal.search.engine.adapter.document.BulkDocumentRequestExecutor#_getBulkResponse (here).
The code would be something like
catch (Exception exception) {
if (_log.isDebugEnabled()) {
_log.debug(bulkRequest);
}
if (i++ >= _numberOfTries) {
The resulting trace would look like:
DEBUG [DocumentsConsumer--2][BulkDocumentRequestExecutor:158] BulkRequest: POST /_bulk [{"index":{"_id":"com.liferay.document.library.kernel.model.DLFolder_PORTLET_6496551","_index":"liferay-220531"}},{"index":{"_id":"com.liferay.document.library.kernel.model.DLFolder_PORTLET_6496552","_index":"liferay-220531"}},[...]
which makes it clear that the entity DLFolder is culprit.
Note: There’s a task (LPD-98501) going in a similar direction, so maybe the DEBUG trace (or something of the sort) could be included there.