Objective
This article aims to describe some steps to follow when you encounter a translation freeze. It also shows what can be done when the SYSTRAN application is not translating anymore.
Introduction
When trying to translate a text or a file, it can happen that the translation job freezes: the text does not translate or the file can be stuck in "translating", even after hours. It can be a frustrating situation, but below some steps are described to show you what can be done.
Steps
- Check of the number of instance(s) and profile status
- Check of the services' status on the Web interface
- Relaunch of the Translation Resource Monitor service
- Check of the queue management
- Check of the logs
1. Check of the number of instance(s) and profile status
Go to the menu, then click on Advanced Configuration. In the Translation Resources tab, you need to be sure that the translation resource(s) is/are green, with the "running" status. The filter also needs to have the same status. The minimum number of instance of each TR used needs to be 1. For the filter, the recommendation is 2.
Regarding the profiles, go to the menu, click on Profiles. The status color has to be green with "activated" and "running". If the status color is orange or red with "deactivated" and/or "not running", the translation will not be functional.
2. Check of the services' status on the Web interface
Now check the status of all services. The SYSTRAN services which affect the translation are in particular:
- for the text translation: Systran Translation Resource Monitor
- for the file translation: Systran Enterprise Server File Translation Consumer
- for the translation by API: Systran Entreprise Server Translation Gateway
The third party service that really impacts the translation is RabbitMQ.
To check the status of all services, go to the menu, then to Advanced Configuration. You can check the status of the services in the Services tab. If a service is up, it will be in green. If it is down, it will appear in red. A service in trouble will be in orange.
3. Relaunch of the Translation Resource Monitor service
When typing a text in the translation box, if dots appear in the right box instead of translated text, the translation is unfunctional. You can do the following commands in the terminal:
systemctl stop systran-translation-resource-monitor.service
docker network rm trm_proxy_network
docker swarm leave --force && docker network rm docker_gwbridge
systemctl stop docker.socket
systemctl start docker.socket
systemctl start systran-translation-resource-monitor.service
Those commands will relaunch Docker and the Translation Resource Monitor, thus in the end the translation process.
4. Check the queue management
As told earlier, RabbitMQ is a third party service that impacts the translation. It is a service that is between the translation computing nodes and the SYSTRAN services, so it has an important role.
On your server web interface, you can go to Advanced Configuration, then to the Queues tab. You can check if there is no message stuck in "Messages". If for some queue names, the number of messages is high, it means that there is no enough consumer, so the translation cannot be properly done.
RabbitMQ has its own web interface and can be reached there: http://systran:SESpassword@[SERVER]:15672
The port 15672 must be opened on the SERVER firewall. You can check this article to know more about that.
Once logged in, look for the queues:
Look for running and idle queues: you can sort them by the column "Total":
If there are messages in an idle queue, it means that the queue is stagnant. You can also check for the queue ID of your translation engine via the web interface to find the stagnant queue(s). You will need to purge the stuck messages. This article explains how to do all the different steps.
However, all the pending documents (for file translation) or translation jobs will need to be restarted. Users will have to redo the translation request(s).
If the issue is still present, then you can consider the last step below.
5. Check the logs
You can check the logs of SYSTRAN services with the command below:
journalctl -fu systran-\*
When checking the logs, you can have some errors. In this article we will particularly look at some errors regarding to the lack of memory and some services.
Lack of memory
If the translation is not done, it can be because of a lack of memory on the server. Then the log will look like this:
ERROR requestId=5127fd6f-baaa-4e86-999c-e7d09e04d423 | Not enough memory after filter import
In that case, you can do the following commands to check the space and memory on your server:
df -hT
free -m
Some solutions can be found to resolve this issue:
- increase the memory/disk space of your server (you will find attached a file with the requirements of server sizing)
- decrease the number of instance(s) used by Translation Resource(s) and by the filter
Error with some SYSTRAN services
If, by checking the logs, you see that the issue is with one SYSTRAN service, such as the file translation consumer service, you can check the logs:
journalctl -u systran-ses-file-translation-consumer.service
Then you can restart the service:
systemctl restart systran-ses-file-translation-consumer.service
Error with RabbitMQ
The SYSTRAN logs can show you that there is an issue with RabbitMQ. You can check the logs of RabbitMQ with the command:
journalctl -u rabbitmq-server
You can restart the service:
systemctl restart rabbitmq-server
You can go to this article to see in details how to check the status and logs of services.