Objective
This article aims to describe some steps to follow when you encounter a translation freeze. It also shows what can be done when the SYSTRAN application is not translating anymore.
Introduction
When trying to translate a text or a file, it can happen that the translation job freezes: the text does not translate or the file can be stuck in "translating", even after hours. It can be a frustrating situation, but below some steps are described to show you what can be done.
Steps
- Check of the number of instance(s) and profile status
- Check of the services' status
- Relaunch of the Translation Resource Monitor service
- Check the logs
- Check the memory and the disk space
- Check the queue management
1. Check of the number of instance(s) and profile status
Go to the menu, then click on Advanced Configuration. In the Translation Resources tab, you need to be sure that the translation resource(s) is/are green, with the "running" status. The filter also needs to have the same status. The minimum number of instance of each TR used needs to be 1. For the filter, the minimum is 2. According to the number of active TR, it can be more: the more you have TR, the more you need to add filter instances.
If you add more instances to the filter, you can check if translation works after.
Regarding the profiles, go to the menu, click on Profiles. The status color has to be green with "activated" and "running". If the status color is orange or red with "deactivated" and/or "not running", the translation will not be functional.
2. Check of the services' status
Check the status of all services on the web interface. The SYSTRAN services which affect the translation are in particular: Systran Translation Resource Monitor, Systran File Translation Consumer (for the file translation), Systran Translation Gateway (for the translation by API).
The third party services that can impact the translation are Docker, RabbitMQ and MongoDB.
To check the status of all services, go to the menu, then to Advanced Configuration. You can check the status of the services in the Services tab. If a service is up, it will be in green. If it is down, it will appear in red. A service in trouble will be in orange.
From the server terminal, the commands to check are:
systemctl status systran*
systemctl status rabbitmq-server mongod docker
3. Relaunch of the Translation Resource Monitor service
When typing a text in the translation box, if dots appear in the right box instead of translated text, the translation is unfunctional. You can do the following commands in the terminal:
systemctl stop systran-translation-resource-monitor.service
docker network rm trm_proxy_network
docker swarm leave --force && docker network rm docker_gwbridge
systemctl stop docker.socket
systemctl start docker.socket
systemctl start systran-translation-resource-monitor.service
Those commands will relaunch Docker and the Translation Resource Monitor, thus in the end the translation process.
4. Check the logs
You can check the logs of SYSTRAN services with the command below:
journalctl -fu systran-\* | grep -v poller
Pay particularly attention to the error logs.
If, by checking the logs, you see that the issue is with one SYSTRAN service, such as the file translation consumer service or the dispatcher, you can restart the concerned service:
systemctl restart systran-ses-file-translation-consumer.service
systemctl restart systran-translation-dispatcher
You can go to this article to see in details how to check the status and logs of services.
5. Check the memory and the disk space
Usually when there is a lack of memory on the server, you can have this error log:
ERROR requestId=5127fd6f-baaa-4e86-999c-e7d09e04d423 | Not enough memory after filter import
In that case, you can do the following commands to check the memory on your server:
free -m
To solve this issue you can:
- increase the memory of your server
- decrease the number of instance(s) used by Translation Resource(s) and by the filter
- check the service that consumes a lot of memory and restart this service:
top
systemctl restart <name_of_service>
You can check the disk space with this command:
df -h
If your disk space exceeds in /, you can identify large folders (specifically in /var and /opt):
du -h --max-depth=1
ls -lShr
You can check with Systran support if those files can be safely deleted. If your server is online, it means TRSL is set to disk, so you can safely remove tar.gz files located here:
rm -rf /opt/systran/apps-node/translation-resource-store/file-backend-storage/*
6. Check the queue management
As told earlier, RabbitMQ is a third party service that impacts the translation. It is a service that is between the translation computing nodes and the SYSTRAN services, so it has an important role.
On your server web interface, you can go to Advanced Configuration, then to the Queues tab. You can check if there is no message stuck in "Messages". If for some queue names, the number of messages is high, it means that there is no enough consumer, so the translation cannot be properly done.
RabbitMQ has its own web interface and can be reached there: http://systran:SESpassword@[SERVER]:15672
The port 15672 must be opened on the SERVER firewall. You can check this article to know more about that.
Once logged in, look for the queues:
Look for running and idle queues: you can sort them by the column "Total":
If there are messages in an idle queue, it means that the queue is stagnant. You can also check for the queue ID of your translation engine via the web interface to find the stagnant queue(s). You will need to purge the stuck messages. This article explains how to do all the different steps.
However, all the pending documents (for file translation) or translation jobs will need to be restarted. Users will have to redo the translation request(s).
If none of these steps helps you to solve your issue, you can contact the SYSTRAN support.