I'm working with the url-to-text API endpoint and facing consistent failures when increasing the request rate.
I've tested the following scenarios using 3 threads:
50 URLs per minute – failed
40 URLs per minute – failed
30 URLs per minute – failed
However, when I reduce the rate to 20 URLs per minute using 2 threads, the requests go through successfully.
This leads me to believe there may be a rate limit or concurrency restriction on the endpoint. Could you please confirm if such a limit exists, and if so, what the recommended rate is for stable usage?
We need to be able to scale our operation significantly. Is there anything we can do to increase our throughput? Are there solutions available for higher volume processing?