Python - Returning Query Result From Server To Client In Flask

What I have I have a Client/Server in Flask. The client sends a query in JSON format to the server and the server creates a JSON file. There is another tool which takes this query

Solution 1:

What you want to do is called a Web Worker Architecture.

To pass in a real-time abstract queryResult from a background job to a client app you can use a combination of a Message Queue (Kafka is recommended, RabbitMQ is OK too) and a Web-Socket. When a client sends a request to the /data endpoint, you should return back some unique token (like UUID if your user is anonymous, or user id if it's authenticated). The same token you should add to the name of the resulting file. When your background worker finished processing the file it uses the token (from a file's name) to create a Kafka or RabbitMQ topic, like topic_for_user_id_1337 or topic_for_uuid_jqwfoj-123qwr, and publishes queryResult as a message.

At the same time, your client should establish a web-socket connection (Flask is quite bad for web-sockets, but there are few fine libs to do that anyway, like socketio) and pass the token through it to your backend so it'll create a message queue subscriber, subscribed to a topic with the token's name so when a background job is finished, a web-backend will receive a message and pass it to the user through a web-socket.

P.S. If it sounds overly complicated, you can avoid the use of MQ and WS and put the queryResult into a db and create a endpoint to check if it exists in a DB. If it's not, you return something like not ready yet and a client retries in a few seconds, if it's ready -- you return the queryResult from the DB.