I wrote the following tool to do load test on Google Cloud PubSub.
It’s a pretty simple but effective tool to generate a lot of data. This assumes user is using gcloud auth application-default login for credentials.
In this particular examples:
- Every third request is adding a different attribute.
- Python dict as a payload that get’s converted to JSON.
- It runs 5 threads in parallel to write data fast
This was tested with 20K requests, and worked as expected.
from google.cloud import pubsub_v1
from uuid import uuid4
import json
from multiprocessing.dummy import Pool
TOPIC_PATH = 'YOUR TOPIC HERE'
publisher = pubsub_v1.PublisherClient()
print("Generating data")
events = []
for i in range(20):
data = {
'run': i,
'requestId': str(uuid4())
}
events.append(data)
def publishEvent(event):
data = json.dumps(event).encode("utf-8")
future = publisher.publish(TOPIC_PATH, data)
print(future.result(), data)
print("Starting to send")
pool = Pool(5) # Number of concurrent threads
asyncresponse = pool.map(publishEvent, events)
pool.close()
pool.join()
print("Done")Code language: PHP (php)Hope it helps.