Load Testing Google Cloud PubSub

I wrote the following tool to do load test on Google Cloud PubSub.

It’s a pretty simple but effective tool to generate a lot of data. This assumes user is using gcloud auth application-default login for credentials.

In this particular examples:

• Every third request is adding a different attribute.
• Python dict as a payload that get’s converted to JSON.
• It runs 5 threads in parallel to write data fast

This was tested with 20K requests, and worked as expected.

from google.cloud import pubsub_v1
from uuid import uuid4
import json
from multiprocessing.dummy import Pool

TOPIC_PATH = 'YOUR TOPIC HERE'
publisher = pubsub_v1.PublisherClient()

print("Generating data")
events = []
for i in range(20):
  data = {
    'run': i,
    'requestId': str(uuid4())
  }
  events.append(data)

def publishEvent(event):
  data = json.dumps(event).encode("utf-8")
  future = publisher.publish(TOPIC_PATH, data)
  print(future.result(), data)

print("Starting to send")
pool = Pool(5) # Number of concurrent threads
asyncresponse = pool.map(publishEvent, events)
pool.close()
pool.join()
print("Done")
Code language: PHP (php)

Hope it helps.