MQTT.pub payload missing somewhere in GCP?

#1
  1. My goal is: Use MQTT.pub to publish a payload to a device telemetry event to GCP (a cloud function registers the event)

  2. My actions are:

On Mongoose OS side:
let res = MQTT.pub(’/devices/’ + Cfg.get(‘device.id’) + ‘/events’ , JSON.stringify({ a: 1, b: 2 }), 1);
print(‘Published msg:’, res ? ‘yes’ : ‘no’);

On GCP Cloud Function side:
module.exports = functions.region(‘europe-west1’).pubsub.topic(‘bg-topics’).onPublish(async (message, context) => {
console.log('TEST: ’ + JSON.stringify(message));

  1. The result I see is:
    In the GCP cloud function console, I don’t see the payload, only this:

“TEST: {“data”:“XXXXX”,“attributes”:{“deviceId”:“XXXX_XXXXX”,“deviceNumId”:“2831456XXX”,“deviceRegistryId”:“bg-registry”,“deviceRegistryLocation”:“europe-west1”,“projectId”:“XXXX”,“subFolder”:”"}}"

So the payload (message.json) is missing here.

  1. My expectation & question is:
    Why? :slight_smile:

Thanks!

#2

There are several power users running customer networks on GCP.
You can sniff your network or check GCP for more in deep analysis of the data. I remember getting data from Pub/Sub, I don’t remember being able to observe data on the cloud console (one+ year ago).
I did something like
gcloud pubsub subscriptions pull --auto-ack projects/projectname/subscriptions/subscriptionname
after following Google’s quickstart (which configured some useful stuff) and correctly configuring the authentication
My report here, though in Spanish

#3

Thanks for the tip, I was able to debug the messages with setting up a subscription.

Interestingly, I see the message body as a json in the view messages page of GCP console:

But when checking the cloud function logs, the body shows up as “data” (I just realized that the json body gets passed as a base64 encoded data string – here the data attribute was “eyJiIjoyLCJhIjoxfQ==”, the json that I passed ):

I then tried changing the pub call to pass directly a string:

let res = MQTT.pub(defaultTopic, ‘{ a: 1, b: 2 }’, 1);

But the effect was the same: the message.json property is not avaible, only message.data.

So now the question is, why does this happen?

Thanks a lot!

#4

All you comment is GCP stuff, probably because (guessing) they internally use HTTPS requests. There is no notion of any type of encoding nor serialization (not even JSON) in MQTT, it is “a binary protocol”; whatever you put into MQTT payload comes out the other end as such.
In mOS, the publish function gets strings (not JSON).
Your original function call with a stringified JSON string looks OK to me.
The article I linked has accompanying code, and in that code you’ll see I did almost the same:

let msg = JSON.stringify({free: Sys.free_ram(), total: Sys.total_ram()}); 

The consumer that gets stringified JSON from Pub/Sub can parse it and get the original JSON object back (it must); whatever happens inside GCP, I don’t know. As far as I remember, GCP does not care about msg payload types, while AWS does process some payloads as JSON (e.g.: shadows).

#5

Thanks! You are right, I omitted one important piece. Actually I am using Firebase functions, which are wrappers to cloud functions. But the Firebase onPublish event defines the json property of the message: https://firebase.google.com/docs/reference/functions/providers_pubsub_.message?hl=en#json
https://firebase.google.com/docs/functions/pubsub-events?hl=en

So there should be a message.json property. But I can live with the solution of having to decode the data property. :slight_smile: