Multipart file transfer

General
#1

I am using mongoose c++ library for my application to see performance.

  1. My goal is: observe multpart file transfer at same time from 100 users.
  2. My actions are: I am using example big_upload.c, I have modified as per need but it’s a single thread application. I would like to get multi threaded application which create thread per user transfer or precess parallel with multiple users. I am doing memcpy when file data arrived ,which is expensive/ time consuming job so if I do with multi threaded for each socket connection would help.
  3. The result I see is: Single thread application slow down quickly.
  4. My expectation & question is: Not getting clue , how can I make it multi threaded. No such documents found and multithread example doesn’t much help. I am still not getting what mg_broadcast() does in this example? Does it really send broadcast message on network?. Sorry I am from RTL development and new to mongoose so wording makes so difficult to understand. Do you have any multi threaded multipart file transfer example or explanation on how to make it?
#2

Mongoose is single threaded.
It can handle multiple connections quite efficiently, 100 concurrent connections is a very low number to even think about multi-threading.

What’s your issue with the big_upload example anyway? If you can share your code and exact issue, the answer could be more constructive.

#3

Thanks, for quick reply.
Sorry I typed wrong number. I mean to say 1000 concurrent connection. I found its struggling to deal with even 100 client concurrent file upload operations on my test setup. There is always questions on how test setup is design like wireless, wired, 1g,10g connection…etc. we have a standard setup which we use for testing which shows how much time taken by system to upload file and return back to device with response. Wireshark has shows sometimes is just holding up connection.
Anyway, what I want to do is around 1000 client concurrent upload file, we run various algorithm and return back to different devices that file with results.
I am just trying to see simple performance with file upload before buying license as development cost is more than license cost if it doesn’t work effectively.
More technical questions is : is there anyway, user can handle struct mg_http_multipart_part buffer data buffer? So when endpoint call handle_upload(–) it’s just passed data buffer pointer to user and return. User deal with data buffer and released memory when done.
Or you can suggest me how can I use multi thread?
What mg_broadcast() does in multi threads example?

#4

Thank you.

It’s hard to tell something constructive without the example code. I suggest you to make a github gist with a minimal benchmark you’re using, and we can go from that.

W.r.t. mg_broadcast - the purpose of that function is to pass data from one thread to the IO thread that does Mongoose event loop (for (;;) mg_mgr_poll(...);). Mongoose is single-threaded and the core is not protected from concurrent accesses, thus you cannot e.g. pass a connection pointer to another thread and from there, call mg_write(). Instead, you can spawn threads, do your calculations, and mg_broadcast data to Mongoose, which will drain the data down to the client.