HTTPS Client example

General
#1

I have been reviewing all information that I can get on HTTPS client operation in Mongoose OS, and I cannot find a conclusive way to connect to an HTTPS web site. There are several partial examples, some with broken links in the old forum. An up to date example would be a really great thing.

Among the questions:

  1. How to frame up an HTTPS connection with the helper function (i. e. which one to use: mg_connect_http or mg_connect_http_opts, ) and how are these function calls formed up.)

  2. What client CA information is required? It seems to me that in the general case, an HTTPS client MUST be able to provide Server Certificate verification with a CA supplied certificate. Since an embedded client is unlikely to be required to behave like a browser, it is probably sufficient only to get a copy of the certificate for the site or sites that the embedded client is going to access. A practical example would be most welcome.

Thanks for your help,

JSW

#2

There is nothing special to do. E.g.

  const char *server = "https://www.ietf.org/timezones/data/leap-seconds.list";
  LOG(LL_INFO, ("Connecting to %s", server));
  struct mg_mgr *mgr = mgos_get_mgr();
  struct mg_connection *conn =
      mg_connect_http(mgr, ev_handler, NULL, server, NULL, NULL);
  if (conn == NULL) {
    LOG(LL_ERROR, ("Failed to connect to %s", server));
  }
2 Likes
#3

Hi nliviu,

Thanks for looking at this.

My code is very similar to your indicated example, and it works for most https sites that I am trying to connect to. However for one site “https://forecast.weather.gov/MapClick.php?lat=37.1257&lon=-121.6489&unit=0&lg=english&FcstType=text&TextType=1” ,
I am receiving the following console printout of the site’s HTML:

[Jul 24 18:22:12.543] SW ECDH curve 3
[Jul 24 18:22:13.372]
[Jul 24 18:22:13.374] Access Denied
[Jul 24 18:22:13.377]
[Jul 24 18:22:13.379]

Access Denied


[Jul 24 18:22:13.382]
[Jul 24 18:22:13.391] You don’t have permission to access “http://forecast.weather.gov/MapClick.php?” on this server.


[Jul 24 18:22:13.407] Reference #18.b4351ab8.1564017732.8037bf3
[Jul 24 18:22:13.407]
[Jul 24 18:22:13.425]
[Jul 24 18:22:13.425]

To me, the “SW ECDH curve 3” message indicates that a TLS handshake is being attempted. When this handshake is unsuccessful, this particular server assumes an HTTP communication and denies access. The likely reason for the the unsuccessful HTTPS handshake is the absence of an appropriate X509 CA certificate in my ESP32’s file system. I think that my successful HTTPS connections to other servers (“www.example.org”) are occurring because those websites have been specifically designed to treat an incoming connection without X509 certificate as HTTP and respond in this fashion.

Please comment on the use of certificates in HTTPS transactions.

Thanks for your help,

JSW

#4

According to https://www.weather.gov/documentation/services-web-api,

User Agent is required to identify your application. This string can be anything, and the more unique to your application the less likely it will be affected by a security event. If you include contact information (website or email), we can contact you if your string is associated to a security event. This will be replaced with an API key in the future.

Add the user agent as an extra header:

      mg_connect_http(mgr, ev_handler, NULL, server,
                      "User-Agent: Mongoose " MG_VERSION "\r\n", NULL);
1 Like
#5

Hi nliviu,

This definitely works. Thanks for the info.

But I still don’t understand why I don’t need some kind of CA certificate in the ESP32 for HTTPS.

Is this something that is up to each individual server to decide?

Thanks,

JSW