"Device-to-smartphone" and "smartphone-to-device" technologies

Nowadays the poor embedded developer faces new challenges that go outside the limited world of C/C++ program on a resource constrained hardware. Now the hardware isn't so resource constrained (Raspberry and many other embedded Linux boards or SOM are just examples) on one side. And the device isn't isolated anymore, it is usually connected (directly or indirectly to Internet).

The Thing/device connected to Internet (IoT) is now a reality on all embedded hardware. With ESP chips you can have a WiFi connection for a couple of dollars. With many Cortex M3 you can have a wired 10/100Mbps Ethernet connection. lwip is the piece of software to have a working TCP/IP stack on those hw. Now we are at the application high-level layer.

Yesterday a typical request for an Internet-oriented device was HTTP server (I designed such a system one year ago).

Today the request is different. The end-user, at least in most consumer applications, doesn't want to spend time for DDNS and port forwarding. Moreover he wants to use his smartphone and not a typical desktop PC.

So now we have typical appliances that can be managed by a smartphone. You can configure the washing machine through your mobile phone and you can be notified when the clothes are ready.

This is a typical request for an embedded Internet-connected device. It should communicated with a registered smartphone. The main question of this post is: what are the technologies that allow a device to communicated with a smartphone?

I understood this problem can be divided in two completely different challenges. The first is "device to smartphone" direction, with its solutions. The second is "smartphone to device" direction, with other solutions.

Regarding device-to-smartphone, I think the best solution is "push notifications". The push notification technology allows to reach whatever mobile phone in every situation, even if our app isn't running (I understood Android apps can be started automatically at startup, but iOS apps can't be started automatically at startup... so a push notification is the only solution). This technology allows to reduce traffic so increasing battery duration.

How to create a push notifications from an embedded device? I understood we need to register (and pay for) a notification service, for example Google Firebase Cloud Messaging (FCM). A notification can be sent with an HTTPs request to a server, so we need an HTTP client and the full TCP/IP plus TLS stack. Of course we need a custom app running on the mobile phone: without a custom app, the notification can't reach the smartphone. This is a big issue, because apps and embedded fw are two completely different monsters that need different technical skills.

The smartphone-to-device direction is more critical, at least for me. The app running on the smartphone can't reach directly our device, that is usually behind a router and configured with a private IP address. The key technlogy is the "cloud", a public server that accepts requests from the smartphone and connections from devices on the field. The cloud server is able to link the smartphone request to a connected device. Here the high-level view block diagram is clear, but I can't go down to the next level of details.

Which protocol do use the smartphone during communcation with the server? Maybe HTTP(s). Which protocol do use the device during communication with the server? I think we have many solutions: MQTT, HTTP(s) and so on. What are the software to use in the server? I don't know.

As a developer, I can think to rent a physical server (the infrastructer) from service providers and: install an OS (maybe a Linux variant), install a HTTP server (for smartphone requests), write and run server-side scripts to process smartphone requests, install a custom software that communicated with the embedded devices. Wait wait, but I don't have the skills to create and maintain a serious, reliable and secure server that is reachable in Internet. Does it mean we need another guy in our team?

Another possibility is to rent (and pay for) a cloud service that let the smartphone to "chat" with the device. MQTT broker? PubNub? Amazon Web Services (AWS)? I don't know.

I also read about serverless solutions where you wrote the program and "launch it in Internet", without knowing the server where it really runs (you usually pay for processing time). In this case I needn't to have good skills to create and maintain a real server.

I'd like to know if you received those type of challenges in your daily embedded work and how did you try to solve them.

Reply to
pozz
Loading thread data ...

CPU power is abundant nowadays, so is connectivity. Just make your device look like say an RFB (aka VNC) server and use the smartphone/ tablet/PC simply as a terminal. We have been doing that for over a decade on our netMCA devices, long before the "IOT" word had been coined I think.

====================================================== Dimiter Popoff, TGI

formatting link
======================================================
formatting link

Reply to
Dimiter_Popoff

Il 10/12/2018 11:04, Dimiter_Popoff ha scritto:

When the device *is* a server, you need at least Dynamic DNS and port forwarding. Too bad for many users.

Reply to
pozz

If being a server is a problem make the device a client. realVNC can be started in "listening" mode, so the device connects to it, it can do that from behind a NAT router without a problem. This is how our netMCA series has been providing online customer support for over a decade now.

====================================================== Dimiter Popoff, TGI

formatting link
======================================================
formatting link

Reply to
Dimiter_Popoff

I'm not sure I got your point.

Where does realVNC run in "listening mode"? On a smartphone?

Reply to
pozz

Do you actually want devices to communicate over the internet, or is same-network communication sufficient?

For example, if the smartphone is on the same wifi network as the device, you just need discovery protocols so you can find suitable devices. Multicast DNS (aka Bonjour) is one such mechanism, but there are others. There are also things like QR codes, NFC and Bluetooth which can be used as bootstrapping (used to send an ID that's enough to identify the device on the network and go from there).

If you do need internet connectivity (such as 'turn on my heating' when you're out of the house) then you need some kind of public endpoint.

One thing you could look at is IPv6 tunnels, which allow your device to be accessible if it's behind IPv4 NAT. That solves the NAT issue (although you would need an endpoint to host the tunnel - third party solutions for that exist), and then you're back to discovery (where protocols like SIP allow negotiation of peer to peer connections). But you may decide you'd prefer to go via a central server if you don't want your device listening to the internet.

Once you're wedded to the central server model, the thing you should worry about is resilience. It's a single point of failure that can affect all devices if something goes wrong - how will you handle that? Some cloud platforms will take care of hardware failure, but they're no help if your buggy code causes the database to become garbled or the certificate expires (cough, Ericsson), or one of a million different software things that could go wrong. What's the cost of all of your devices going down at the same time?

I don't know if any ready made platform handles these things, although in theory they should exist. Next question: how much can you handle being locked in to that platform?

Theo

Reply to
Theo

The android version cannot do that indeed.

So write your own viewer, RFB is simple enough, this is a huge advantage. Anything else you contemplate would take a lot more work for android or whatever.

Dimiter

====================================================== Dimiter Popoff, TGI

formatting link
======================================================
formatting link

Reply to
Dimiter_Popoff

never tried it but, Blynk ?

Reply to
lasselangwadtchristensen

Il 10/12/2018 22:17, snipped-for-privacy@gmail.com ha scritto:

It seems Arduino oriented... I will check better.

Reply to
pozz

Il 10/12/2018 15:34, Theo ha scritto: > pozz wrote: >> Which protocol do use the smartphone during communcation with the >> server? Maybe HTTP(s). >> Which protocol do use the device during communication with the server? I >> think we have many solutions: MQTT, HTTP(s) and so on. >> What are the software to use in the server? I don't know. > > Do you actually want devices to communicate over the internet, or is > same-network communication sufficient?

Over the Internet, of course.

This is for smartphone-to-device direction. What about device-to-smartphone, for example when the smartphone is off? I think it's impossible without connecting to a Cloud service.

More questions than answers

Reply to
pozz

If you're going to use a public server to relay between the device and the client (smartphone, or whatever), you have all sorts of ways to do it depending on what you're expecting of the machines in question.

For example, if the device is smart enough to be a web server, you can set it up to connect to the public server (via a TCP connection), and then have the public server forward HTTP to the device. Issues like authentication and identification present a number of complications, but these have been dealt with in various ways. Specifically you'd be doing something like a reverse proxy HTTP server for the public host, which is something a number of web servers (including Apache) can do. You might have to fiddle the back end connection to the device a bit.

If the device is not that smart, again, just establish a TCP connection to the public server, and then let the public server handle the clients. If the clients are to see a web page, then you just write the CGI (or whatever) pages to send requests to the devices as necessary). Any protocol you care to establish on the link to the device. If the client device is to be running an application, then it's your choice what the protocol between the client app and the public server is, although something SOAP-ish might be a good idea (although hardly necessary).

In all cases authentication and identification are issues.

Reply to
Robert Wessel

OK - that wasn't immediately clear from your question.

The device could store messages ready for the next time the phone is connected. There's not much you can do until the phone is turned on.

I think this is actually a lot harder than people think.

Big question: what's the expected longevity of your system?

A lot of the IoT stuff is aimed at essentially disposable consumer devices - after somewhere between 1 and 5 years they stop working. If you want to make your system still work after 5/10/20/30 years there are some big challenges:

- will your node software be upgradeable, and how will you manage that?

- if your nodes aren't upgradeable, can you backend cope with all the different versions of shipped firmware? If they are, this issue won't entirely go away (at some point, you reach the limits of a particular hardware platform and further upgrades are infeasible)

- how will you test over all the models that might have been shipped?

- over time, communication protocols will change (eg SSL 2, SSL 3, TLS 1.0 /

1.1 / 1.2 / 1.3). How will you handle that? What should we do about a product that's speaking what's now regarded as insecure SSL 2?

- a lot of IoT players are quite small. There's a substantial chance that any platform you pick goes out of business, gets bought out by someone who isn't interested, 'pivots' to something else, or jacks up the price to infeasible levels.

- running things yourself will insulate from changes at a third-party platform. But can you commit yourself to running the system in 5/10/20 years?

- what is the revenue stream that will keep the lights on? Funding everything from an upfront purchase price tends not to work.

On smartphone it's even harder. iOS is dropping support for 32 bit apps. So any app that hasn't been updateed in the last 5 years isn't going to work any more. Can you commit to keeping your software up to date (even if it's no longer a shipping product)?

One option is to rely on simple technologies that are sufficiently baked in to society that they aren't going away any time soon - for example voice calls and SMS. That fixes the smartphone end, but not the platform end (still a need to provide an SMS gateway)

If this is not a product, but merely a means to an end (eg you have a project for 6 months and don't care what happens after that) then a lot of these headaches go away.

IFTTT is a possible option, although I have no experience of it.

Theo

Reply to
Theo

You insist suggesting a point-to-point (device-to-smartphone) solution. I don't think this can be done.

For notifications, the device could alert the smartphone at any time, when the user is working at PC or playing tennis. In this case, the smartphone OS is sleeping, so all the services/apps aren't using Internet (all connections are closed). How the device could reach the smartphone/viewer?

Even for generic remote control your solution doesn't work. The user may want to connect to his home at any time, when the device behind a router doesn't know. How the smartphone could reach the device?

Reply to
pozz

Sorry, I mean when the smartphone is in standby (display is off while your are watching TV). Even if the device and the smartphone share the same home network, how to reach the smartphone to alert for some event?

I know, this is the reason of my post.

All those are good questions... from a technical point of view. However we are on the market and we need to sell products. Do you think the end customer asks so many questions when he is purchasing IoT gadgets on Amazon? I don't think. So here the main question is how to satisfy the stupid requirements of modern customers: playing with his smartphone

*today* in couple of minutes. He isn't interested (and he will not be able to understand) if the gadget will work for 1 or 5 years.

You will not sell any product with those technologies, even if they are more robust and should have a longer life.

If I'm not wrong, IFTTT needs an already working cloud server. It talks to servers, not real devices behind a NAT.

Reply to
pozz

So how does the device reach anything you contemplate running on a phone which is off. You seem to keep on trying to find a solution to your quest "how do I have a DNS capable server without having one". The answer to that is obvious enough I hope.

Oh it works and has worked for about a decade now. The netMCA keeps on attempting to connect to the listening VNC nodes (you can set multiple of these) every 20 seconds or so.

====================================================== Dimiter Popoff, TGI

formatting link
======================================================
formatting link

Reply to
Dimiter_Popoff

I usually don't power off my smartphone when working at PC or during a tennis mach bettwen hobbyist. The smartphone is in standby and this means all apps aren't active, in other words they can't communicate over the Internet (at least on iOS).

However there is a mechanism that let to push something to a sleeping smartphone, this is "push notifications". It's the same technology that creates a notification on your phone when you receive a Whatsapp message.

For what I know, this technlogy is based on a high-priority service running at the OS kernel level. It's a process that can always communicate over the Internet thanks to a persistent TCP connection.

Google and Apple have service for app developers that let them send notifications to one or mode smartphones. Many other services were born that let the developer to send notifications in a simpler manner.

I think this is the only way for device-to-smartphone alerts/notifications. IMHO here your RFB/VNC doesn't work.

I can't see how. RFB is a TCP protocol. This means one is the server and one is the client. As we already discussed, the device behind a router can't be a server, so it must be a client.

Now, the server should be the smartphone. We can think to update our public address on a dynamic DNS service when the user open our app (myphone.dyndns.com). The device/client behind the NAT could try to connect to myphone.dyndns.com every 20 seconds. The app should be able to create a connection with the device in maximum 20 seconds. This could be ok in some cases.

However I'm not sure if the mobile operator will forward to your phone all the TCP packets, including the ones for TCP connection establishment. Are you sure the mobile operator assigns to your phone a

*public* IP address, even if dynamic. In the latter case, you would need port forwarding on mobile network... much more difficult than port forwarding in my home.
Reply to
pozz

So in what will your app differ from a VNC viewer app which "pushes a notification" in the respect under discussion.

You will likely find better answers regarding how to write android/IOS apps on one of their forums, not sure many of us here know a lot about that. At least I don't know much if anything. But surely since there are apps doing it it should be doable, unless there are services in these systems available to politburo members only (no idea if this is the case, nor would I be surprised at all if it were).

Well RFB is widely used over tcp, I have not seen another use of it.

Not at all, routers can do port forwarding; VNC (RFB) uses port

5900 in listening (viewer is server) mode and 5500 in active (viewer is client) mode.

Well I can't be sure what the phone operator does today let alone what they will do in the future. Whether the operator provides internet access or just some limited part of it is another matter. But basically if they claim to give you internet access this is what they should be giving you for the money you pay them.

On Android I can't even see the IP address my provider has given me unless I look say at the logs of my domain when I visit it. May be I just don't know where to look of course but I don't even know if the phone has a real IP address, in fact I doubt that. So when I want to get around the restrictions google and the rest of the gang have put on the phone's net access I just use a server under a domain I control in some way google can't yet block if they want to stay on the OS market (http, ftp) so both devices can find each other. If you want to sell on the mass market (you mentioned Amazon in another post I think) the $5/month such a server costs would be the least of your expenses, and you will not even need a dynamic dns as you will have your own domain included in the $5/month. My hosting provider allows me to set subdomains such that some of them are on their server and others are not, e.g. the online support listening VNC runs on my desk via my home internet.

Dimiter

====================================================== Dimiter Popoff, TGI

formatting link
======================================================
formatting link

Reply to
Dimiter_Popoff

Yes, this is an experise area that in production quality requires specific skills. My field is building this kind of systems from sensor hardware to cloud and applications. I've done all of the parts to some level to understand the basic challenges, but I don't even dream of building a production quality system by myself.

I've worked with few developers that have been able to grasp embedded and serverside, but this is quite rare and few.

Doing IoT securely is then an another matter - I've had the luck of working with the best in this field and picked up enough from them to know what are the main limitations and how difficult this is to do well.

As a minimum team for doing something you're describing, I'd pick a few talented persons with skills in:

-Electronics/FW to get the HW done

-Design/mechanical, especially if it has to look nicer than typical industrial stuff

-Cloud/server/mobile developer.

With these you could start, but to do something bigger you need to go deeper in each part. As an example, mobile developers come mostly in two flavours: iOS and Android. And then there's the possible web interface. And I've only ever seen one single developer being able to work well on web UI and protocol between device and cloud.

So yes, you'll need an another person.

Check out the Google Cloud, Amazon AWS and Microsoft Azure IoT architectures to get an idea. There's a lot of buzzword compliance (AI, ML, IoT..), but by checking those three you can get the basic idea on how it's usually done. There are a lot of product/application specific stuff you'll need in addition, but you can start with any of those.

Serverless is really nice approach for cloud, but finding true experience in that field is difficult.

If I were to create a small scale mobile-controlled IoT device, I would pick

-Linux SOM or ESP32 (if you've got mains power - battery is more difficult!)

-Some of the commercial cloud stacks or my own MQTTS/AMQPS server with my own server.

-Firmware updates from the server (PKI signed with offline keys!)

-Web UI with some node.js based stack

-Mobile app to connect through the cloud - there are tools that allow reusing web development - not the best ones, but possible.

-Pay at least some thought on how your system will be attacked and how to mitigate

Some thoughts - now need to get back to work..

--
mikko
Reply to
Mikko OH2HVJ

Do you think a simple Cortex-M3 running FreeRTOS/superloop with lwip TCP/IP stack is not sufficient? This hw actually runs a HTTP server without big issues.

Any real suggestions? I tried to read Amazon Web Services and Google Cloud Platform, but sincerely I lost the way.

Do you mean a MQTT broker? So you imagine that the app on the smartphone is a MQTT client, as the device?

Yes.

Web UI running in the app of the smartphone?

Yes. The mobile app should connect to the cloud through MQTT or similar protocols, right?

Thank you very much for your post. Those are precious info.

Reply to
pozz

Yeah, but it's not running HTTPS. I think a CM3 would be woefully outclassed by having to slap SSL on top of things, and if you're not securing your traffic you're adding to the Internet of Crap problem.

--
Rob Gaddi, Highland Technology -- www.highlandtechnology.com 
Email address domain is currently out of order.  See above to fix.
Reply to
Rob Gaddi

ElectronDepot website is not affiliated with any of the manufacturers or service providers discussed here. All logos and trade names are the property of their respective owners.