EP 05: Arlen Nipper & MQTT

00:05
Speaker 1
Hello and welcome to the most recent episode of the Human and Machine podcast. If you have been listening to our series of podcasts, you would know that it is all about the industrial technology and manufacturing space in South Africa. This is podcast number five. We were just discussing it feels like it’s been many more already. This is podcast number five. If you have been listening, thank you for the feedback and the comments. We do see those listening numbers increasing. We really appreciate the feedback. I’m here with my co host, of course, Lenny. My name is Yaku. Lenny, podcast number five. Unreal.

00:40
Speaker 2
Yeah, no, it’s great, Yaku. I’m kind of enjoying it. And I think we might be a little bit nervous for today’s podcast. We really pulled out the big guns. We’ve got a big heavy heater here that we’re going to interview. So very looking forward to the session that we’re going to have.

00:55
Speaker 1
I do feel a little bit intimidated by this episode, but we really excited. So you would have noticed over the last few episodes that we’ve spoken a lot about industry 4.0 digital transformation and really what many people described as buzzwords. And of course, part of their discussion is the crucial role of adopting an IoT, or Internet of Things approach, and more importantly, a network. And if you’re following the trend industry, you would see that the benefits are already being realized by many businesses, and that trend is definitely set to continue. I think Gartner predicts that by the end of this year, there will be an install base of around 20,000,000,020.4 billion IoT devices worldwide. What we have also heard from some of the conversations is that many of these IoT projects fail due to unpredictable costs.

01:47
Speaker 1
And in many cases, these projects have become non starters. And obviously a major factor behind this, and probably one of the biggest contributors to the cost per device is simply the amount of data being transmitted and how that is being handled.

02:01
Speaker 2
Exactly.

02:02
Speaker 1
And that is a cost that of course can be mitigated by choosing an efficient way of communicating between these devices and applications. And then of course into our industry, Euro MQTT, which is message query telemetry transport, which is really a lightweight machine to machine messaging protocol, which is obviously due to its very small code footprint and small message size, ideal for large scale networks. So in this week’s episode, we’re super excited to speak with. I feel like we need a drum roll with Alan Nipper, who is firstly the president and CTO at Cirruslink, very importantly the co inventor of MQTT and just a plainly industry legend. So, Alan, thank you. Welcome. And it’s so exciting to chat to you today.

02:52
Speaker 3
Well, thanks, guys. I really appreciate the opportunity. And really, you don’t need to be intimidated. We’ll go into this and talk about all the cool things that we can do with MQTT, especially how it applies to the industrial Internet of things. And I’m very pleased to see its progress and really looking forward to speaking about that today.

03:16
Speaker 1
Fantastic. Alan, you’ve been in the industry I think 42 years, so you’ve really, from your early days, I’d love to get a view of, I think you started off as a SCADA automation engineer in your early days. I’d love to get a view of maybe some anecdotes from what life was like back then and how, where we are today. That would be great.

03:39
Speaker 2
Sure.

03:40
Speaker 3
So interestingly enough, so I was an electrical engineering student at Oklahoma State. And about second year in, I’m going, man, this is boring. I got to get out of here. I’m going to do something else. And on my way to my advisor’s office, I saw a job posting and it said, looking for intern students for the summer at Amico Oil company. Amico. That was before BP bought them and they were located in Tulsa, Oklahoma. So instead of quitting, I signed up for the internship. And so I show up at my first day at Amico and I said, okay, what am I going to do? And I go, well, Arlen, we’ve got this new department over here. It’s called the microprocessor R and D Lab.

04:24
Speaker 1
Wow.

04:27
Speaker 3
This was in 1976. And so that summer I got to wire wrap an RCA 18 two microprocessor. It was the first ever CMOS microprocessor. It ran at 800 khz. And we had two k eproms that we would write machine language, compile them on the IBM mainframe. The IBM mainframe would then burn the code on the eprom. I would take the EPROM, plug it into my circuit board, see if the program would run, and then rinse.

05:00
Speaker 1
And, I mean, that’s the kind of hardware that would typically be lifted into the building with a hoist or a lift.

05:06
Speaker 3
Oh, yeah, absolutely. So after that summer, I came back and I said, look, I know exactly what I want to do. So I basically kind of concentrated on digital side of electrical engineering, got a minor in computer science, and then coming out of Oklahoma State, the Koch brothers coke oil had just started. And so I got a job with coke oil, and they were just building a refinery in central Oklahoma and building the pipelines down to Texas to deliver ethane to the plastic markets. So, nice thing about coke, you could do anything that you want. So over the course of building those pipelines, I got to do everything from run the ditch, which put conduit in the ground, connect wiring to.

05:53
Speaker 1
You will probably have to explain what a ditch. Which is.

05:56
Speaker 3
The ditch, which is. That’s what you dig your ditches in the ground to put your conduit in.

06:01
Speaker 1
Exactly. There’s nothing computer it related to that.

06:06
Speaker 3
Nor is there setting a 5000 horse electric motor, connecting it to a centrifugal pump, but then taking and running the wires from the motor to the Toshiba vacuum tube starters. And then 1978, Modicon had just come out. That was the first year for Modbus protocol, and I got to learn how to program a Modicon 484 and then connect the Modicon 484 to an RTU and connect the RTU to a Bell 202 modem that ran at 300 bod. And then all of that went back to our ScaDA host system, which was running on a PDP 1144. So over the eight years that I worked for coke, I got to put in complete pipeline SCaDA systems, got to learn all of the protocols, legacy protocols of the 1970s, put in tank farm automation systems, truck loading systems.

07:06
Speaker 3
So just a lot of general purpose experience in the industrial world. So after that, I was one of the co founders of a company called Arcom Control Systems. And what we did was we built embedded computers. But not just embedded computers. They had to do something. So we manufacture protocol converters. So now we’re in the. We had all of these customers that had protocols going all the way back to the 70s. So we had a very good business in building protocol converters for the oil and gas, water and wastewater electric utility.

07:43
Speaker 1
Market until, obviously, standards like Modbus.

07:45
Speaker 3
Yes. So we would take all these crazy protocols that nobody has ever heard about, like Westac and Redac and Perk 22 and Tano, and we would convert those to Modbus. And life was great. But you have to realize is that in the mid 1990s, a really disruptive thing happened in the United States in that at and t was our primary communication provider for all of these wide area networks. Well, they got deregulated, and within the course of about two years, our whole multi drop lease line infrastructure just kind of fell apart. So guess what happened? The VSAT guys stepped in. So now we had GE, Spacenet, Galat, scientific Atlanta, at T Trident. They all had very small aperture terminals you could set out at your tank farms or your booster stations. So that started to be adopted. But guess what?

08:45
Speaker 3
Everyone had their own proprietary transport protocol, right? So now we had a double whammy. We had proprietary protocols. You had to wrap them in proprietary transports. Rolling up to that is that 1998 Philip 66 got the first TCP IP based VSAT system. Now, what you have to realize, Yako, is that in the 90s, if I went into a skated department and said the word TCP IP, really, nobody knew what I was talking about, because everybody was RS 232. So Phillips looked at this and they go, Arlin, this is very expensive. We’re polling over the VSAT. It’s got long propagation delays. We want more data, we want it faster, we don’t want to pay more. And we’ve got an IT department, and they’re using this IBM product called MQ series, and it uses a technology called Pub Sub.

09:44
Speaker 3
Now, if you think about what were doing in the protocol converter business, it really was pub sub. It’s just that the subscription was implicit to that ScAda host system. So IBM was working with Phillips, Arcom was working. So I got to work with an IBM fellow. His name is Andy Stanford Clark, and he was the co inventor of MQ. So what we did in about six months is we took this huge messaging structure that IBM had originally and the way that their pub sub worked, and we smashed it down, and we looked at things like state awareness. And that was the original version 30 version of MQTT was released in 1999, and the genesis of it was for a mission critical real time control system. So that’s what’s ironic to me, Leonard Nyako, is that it’s almost come full circle.

10:45
Speaker 3
It started as an industrial focused transport. It was adopted heavily by it, as we see today. And now it’s come back full circle to where we’re getting it reestablished out in the operational world.

11:02
Speaker 1
Yeah, fascinating. We actually spoke about it a few episodes ago about that exact point. So really it was an opportunity that was born out of, like, most other innovations and being innovative out of just the need to make better use of new infrastructure and new other technologies. And there was the birth of MQTT. It’s incredible. It doesn’t feel like it should be that long ago.

11:26
Speaker 2
Exactly. I think that’s the scary part about it. We’re talking about the 90s here, and a lot of people that’s probably listening to us probably go, well, they’ve only heard about it recently. Right? It is quite scary.

11:39
Speaker 3
Again, as you can imagine, let’s put this in context. IBM at that time were the only people that had MQ series, and MQ series was a $600,000 piece of software, and you had to get that just to get that little MQTT broker that was at the bottom of that. But what happened is that several things. First, Andy and I worked very hard on two things. One is to get MQTT through a standards body. Okay? So it’s in two standards bodies now. And to get the eclipse Paho project set up so that there could be a community where anybody that wanted to implement MQTT could go to and get started. And that is the Pahoo project on the Eclipse foundation.

12:30
Speaker 1
You gave it away.

12:32
Speaker 3
We gave it away, absolutely. Well, I tell the funny story. Andy and I literally had the first version of MQTT up and running in about six months. It took us 18 months negotiating with the IBM lawyers to make it open, because, remember, 1998, IBM wasn’t in the. They weren’t in the game of open source at that time. But had we not worked so hard to make it open originally, we wouldn’t be having this conversation.

13:02
Speaker 1
Absolutely. The world would if the landscape would have looked very different. Definitely very different. I wanted to just quickly, MQTT, in terms of IoT, we’re using a lot of tlas, a lot of three letter acronyms today. But MQTT, with regards to the industrial Internet of things, obviously that interoperability that you mentioned makes it a little bit difficult in the industrial space. And I think that was probably the birth of something called Spark plug, which you would probably have to maybe tell us a little bit more about, and especially our listeners, where that fits in.

13:44
Speaker 3
Okay, well, let’s wrap back. So my partner Chris and I started Cirruslink six years ago, basically to focus on industrial applications of MQTT. And were working on a project. Our first big customer was Plains midstream, which is one of the largest pipeline companies in Canada. And were working with a product from, it’s now owned by Aviva. But at the time it was Val met was called Oasis DNA. And what people may not realize, even Schneider may not realize this, is that they had the first ever MQtT based SCADA system. Because when we did that system for Philips 66, well, that Scada host system had to talk MQTT. So literally, Oasis DNA has had MqtT in his Omnicom protocols.

14:36
Speaker 1
That’s incredible. Didn’t know that. That’s obviously very prevalent to the oil and gas space. I mean, that was the birth of it, right?

14:44
Speaker 3
Yeah, that was the birth of it. That was the other issue. So here we’re working with planes, and we had no tools. It was just a black box. And I needed some tools to be able to help the customer see messages, validate our process, variables, things like that. So I was looking for a platform, and somebody said, hey, Arlen, have you seen this product called ignition? I said, no, I’ve never heard about it. So now we have to tell the story of, well, one of the integrators that I was working with called Travis Cox at inductive and need to. Can you set up a meeting with Arlene? So I flew to Folsom, I think this was four years ago ish, and had a meeting all day, meeting with Travis, and Travis and Colby and the team, they go, this is pretty cool, Arlen.

15:35
Speaker 3
Here’s an SDK. And so I took that SDK, flew back to Kansas City, and in a few days, I had a working prototype of our first ever product called MQTT engine. Now. So the story is that MQtT never had an appropriate platform for people to see how good it was until ignition came along. So I blame Don Pearson and Travis and Colby and Carl for not coming up with ignition 20 years earlier. So we could have leveraged MqtT earlier.

16:10
Speaker 1
I think we all feel the same way.

16:15
Speaker 2
Sorry. At that point, did you then see the adoption of MQTT moving a little bit away from just the kind of oil and gas pipeline kind of industry into more industrial sectors as we see today? Do you think that was kind of.

16:29
Speaker 1
The splitting road of where that was the defining moment?

16:32
Speaker 3
Yes, that was the defining moment, when we could get it into a platform that was already across all these other verticals. All of a sudden, the exposure went from oil and gas, water and wastewater, wide area scada. All of a sudden, then it started to encompass all of the verticals that inductive were in. Now, having said that was the first version of MQTT engine. And we started going to conferences and we started seeing that the common denominator, the biggest common denominator with IBM, Watson IoT, Google Cloud platform, AWS, IoT core, Azure IoT event hub, is they all use MQTT. And the manufacturers out there, the hilsers, the opto, twenty two s, the Moxas, the Advantech, they were all talking about MQTT. But as I looked around, everybody was doing it different.

17:29
Speaker 3
So the beautiful thing with MQTT is you can publish anything that you want on any topic. So, Janie garage door openers use it. Roomba vacuum sweepers use it. Facebook messenger use it. That’s the beautiful thing of it. But when you get into a particular vertical, you’ve got to start setting some standards. So what we did is the team at Cirruslink and all my developers, and we’ve all been using MQTT for the last 20 years, I said, guys, let’s take best practices lessons learned, and let’s write a specification that will help standardize that. So Spark plug is a specification, and I tell people it does not change MQTT at all. It does three simple things. It defines a standard topic namespace that we can all agree on. That makes sense in the industrial sector.

18:26
Speaker 3
Number two, it realizes that we need to keep MQTT lean and mean. So although the cloud providers and everybody were starting to use JSON very prevalently, in sending data around, we’re still constrained. In our world, bandwidth is neither free nor unlimited.

18:46
Speaker 1
Very interesting.

18:47
Speaker 3
Start taking PLC registers, RTU registers, and flow computer registers. And if you do all of those in JSon, all of a sudden your messages get huge, and you can’t afford that with low bandwidth on cellular or VSAT or spread spectrum radios or whatever. So the second thing we did in spark plug is we took a very prevalent technology called Google protocol buffers, and we designed the schema to efficiently encode process variables binary and put them in the payload. So we’ve got a topic namespace, we’ve got a payload definition. And then the last thing that the spark plug spec states is state, because a lot of people like MQTT, oh, I can publish this, I subscribe to this. But you have to realize that in a real time control system, you’ve got to have state.

19:43
Speaker 3
No real ScaDA engineer is going to trust a system that I tell him, oh yeah, the PLC will publish a register when it changes unless I know that device is out there. And that is that death certificate that Andy and I built into NQTT, so that we have continuous session awareness for all of our remote sites. So let’s take Philip 66, for example. They’ve been running their pipeline infrastructure for 21 years. On MQTT, they have 5000 Alan Bradley plcs, 1200 flow computers, 500 truck terminal and tank farm systems. But they know the state of all of those edge of network devices in real time.

20:29
Speaker 1
Sorry, that is the intrinsic differentiator when it comes to industrial IoT.

20:33
Speaker 3
Yes, that is so I always be very careful here in that we’ve got OtMQTT brokers that care about state and then we have itMqtt brokers that they don’t care so much. Cool.

20:50
Speaker 2
Olin, I think you also hit an important topic there about, obviously you’re using the Google protocol buffers to really package this small and amount of bandwidth. Kind of utilization of MQT is really bandwidth efficient. Another thing that also is quite interesting is the difference between other type of polling, normal polling protocols, and the way that MQTT is also different from that to also help and alleviate bandwidth, such.

21:19
Speaker 1
As opcua, for example.

21:20
Speaker 2
Exactly, yes. And that’s the notion that I only send when I need to send.

21:26
Speaker 3
Right. Well, typically what we’ve seen, guys, over the last 20 years that we’ve been doing this kind of rule of thumb is that if we replace any pole response SCADA system as we migrate it towards pure MQTT for the same amount of information at the same update rate, we see an 80% to 95% bandwidth reduction. In 1998, that meant cost savings in your communications. In 2020, that means that’s 80% to 95% more of that stranded data that’s out in the field that you can bring back over the same bandwidth.

22:12
Speaker 2
Correct. And I see we do, especially in the age that we’re living in now, is it’s so easy these days to get a device online, it’s so easy just to.

22:22
Speaker 1
And it should be wide inclusion.

22:23
Speaker 2
Exactly. And we do see that adoption of more and more data and to get that data and to start historizing and to collect that data is becoming a big player in the market.

22:33
Speaker 1
Yeah, cheaper networks, cheaper devices.

22:36
Speaker 2
Exactly.

22:37
Speaker 3
But remember, the three tenets, though, that both Siriuslink and inductive automation preach from day one, is that forget about MQTT and all this other stuff. In fact, I love Tod Ansinger, who is the head IoT executive for Chevron, and he’s on the Spark plug working group. His slide has men in black with the mind eraser. And he says, forget everything that you know about.

23:08
Speaker 2
Yeah.

23:09
Speaker 3
How would you want it to work today? So the tenets are, in today’s world. We should look at connecting devices to infrastructure, not to applications. So decouple.

23:21
Speaker 1
Exactly.

23:22
Speaker 3
Number two, provide a single source of truth for all process variables. Number three, remember, we’re in the world of operations. So I need to show Mr. Operations manager a better, faster, more scalable, more available, more redundant operational system, first and foremost. Because if I can’t show that on the plant floor, at the pipeline, at the solar farm, then forget about digital transformation. They’re still going to put in poll response, they’re going to have these silos of data, so we’re never going to get there unless we think differently about how we architect our overall system.

24:06
Speaker 1
Yeah, and that approach.

24:07
Speaker 2
Exactly. I think we’ve been stuck in a mindset of the typical old automation pyramid, if I can call it that, with the different layers, and I have to jump from layer x to, or layer zero to layer one to layer two, to get my integration all the way up to my ERP or my business systems. And I think with this broker centric kind of topology that we can now employ, it’s 100% right. You take the mind eraser, you erase what you’ve learned about that kind of hierarchy, and it’s a complete different ballgame.

24:40
Speaker 1
It’s easier said than done for a lot of industry folks, eh? I know, just a familiarity, and this is how we have done it, and probably always will do.

24:49
Speaker 2
Exactly.

24:53
Speaker 1
I quickly wanted to maybe just get a comparison when we’re talking about adoption in the industrial sector, in comparison to other industries and verticals, what does that look like? Is there an earlier adopter versus a legged, depending on industry or use? Is one catching up to the other? What does that comparison look like at the moment between industrial versus other industries and verticals? Well, again, and can you differentiate them so easily? Sorry to interrupt you. Can you now probably today differentiate or split them so easily? Maybe it’s all one supply chain nowadays?

25:33
Speaker 3
Well, what we found out is that, to your point earlier, yes, the genesis of MQGT was in the oil and gas market, and for all the right reasons. And then when we got the modules into ignition and that became available, the oil and gas industry was very quick to start picking that up, for all the right reasons. But at the same time, through all of the integrators that work with the ignition platform, all of a sudden we had an application come in from pulp and paper, and then logistics have just gone off the chart here lately. And then the food and beverage industry starting to pick it up, and then solar farm monitoring starting to come up.

26:24
Speaker 3
So all of a sudden, I can say that we’ve got MQTT solutions across every vertical that are in all the charts that ignition shows, as far as their customer base. So the adoption has been fantastic. And then we’ll start talking about how do we leverage this into that next step where a customer now he’s got an MQTT infrastructure. And really the cool thing is that you build your OT infrastructure to be it ready. All of these things that people are looking at, cloud, and we’re all starting on these digital transformation projects, and we’re already hearing they’re failing, they’re failing. And I think there’s several reasons for that we can go into, but we’ve got to focus on OT. It’s not it down to OT. It’s providing OT with the tools and the technology to be cloud ready when they want to move that direction.

27:28
Speaker 1
And typically in the OT space, you are talking about historically, you’re talking about equipment and devices that were built to last forever, number one. Number two, not built for being online. And we see it in South Africa, and we always jokingly refer to the scenario that we have of it versus OT. And we actually had a few. On a few of the recent episodes of the podcast, we spoke with an end user specifically that spoke about that a little bit. That divide has shrunk as a result, and it’s definitely becoming one more cohesive team now as a result of the introduction of these new technologies.

28:10
Speaker 3
I agree. I see that daily now. But we start thinking about this. That goes back to that single source of truth tenant, right? I want to get a piece of data. I don’t want Modbus register 40,012 with a value of 17. And that’s always my joke, right? Is that what do we do today? We engineer a system. We get Modbus register 40,012. It comes up to the dashboard. And what do we do? We double click on it and we edit it to give it context, and we give it engineering units, we give it engineering ranges. And we say, oh, it’s a zero to 495. We’re going to have to scale it to zero to 100 somethings. Those somethings are going to be kilopascal. And then when another application wants it, they got to start over and they do it again.

29:01
Speaker 3
And what I see is that people are using our injector modules. We have modules for Amazon, IBM, Google, and Microsoft, where we can take and publish process variable data into data lakes. But what I’m figuring out is we’re kicking the can down the road, because now this customer’s got a terabyte of data in a data lake, and now we got to hire a system integrator to get it out of that data lake and do something interesting with it.

29:31
Speaker 1
Yeah, and try and make something useful context out of it.

29:35
Speaker 3
Right. Then the CFO is coming down after twelve months, the CFO is coming down to operations and say, what are you guys doing? And operations said, well, we had a big executive meeting. You guys wanted all the data operational data in the cloud. So we put in a data lake. So then the CFO goes, okay, well, who’s using this? Well, Mr. CFO, we really haven’t got the tools to pull out of the data lake to take Modbus register 40,012 because we can’t figure out what it is.

30:05
Speaker 1
We’re still not sure.

30:08
Speaker 3
So I will have to say if you or your listeners saw a pretty important announcement last week from Amazon, and they just launched a new service called Sitewise. And sitewise is a service that Amazon has that lets you build a machine model. Finally. Now I can define a CNC machine. I can define a centrifugal pump. I can define whatever machine that I’ve got out there. I can build a data model. So inductive automation and Cirruslink worked with Amazon while they were developing this. So that now, and we can announce this today, now we can use spark plug coming from ignition on the birth message going into IoT core. It knows how to build a machine model in sitewise. Then we can populate the process variables of that model using d data messages. So imagine this.

31:15
Speaker 3
Imagine you’ve got a customer that has a perfectly great ignition system in the factory, and he wants to start on his journey to digital transformation. Now, literally all he has to do is install a transmission module, take and build the UDT of the machines that he wants to represent in Amazon. Point it at their MQTT broker, and in less than a minute, he’ll have models and data in Amazon.

31:50
Speaker 2
That’s impressive.

31:51
Speaker 1
That’s very impressive. That’s incredible. I’ve made a note here just on what you spoke about earlier in terms of typically how to get started. So when we’re talking about IoT and IIoT projects, can you maybe share some of the real world use cases where NQTT really has sort of shown and proved its worth? And how you typically at Siriuslink, how you engage with customers wanting to embark with IoT and IoT, there’s obviously many different ways to skin a cat. And very often those requirements come from top down in the form of a business case or simply just the want and need to connect as many things as possible. What does your approach look like typically when you look at these projects? Where do you start?

32:42
Speaker 3
Well, first of all, the customer has to realize that they want to take that step. So what they’ll do is they’re either working with an integrator that knows ignition and knows all the tools and knows about the MQTT technology, or more and more they’re working know, like I said, Amazon or Microsoft, and they’re going, wait a minute, there’s this MQTT thing that you’ve got. How do we take advantage of that? So, on the one side, I will tell you that water, wastewater mining, anything that’s wide area Scada, it’s kind of a no brainer, because they already realize that if I can save 80% to 95% on my cellular, my VSAT, my radio, I can get more information, I can get it faster. Then that’s all the proof point right there.

33:38
Speaker 3
But then you go to manufacturing and you go, well, Arlen, where does that make sense? Well, what we’re seeing is that people have factories where they are burning up their internal networks. Yes, they’ve got high speed networks, but for their mes system, they want this digital state, and they want to catch it within 1 second. And so what are they doing? They’re polling at 1 second. Well, what we’re seeing is that they are taking these factories and knocking them down into cells so that we can put ignition edge right by the area that they want to watch high speed. And it can sit there and pull that digital state at 250 milliseconds, but only publish it when it changes. Now, the other thing in the factory is that factory networks go down. Mes systems don’t like gaps in their data.

34:29
Speaker 3
So one of the other huge advantages using the spark plug specification is that we’ve now got the ability for store and forward. So for everything that I’ve talked about, if our network goes down, we still have objects, these spark plug objects that are process variables with their timestamp, with their engineering units. So I can just queue those up. And now when my MQTT broker comes back up online, my network comes back up. I can take all those historical process variables and republish those with the historical flag set to true and do backfill into my time series database.

35:10
Speaker 2
And I think, Alan, a big thing with that as well, is we’ve made the example of a state that changes on the machine, that’s being monitored, that’s now being sent to potentially an MES solution. But that data is now available for any potential other application that also requires that data. And we’re not increasing the bandwidth requirement. It’s not another system that’s now also polling for the same state value just via a different mechanism that nobody even knew about. It’s really decoupling that and becoming this one broker central kind of architecture where not just one piece of application can get that data, but it’s now available to anybody that subscribed to that topic.

35:51
Speaker 2
I think that’s also a key thing to understand is that it opens up this sharing of information to so many different applications that we struggled to do in the past because of the way that we need to integrate and have that individual silos of integration.

36:05
Speaker 3
Yeah, a very good example of that, Leonard, is Canary Labs. So canary, looking at the oil and gas market, talking to customers, and they had several very large customers, they said, well, how are we going to get the data? And the manager for this large oil company said, well, hey, we’ve got it. It’s all right here in our MQTT. So you need to do spark plug and just go in and automatically discover all the tags and put those directly into Canary. So now we’ve got ignition setting there, we’ve got Canary lab setting there, and we have a system that is, we plug them in 10 seconds. They learn every tag, all the engineering units, all the engineering ranges, all of the properties on every tag in ignition, and they’re done. So we have self learning systems. Now.

37:01
Speaker 2
I’ve seen a few of the webinars. I see some of your posts about the scale, and just you talk about things happen, and in a minute it’s available in ten minutes, all of these things just appear. I can’t remember the exact numbers, but.

37:15
Speaker 1
It was a few weeks ago. I think you had a virtual demo. I think it was a few million, if I’m not mistaken.

37:21
Speaker 2
Yeah. So maybe just give the listeners kind of an idea that when we talk about one tag, it’s not stopping there. The scale of this and the birthing of devices, it’s quite impressive when you see your live demo of how many different.

37:33
Speaker 1
What is that scale and what is that capability just comes alive massive.

37:39
Speaker 3
So remember, what we did in the spark plug specification is, remember, the first thing I said is that we define a standard topic namespace. So that canary Labs, ignition, and I will mention it here. The other big company that’s jumping on the Spark plug bandwagon is Aviva. So we have a lot of companies manufacturing signal fire, manufacturing sensors. Do spark plug native, opto 22, all of their I O equipment does spark plug natively. So we’re starting to get this whole ecosystem of interoperability, if you will. So the demo that I do is that I literally have in my lab, I’ve got 43 OEM devices, I’ve got opto, I’ve got Moxa, I’ve got, you name it, I’ve got it. Right. And they all are running ignition edge or spark plug natively. So then I’ve got another properly simulated 2000 plcs.

38:42
Speaker 3
And all of this stuff is connected into multiple MQTT servers, probably beyond the scope of this, but redundancy and high availability is the old keep it simple, stupid principle is that you just add more MQTT servers. So the demo that I do is I have an ignition instance that I’ve installed from scratch, so it doesn’t know anything. I’ve got 2000 devices, I’ve got 200,000 tags. All those tags are changing from one to 5 seconds. And then I plug NQTT engine, which is the module that runs on ignition, that knows NQTT one side, and it knows how to create folders and tag structure inside of ignition the other side. So what I do is I plug into our infrastructure, and in 50 seconds, we learn all 2000 plcs, all 200,000 tags.

39:49
Speaker 3
With every tag, all 200,000 tags, we learn their engineering units, their engineering ranges, any custom properties that you’ve got on that tag, and you are ready to start using that data. Now, if Canary Labs was there, they would also plug in, and in that same 50 seconds, they would learn those 200,000 tags as well. And if we wanted that to go up to Amazon, let’s say, to this new sitewise service, I don’t know if it’s going to scale to 200,000 that quickly, but that’s my point, is we’re plugging all this into infrastructure, not into applications.

40:28
Speaker 2
No, that’s incredible. That’s incredible. And I think sometimes we forget that it’s not just the process variable. And I think that’s a very valid point. It’s all the metadata and all the engineering properties that goes with that. I think that’s just what makes it so much more impressive.

40:42
Speaker 3
Right. And what you have to realize, Leonard, is if you’re using ignition edge for that brownfield equipment that’s out there, and your engineer goes out and they put a new transmitter, a new four to 20 milliamp transmitter on it, and it has a different range. So now he goes into his designer and ignition, he double clicks on that tag, and he says, now, instead of this being zero to 495 is zero to 100, it’s zero to 250 kilopascal. As soon as he saves that, it republishes that tag. So everybody that subscribed to that was now informed in real time. Oh, hey, Leonard. They just changed the engineering scaling on that one process.

41:27
Speaker 2
Think. And I like that all systems, and I think that’s the beauty. Everybody that subscribed just has that new data available and that’s really impressive.

41:37
Speaker 1
Absolutely. So you’re effectively reducing all these layers of technology and integration that you typically would have had in between.

41:43
Speaker 2
Exactly.

41:44
Speaker 3
Oh, absolutely. Again, we have this mindset. We’ve got a 40 year mindset that I see people building out SCADA systems the same way I built out that system for coke oil in 1978. Right. We haven’t progressed much when we think about ScADA. For some reason we’ve got this mindset and we’re not taking advantage of TCP ip the way the Internet of people did. And we’ve got to, I mean, in my opinion, the Internet of people took off because of two things. There was a transport out there that was free, that was, everybody knew it was called HTTP and bulletin boards were starting to use it. And then somebody came along and said, hey, why don’t we define what is in that HTTP transport? We’ll call it HTML. And guess what? The Internet of people exploded.

42:46
Speaker 3
Now in my mind, for the industrial Internet of things to explode, we need a transport just like the Internet of people. And that already is dominant. It’s already established that MQTT is the dominant IoT transport, but we need that HTML part of it, and that’s spark plug.

43:10
Speaker 1
Yeah.

43:11
Speaker 2
And Oliver, maybe that leads into the next little bit of a discussion point is the Spark plug user group. Can you maybe tell us a little bit more about the Spark plug user group who’s currently involved with it and what kind of, is the purpose of that user group, especially with the Spark plug protocol?

43:30
Speaker 3
Okay, well, three years ago, when Siriuslink first thought about the idea of Spark plug, we put it up on our GitHub site and we told everybody that it was open and it was free. You could download the spec, you could download reference implementation code that we had written in C, Javascript and Python. And it was all on our public GitHub. But then customers started asking the hard question and the first one was Chevron. And so we’re on a call with inductive automation and Todd Anslinger is the executive at Chevron for all of automation for Chevron. And he’s saying, arlen, who owns. It’s free. No, really, Arlen, who owns this? Now let me go back to what I said about Andy and I getting MQTT into the Eclipse foundation. So I called Mike Milinkovich, who’s the CEO of the eclipse Foundation.

44:36
Speaker 3
I said, mike, what if we contributed all of the intellectual property for Spark Plug to eclipse. So two years ago that’s exactly what we did. So now Siriuslink don’t own it. All of the IP, the spec, all the represent limitation code is owned by the Eclipse foundation. So then we didn’t want Siriuslink just to go start because we had all this customer feedback of, hey, wouldn’t it be cool if Spark Plug did this? And wouldn’t it be cool if Spark Plug did that? Well, I didn’t want to do that in a vacuum. I wanted to do that under the auspice of having an organization where we could all talk about it. And that was the Spark Plug users group. It was kicked off here a couple of months ago, and the founding members of that are chevron, oring, Canary Lab, Cirruslink and inductive automation.

45:32
Speaker 3
Now, I would invite everybody listening to this podcast to Google Eclipse, Spark Plug. You’ll find the information and join the Slack channel, participate, see what we’re doing. Because what we’re going to do is we’re going to evolve Spark plug specification, but we’re going to do it within this working group so that everybody can become involved. So we’ve invited Aviva and Rockwell and Microsoft and everybody else. Now, organizations tend to be funny about which thing they join, but this is the vehicle that we’re going to mature Spark plug, it’s already adopted. The cool thing is that it’s already out there. It’s not like a pie in the sky. Oh, if we come out with this spec, we’re going to refine the spec through the Spark Plug users group.

46:29
Speaker 1
Yeah. And that’s so important. I mean, if you want to see any kind of rapid growth and inclusion of ideas and just progression on any kind of tech or product, I think the inclusion and opening that up to as many different folks as possible is a very key part of that. I love the fact that all of these vendors are involved and the world has also changed for them. I think the introduction of tech always aims to improve a process or enable a human being. That’s always the driving force behind some of the tech and some of these very large vendors, they’ve had a specific technology, they’ve had a specific way of doing things for a very long time.

47:11
Speaker 1
And there definitely is a sense in the market, in the landscape that all of that is changing and everybody is working towards the same direction. And spark plug obviously is going to play a massive key role in that.

47:24
Speaker 3
The way that I look at it, I do a lot of speaking at universities. Let me put it in this context. What do we have now, if I go into any customer and I go, hey, where’s your Scada engineer that knows how everything works? Oh, well, that’s Joe. He sits over there in cubicle seven.

47:42
Speaker 1
The guy with all the tested knowledge in his head that if Joe disappears, nobody knows what’s happening.

47:46
Speaker 3
Exactly. And so if I go to a university and I talk with their electrical engineer and computer science majors, and I go, how many of you students, if you go into industrial automation, how many of you ever have heard of OpcWay? Raise your hands. Nothing?

48:04
Speaker 1
Yeah, no.

48:05
Speaker 3
Modbus. Alan Bradley, DF one. Raise your hands. Nothing. But if I ask that same engineering students, how many of you have a raspberry PI in your dorm room? Everybody raises their hand. How many of you students are using node red? Everybody raises their hand and they know that they can download the Alexa app from Amazon and do a rule talk to Alexa and it will send MQTT commands to the raspberry PI to turn led on and off. So my point here is that what we’re promoting is a technology that’s going to be own, so that as these students come out of college and they’re looking at this and go, yeah, I know how NQCT works, and it’s not sequestered to just one individual or a few individuals. And I think it’s the democracy.