Mediclinic: Utilities Optimised with Ignition and Canary

00:11
Speaker 1
Thank you. So, some good news. We averted the crisis. I spoke with Marcus. So the crisis was this morning, and maybe I shouldn’t jinx it, but essentially, his model that he built broke. And I said to him, Well, were you building it on a VM? He says, No. I bolted on my PI. I said, What do you mean, my PI? He says, My Raspberry Pi. I said, which PI? He said, oh, it was my old weather station at home. I said, geez, Marcus, we don’t want to build live demos on the old weather station pie. But anyway, it’s a crisis averted. So we’re looking forward to Markus and his demonstration. Starting off next is a very important customer showcase.

00:46
Speaker 1
The stories that we promised to share, and it is intimidating to tell your story, especially on the stage in front of an audience. So big thank you. We’ve got with us today, we’ve got Andrew Felton, Control tech, automation engineer, systems engineer, software engineer.

01:04
Speaker 2
Hi there.

01:04
Speaker 1
Plus, and then we have Albert Kasselman, technical manager at Mediclinic. So guys, thank you so much. Give them a round of applause, please. Joining us. Cool. So there is a. As part of this project, there are a lot of elements and a lot of things to cover, and it has essentially been a one-year journey to date.

01:27
Speaker 2
I’d say about a one-year journey from start to now. Obviously still busy.

01:33
Speaker 1
Yeah, essentially from ELEV8 last year. Okay, fantastic. Let’s kick it off with. With some introductions if you don’t mind. Albert, we can maybe start with you. Just give us an idea of your role in the business.

01:45
Speaker 3
Thank you. Thank you very much. Yeah, thank you. My name is Albert Kasselman. I’m actually the Industrial Control Manager at Mini Clinic. There was a bit of a role change.

01:57
Speaker 1
So you found out over email? Sorry, you found out over email?

02:01
Speaker 3
Yeah, this was actually WhatsApp. But okay, so yeah, I’m an industrial control manager at Mediclinic. So I’m overseeing the SCADA systems and the control philosophies and systems that are in Mediclinic.

02:21
Speaker 2
Yeah, that’s me, and I’m Andrew. I’m a control systems engineer for Control Tech, Controltech-Cape. We do systems integration for SCADA platforms, PLC, and a few other things. But yeah, I do a lot of the engineering side of things. And the team’s also. Yeah, we’ve got Saville and Drian as well.

02:42
Speaker 1
Fantastic. And I believe you guys were commissioning yesterday while Saville had lunch with us. That’s.

02:48
Speaker 2
Yeah, we were playing both parties.

02:51
Speaker 1
Fantastic. And sorry, there was remiss of me. I’ve got to my right-hand side. I’m so used to Gary. I’ve worked Gary for so long I kind of don’t even introduce him anymore. So Gary is our grey wizard at Element8. Gary, Sales Engineer or whatever you want.

03:07
Speaker 4
Me to do for the day.

03:09
Speaker 1
And I’ve asked Gary to join us. He was quite close to the project, and he assisted in many ways, beyond just guiding. We didn’t do any work but guiding. So I’ve asked Gary to join us. So, Mediclinic, it’s probably valuable to understand a little bit about the business and where the project or the requirement was born from.

03:31
Speaker 3
Okay. Yes, thanks. So, like most of you will know, Mediclinic is a hospital group in South Africa. So we are actually an international group. So we’ve got a footprint in the Middle East, UAE, Switzerland, the East London group and South Africa, where we have more or less 50 hospitals, 75 facilities. That’s roughly our footprint in South Africa. So we identified the need to be able to see services and plant and equipment in hospitals for the technical managers that’s in the facilities to oversee the equipment and to know what’s going on. So we’ve deployed SCADA systems about 10 years ago. So, old legacy systems have no scalability. It was quiet, it was easy to set up in the beginning, but we made a lot of mistakes when implementing it. There was nobody really taking control over it. So it was a new thing.

04:43
Speaker 3
Each facility did whatever they needed to do to get the system up, and the architecture and those types of things weren’t really thought through. So we got to this point where we had this fragmented system. We had 50 systems running. We had one in each hospital. We had a lot of data capturing that went on manually, where people still had the clipboard and a piece of paper, writing down things. We had limited visibility, so we couldn’t have a look at the system we had. We couldn’t have an overview of what’s going on in the group. Especially, you can think of the chaos if you have load shedding and you have 150 generators across South Africa running at intervals. It’s diesel, it’s generators, and all of that equipment is life support. So it is. People’s lives depend on your power, your water and all of that in the hospital.

05:50
Speaker 3
So our data is also trapped in silos. We had a lot of systems, including BMS systems, which controlled HVAC. And then we have energy and water in an old legacy SCADA system, and we have plant data that was separate. So it was scattered all over different databases, and it was really difficult to do reporting and to get to a single view of what’s going on in facilities. And then, as we all know, regulatory sustainable pressures. So we need to know, or we need to have accurate reporting on water, electricity, and our carbon emissions. And it was very difficult to do with what we had. And then, going hand in hand with regulatory and sustainability pressures was mitigating risks. So in your plant, we need to know what’s going on in the plant. So alarming and all of that around that.

06:58
Speaker 3
And then you have a lot of reactive maintenance if you don’t know what’s going on in your plant. If you only send a person once a day to a plant room to see if something is working or not, it’s. It’s quite difficult. And yeah, so we had a lot of inefficiencies in the plant which we want to handle better then. So should I carry on with?

07:22
Speaker 1
Yeah, yeah.

07:23
Speaker 3
Sorry, so what we were looking for was the functional requirements. So we wanted the system, like I said, so that we can centrally monitor it. We can have all our critical systems on one system. We can have standardised alarming with escalation built in. We can log all our process variables into a central historian. So our technical requirements are all ours, most of our equipment is Modbus TCP, that is currently in there. And one of our philosophies is that each of our systems in the facility needs to run on its own because it’s life support and it’s critical. You can’t be dependent on a central system to control everything. If your, if your oxygen system fails because of a central system, then you’ve got problems. So we needed to keep everything separate but still have control and monitoring over everything.

08:27
Speaker 3
Then we needed to have maintenance support requirements. So I made a joke yesterday. I said to Jaco, I rolled out a report in the group in the 50 hospitals, and it took me six weeks to roll out the report. And it’s not just because I’m over 40. It’s really because of the difficulty of the system, because you needed to go into each system. It’s a remote desktop connection to a site, and then you need to do everything. There’s no scalability. And that was really one of the key things we needed: to have Scalability, easy rollout, and easy support to hospitals if they need something. Yeah, and yeah, that was ours. And yeah, like I said last year, I sat here and didn’t know anything about Ignition. And by now, I’m trained in scripting.

09:26
Speaker 1
And you’ve got a report that’s sent on at midnight now, and somebody complimented you for working those hours of the night.

09:34
Speaker 3
Yeah, when you do this hospital visit, you get to a hospital, and the emails are sent out in my name. So when you get to a hospital visit and you meet some of the management, and they’re like, Oh, you’re the guy that sends me that email every night at 12. It’s like, yeah, it’s me. And it’s like, wow, that’s quite done diligently. I’m like, yeah, you must see how much it takes from me. You wake up every night at 12 and exactly at 12 send that email.

10:03
Speaker 2
But maybe they should get you a chocolate or something.

10:08
Speaker 3
Yeah, it might work.

10:10
Speaker 1
Cool, thanks. Thanks, Albert. So maybe some of the challenges we. So essentially, we’ll go through the architecture in a minute, but we’re referring to 40/50 facilities, data collection, redundancy, and the ability to send that data securely. Security is obviously a big aspect. Receive it centrally with some kind of store-forward capability filled in the correct order if it’s late data, all of those things. But one of the core things was actually the infrastructure that exists underneath those 50 facilities. And I think it’s maybe there’s a bit of a view, I’m not sure how big it is on the screen in terms of the types of equipment and hardware that exist as part of that infrastructure. It’s vast.

11:00
Speaker 2
Look, we try to give a quick breakdown of the various devices in a little pie chart, but it was hard to fit the numbers into that pie chart. So yeah, we had about 50 hospitals and we were trying to standardise on naming conventions and various devices. UDTs were quite difficult, but I think coming up with a few templates and internal standards was the way to do it.

11:30
Speaker 1
So, UDT, just briefly, user-defined types, some people call it user-defined data types. But user defined types in terms of scalability, that’s kind of a non negotiable.

11:39
Speaker 2
Yeah, I mean, when you’re pulling from various devices using different technologies like BacNet or Modbus TCP, pulling them into the same place helps a lot with scalability, especially when you’re scaling over a lot more hospitals, like 50 of them.

11:55
Speaker 1
But yeah, so. So it mentions one of the challenges, naming conventions, that’s never been a challenge.

12:02
Speaker 3
You’ve.

12:03
Speaker 2
I mean, yeah.

12:07
Speaker 3
I think one of the dynamic events and biggest challenges was that previously, when we rolled out each hospital, they named everything and named each piece of equipment what they wanted at that time. So at some stage, we tried to roll out the naming convention. And Rowan is here. He was part of that fight to roll out a naming convention. And. And one of the jokes I always have is that I’m so tired of getting a power meter or a generator called test because normally the guy will call it test, and then when it works, he just leaves it because he doesn’t want to touch it again. And that was one of our things. And then he’s like, just go to test number four. That’s the generator you need to use.

12:53
Speaker 3
And that was really one of the things that we wanted to make sure that we’ve got a proper naming convention that is properly documented and can be rolled out and standardised.

13:06
Speaker 1
Yeah. Training and culture change. Moving teams from local to centralised thinking. Albert, I think you did that incredibly well. The human change management aspect, as part of any new technology, is a journey to take people through. So that was done incredibly well. Documents and templates spoke about those a little bit. And, and security, that naturally in. In the nature of your business, that. That was a.. There were a couple of non-negotiables. On the security side.

13:38
Speaker 3
Yeah, yeah. So on the security side, because we’re a medical facility, our data is. It is some of the most sought-after data in the. In the world. So our current system, which we had, totally didn’t comply with any security. No encryption, nothing. So this gave us the opportunity to. The cybersecurity guys were all like, when they saw what Ignition and Kinetic can do for us, they were very happy. They just said yes, go for it. And now the problem is every second day they ask me when you’re finished? Because they need to switch off all the old legacy stuff. So yeah, I think that worked brilliantly in our field, and it’s implemented quite nicely. Okay.

14:23
Speaker 2
Yeah. And we’re able to set up Entra ID so that users can log in with their Mediclinic emails. Yeah, all that other stuff.

14:30
Speaker 1
But yeah, so you’re not managing essentially two sets of.

14:33
Speaker 2
Yes. We’re not, we’re not creating users on the fly. Medically creates it once, and then we apply those security protocols that they implemented into our own system.

14:41
Speaker 1
Okay. There’s no user, one user, two users.

14:46
Speaker 2
Test users. Yeah. Cool.

14:48
Speaker 1
And the results we should probably move. Well, let’s look at the results because those are important. We probably covered a couple of them already. But this is essentially where you currently are, very close to being on the results side.

15:02
Speaker 2
Yeah, we are currently looking at these results. I mean, we’ve delivered a scalable system using a lot of our template-based and project, parent project inheritance. Inheritance through those, we’ve obviously got integration power, which means even though we are currently talking to a lot of legacy systems, we can integrate into newer systems and implement those as well. Unlimited possibilities. I mean, yeah, that’s just Ignition’s power. You get unlimited tags and a few other really cool features like unlimited clients and a few other things. Web based access, we decided on that quite early where we wanted users to be able to access it through the web.

15:44
Speaker 2
What is quite cool is that from there you can use the workstation to have a perspective app on your phone, so it becomes really usable throughout the industry, and then some open standards, and yeah, rapid development for us because we get to make one parent project and then develop it quite rapidly. You can talk to Savile about how quick his rapid development is at the moment. But yeah, we’ll let him talk about that.

16:13
Speaker 1
Okay, so this is the architecture. There are probably a couple of. No, I, maybe, MQTT is. I would imagine most of us are familiar with MQTT by now. Yep. Maybe talk us through the architecture.

16:29
Speaker 2
So there’s the way that I saw it was there’s two projects within this major project. We’ve got our site-based projects, which are primarily used for gathering data from the field devices. We’ll have a Ignition edge device sitting out there collecting data from your OPC devices using either a Modbus TCP or a bacnet and then that goes into your Edge gateway. You then have a screen at your Edge gateway that users can interact with real-time data, so that if the connection is lost, they can still monitor those crucial systems. But then that data is connected through a VPN to your central gateway, which is running an MQTT broker, and then that is stored directly to Canary because we have the MQTT module connected to that.

17:22
Speaker 2
And yeah, I mean from there we’ve got a SQL database used for basic SQL queries and for keeping of crucial data that you wouldn’t necessarily need a time series database for. And then. Yeah, okay. And then you’ve got unlimited clients at the end, also connected via a VPN.

17:42
Speaker 1
Yeah, and the very important module that’s on there is the. So we refer to it, another three-letter acronym is the EAM, the Enterprise Administration Module.

17:51
Speaker 2
Yeah, sorry, the modules are all written on the side. I forgot to mention it. But yeah, we used an EAM module to pretty much update tags, projects and a few other crucial things like make backups and also monitor our various edge devices to make sure that they’re all online.

18:10
Speaker 1
Yeah. And you can also do automated, you can do upgrades, which eliminates the need for that one-to-one kind of maintenance that you need to do with every edge.

18:19
Speaker 2
And also, licensing becomes a lot easier through that EAM.

18:23
Speaker 1
That’s right.

18:24
Speaker 2
Yeah.

18:25
Speaker 1
Doing one license is easy. Doing 50 is a little bit more cumbersome. Yeah. Okay.

18:30
Speaker 4
I think what’s important there as well is that you’re using MQTT for everything. So your data that you’re getting into your canary is the same as the data that originates at your source. So there’s no manipulation or anything that’s happening. If anybody comes to audit anything, it’s always one single source of your data, which is important.

18:50
Speaker 2
Yeah, 100%. So one single source of data, just for purely, if something goes down, then we at least have a back trace of it.

18:57
Speaker 3
Yeah.

18:59
Speaker 1
Okay, let’s move on to the hospital, essentially the hospital view.

19:05
Speaker 2
Yeah. So as we, as I said before, the hospital primarily was used as an edge node just to gather data and then also used to almost contextualise that data into a format that we wanted to see at our UCC. So how we wanted to see that data coming into our parent gateway or however you want to call it, we called it a Unified Control Center UCC. But yeah, we then did the pre-processing and filtering through there. And yeah, we designed these projects to be very scalable. So, at the edge, you could have a project where you could have various devices that aren’t necessarily the same throughout each of your hospitals, and your parent project would then react to that single source of changes. And obviously, the single project in essence.

19:59
Speaker 1
Okay.

19:59
Speaker 2
But then yeah, we’ve got a little video of what that might look like as well. Yeah. So these are just the various ways to navigate the project.

20:22
Speaker 1
Okay. Any specific kind of design philosophies, or abnormal situation management.

20:28
Speaker 2
The navigation itself is reactive to what you have in the field. So if you have a UPS or you have four UPSs, you would still see the same type of navigation, but you wouldn’t necessarily see it if you have known UPSs. But yeah, then alarming at the edge as well. Sorry to help.

20:47
Speaker 3
One of the things that was also important was when you, like Andrew said, when you have a UPS, you might have four different types of UPS. But in each hospital, the screen will look exactly the same. So it doesn’t matter if you are in Bloemfontein or in Uppington or in Cape Town, you will have the exact same view, although the background collects data from a different source. Because the important information is exactly the same.

21:15
Speaker 4
Yeah.

21:15
Speaker 3
For all of us.

21:15
Speaker 1
And there’s consistency and familiarity and look and feel and. Yes, okay. And it’s almost irrelevant what’s in the background if you’ve done it correctly.

21:23
Speaker 3
Yes.

21:24
Speaker 1
It’s not when you build it, but.

21:26
Speaker 2
Yeah, yeah, we won’t say that one, but yeah.

21:29
Speaker 1
Cool.

21:30
Speaker 3
Maybe this port is also. The calculated tax that we’ve built was one of the requirements to try to standardise. We do a lot of reporting at the corporate level. So, for instance, if you have. Take a simple example. We want to know exactly how much energy do you do we use in a kitchen for the hospital. But now in one hospital, you have three power meters because they have three different supplies and two water meters. In the next hospital, you have one water meter and one power meter. So to standardise the reporting, we. We bring in calculated variables where you can have a variable to say total power for a kitchen. And that variable will be the same in all the hospitals. So what’s in that variable differs at each hospital.

22:22
Speaker 3
But when you do reporting, you just pull up that one variable from each hospital. So it just makes it a bit.

22:29
Speaker 1
Easier to compare apples with apples.

22:31
Speaker 3
Yes, yes.

22:33
Speaker 1
Okay. So this is the UCC.

22:37
Speaker 2
So as I said before, the UCC, our Unified Control Center, was pretty much a single access point for our users to see all their data that they wanted to access. Obviously, to just pretty much also then control those edge nodes, do licensing, do updates, do backups. We also include all our reports for those various sites in our UCC. So we had a lot more scalability in terms of reports. So Bloom’s reports would be sent through the UCC as well. So that we can all control everything from a single place.

23:12
Speaker 1
Okay. And there’s no control from the UCC.

23:15
Speaker 2
There’s. There’s not much control in the system. I would say it’s more a monitoring system than anything else.

23:22
Speaker 3
You have one or two set points which can be set, but it’s. You can’t control up to a point we can start. Stop a generator. Yeah, it’s basically a set point for theatre temperature where you can set it quickly. From there, you don’t have to physically go to a theatre to change the set point if the need arises.

23:41
Speaker 1
So the one right at the bottom, which is actually very important, is the dynamic visualisation screens. And I’m not sure if you’ve got that in the view or explained that, but.

23:50
Speaker 2
Sorry, yeah, the dynamic visualisation screens, they pretty much do what they do, which is allow you to. Well, that’s more just on. If you go to a hospital within your UCC, you’re going to see that hospital’s data, that exact configuration. What you would see at your edge, you will then see at the UCC. So you’ll see that navigation, you’ll see that overview. And then yeah, you can see we’ve got a map as well that we can see all our hospitals, we can see the ones that are going offline.

24:18
Speaker 1
And the map provider that you used or the map data provider. It’s a static map.

24:25
Speaker 2
Yeah, it’s a static map provider.

24:27
Speaker 1
Okay.

24:28
Speaker 3
You will see on that there’s a little pop-up with each hospital. So if you have a hospital, if you don’t have electricity, you don’t have water, and you don’t have oxygen, then you have a big problem. So that’s the three metrics that are most important to us. In COVID, it taught us a lot about oxygen. So oxygen is very high on our priority list. So we could immediately see if any of those hospitals have an issue with water, electricity or power. You can see if they’re running from a generator, if they have backup power and oxygen as well, if you are running from your backup supply or if you are on your main feed. So that is one of the things that we wanted to see centrally on the map is to quickly see how many critical alarms do they have.

25:15
Speaker 3
If those three things are in place.

25:17
Speaker 1
Then have an immediate visual.

25:27
Speaker 2
And then yeah, we had obviously a Canary historian to store all our data. We use the on the, on that other note, in the Ignition platform, a lot of the graphing is now done through our Canary storing as well, that plugin. But yeah, we use Canary storing just because of its lightweight nature and the ability to track and store data over a long period of time. And then yeah, we. It obviously also offers a lot of easy integration with mqtt. You don’t even have to do any real processing at all, on your own configuration, or on your administrator’s configuration, besides the views, which then allows you to segregate various things into your various folders, how you want to see that data at your UCC.

26:12
Speaker 2
And then yeah, we also use Entra ID for logging in obviously, and it offers a very easy integration with those types of standards.

26:23
Speaker 1
Yeah, we did actually have a question over the tea. Break around support Cirrus Link support for. For Ignition 8.3, which was released last week, if I’m not mistaken. Version 4, which now supports Ignition 8.3. All the Cirrus Link MQTT components and modules.

26:47
Speaker 2
Yeah.

26:47
Speaker 4
What couple of interesting things I saw in the project, Andrew, was that you guys built a very nice UNS using the MQTT engine inside of Ignition. It’s really well defined.

27:02
Speaker 1
Un.

27:02
Speaker 4
Complimentary.

27:03
Speaker 2
What?

27:03
Speaker 1
What’s UNS? I.

27:05
Speaker 4
Sorry. The three that please. The unified Namespace. It’s really. It.

27:12
Speaker 4
That’s one of the things that’s going to make it really scalable is the fact that it’s going to auto build it for you and it really is going to be very helpful was one of the things that I noticed. The other thing that I noticed is that you are making good use of the views inside of Canary, which allows you to build reports really rapidly.

27:32
Speaker 2
Yeah, the views of. I mean, for those who don’t know much about the views, I’d go have a deep dive into it. But views are very important within Canary, and obviously, security around them as well. They allow various users to see various data and all that. Those things. But in terms of the unified namespace, Ignition makes it so easy to build out your namespace in a. In a very nice tag structure. Yeah. And then from there publish that almost straight into an MQTT and get that.

27:59
Speaker 1
One unified view is essentially with the engine now a tick box. Yeah, yeah. Which it wasn’t before.

28:06
Speaker 3
Yeah.

28:07
Speaker 2
All right.

28:08
Speaker 1
And. And so Axiom, obviously, the client-facing side of. Of Canary using that. Primarily for trending.

28:14
Speaker 2
Yeah, primarily for trending. A lot of it is going to be done through Ignition’s interface because we still do want 1, 1, 1 central access point for users into the project. And obviously, Canary has a very nice dashboarding and charting tool which we can use. But yeah, we chose to use both an RFRAME and also Ignitions’ built-in modules for that.

28:43
Speaker 1
Okay, What’s next? A long holiday or.

28:52
Speaker 3
Yeah, well, I said earlier I don’t think this project is ever going to end because as we start, we keep on seeing space for or we see opportunity for more and more. So I don’t think this project ever going to have an end date where we say we’ve achieved what we have envisioned, but we’ll never probably end the project.

29:24
Speaker 1
That’s good news.

29:25
Speaker 3
Me and my whole team and I are going to be very busy. Oh, it’s just me. There’s no team, but it’s fine.

29:34
Speaker 1
So the entire team is here today?

29:36
Speaker 3
No, it’s me and my entire team that’s here today.

29:38
Speaker 1
So you did. I do recall there was a. Inductive University that was the departure point for the, for the team. I think you had around eight or 12 regional people who started with Inductive University. They did training after that.

29:55
Speaker 3
Yeah, like in all. All corporations. Adding new positions to a corporate company is not that easy. Easy. So we, we decided that our whole operation in South Africa is divided into five regions. So we decided that in each region we’re going to train two super users. So it’s people that’s currently working for us who have a feel for it, who want to use the technology, and want to help us. And we started all of them with Inductive University. I think six of them is already core trained. So we have help in the, in each region. We have, we have people and it’s. I think it’s one of the functions that we didn’t actually bring up, but several builders that log a ticket.

30:45
Speaker 3
So we have a bit of control in to say if somebody at a hospital which don’t have the capability to add a new device or something, they could log a ticket. It will get to me, and I can then either give it to one of the super users in that region.

30:59
Speaker 1
And that lives inside Ignition.

31:01
Speaker 3
It lives inside Ignition and then we can have bit more control and support to the hospitals.

31:08
Speaker 1
Okay. All right, Fantastic. Any questions for Albert or Andrew? No questions. It’s got to be one or two questions. It was easy day for these guys. Okay. Maybe somebody will catch you afterwards. Guys, thank you so much. We appreciate it. And a round of applause.

31:31
Speaker 3
Thank you.

31:35
Speaker 1
All right. Do you have a. I do have a gift for you, naturally. The envelope is inside with them. Thank you so much for presenting. Lovely. Appreciate it. Thanks, Andrew.