Hulamin: Multi-Site Production Visibility & System Integration with Ignition & Canary

00:11
Speaker 1
Then the next application. Before we get to the demo, we’re going to spend some time with Hulamin. So I would like to invite Elvis Phala, systems engineer from Hulamin, to the stage. Elvis, is anybody else joining you today on the stage? Oh, is Leon joining you? Fantastic. Leon Potgieter. Thank you. Thanks very much. I do apologise, I wasn’t aware that you’re joining us, Leon. That’s okay. Well great. Awesome, thank you. Take us, take a seat. Elvis, you are all right. So we probably very similarly want to, before we get stuck in the screens and the technical speak, just get a sense of each of your roles within the business background of the business. That context is always very important. I don’t know who wants to kick that off.

01:06
Speaker 2
Thank you very much. So I’ll kick that off. I’m Leon, automation manager for Hulamin. Just a bit of background on Hulamin itself. It’s a very old company that was established in 1949, and as companies go.

01:21
Speaker 1
You use the second generation, right?

01:23
Speaker 2
Second generation, yeah, that’s right, yeah. And yes, as expansion goes, you know, companies buy OEM equipment, and at some stage, I think it was 1998, we decided, okay, we’ll standardise on an automation platform for the third parties, for any third party coming in and not going to mention names, but it was a good journey. And I was tasked about two, three years ago, to probably look at alternatives because, as we know, with any software, licensing is very expensive. And I was on LinkedIn, and I saw your face. My face? Yeah.

02:05
Speaker 1
Oh wow.

02:06
Speaker 2
Okay. Recognized you.

02:07
Speaker 1
Yeah, I’m sorry about that. I hope it wasn’t the first thing when you woke up.

02:13
Speaker 2
I saw, I saw this product called Ignition. It looked interesting. At that stage, there weren’t many system integrators.

02:23
Speaker 1
That’s right, yeah.

02:25
Speaker 2
So I asked Elvis and another colleague, Stian, to come to one of your breakfasts, and I said Just gonna have a look at this and see if it’s worthwhile.

02:32
Speaker 1
Okay.

02:33
Speaker 2
And they came back very excited. I did a proposal to the company, and yes, we have carried on and we’re taking out all the third-party software now and replacing it with Ignition. It’s a five year plan and as I said before, we’re sitting on 11 projects, not all done analysis by ourselves but obviously using different service, you know, different systems.

02:59
Speaker 1
The greatest, amazing. And the fact that you’ve done a significant amount of work within the team, that’s amazing.

03:07
Speaker 2
I’m very fortunate that some of my team members are here. We are very Fortunate that the guys take like change. So my department loves change. We’re not resistant to any change. And I think we’re right in the beginning when you first came and saw us. How are we going to get the plant guys involved? So we gamified it. Everybody had a task to try and build a game or do something out of it. Yeah. And that’s. That started stirring, stirring up some interest.

03:33
Speaker 1
Yeah.

03:34
Speaker 2
And then just saying that we all, we’re all trying to save money, and literally we’re saving millions.

03:40
Speaker 1
Okay.

03:41
Speaker 2
So it’s not hundreds of thousands. We’re saving millions just by going over to Ignition.

03:44
Speaker 1
Okay. And in terms of the licensing, maybe just to clarify the different licensing models, Ignition specifically is perpetual. So you buy their license and you own it outright. There is naturally an annual kind of care component just in terms of upgrade protection, but the licensing essentially is perpetual. There’s no subscription-based licensing for Ignition.

04:08
Speaker 2
And then I just want to say the support’s been amazing. It’s not a difficult software to learn. So if anybody comes from a software background, it’s pretty easy to pick up and very quickly to pick up. And then I’ll hand over to Mr Ignition.

04:23
Speaker 1
Yeah.

04:23
Speaker 2
Everybody knows you as Mr Ignition.

04:26
Speaker 3
Yeah.

04:27
Speaker 2
Office.

04:27
Speaker 1
Thank you.

04:29
Speaker 3
Thank you, Leon. My name is Elvis Phala, and I’m the systems engineer at Hulamin. And yeah, essentially, I’m the ignition custodian for our plant, and I’ll take us through the project that we’re going to present today.

04:44
Speaker 1
Fantastic. And I must say, the couple of times that we probably don’t visit you often enough, but we’ve been there a few times, and we always bring gifts. Not those kinds of gifts. Not those kinds of gifts. But every time we visit it, there’s always the next thing that Elvis and the team are working on, which is always so lovely to see when somebody is excited about building the next thing. So yes, maybe take us through what that is, what the initial project was.

05:15
Speaker 3
Yes. So like Leon mentioned, we’ve already done quite a few projects and we’ve worked with different system integrators in those projects. But today we chose to present a project on our foil annealing furnaces. And the reason we did that was because we managed to do this one in-house. So we’re very proud of ourselves. Just to show that yes, we can maintain the systems that the other guys develop for us, but we can actually work with the software ourselves and develop the projects ourselves. So this was a request from one of our manufacturing departments, which is the foil department. They were having continuous quality issues, and the existing system that they were using for their SCADA wasn’t allowing them to pick up when we’ve got fairness deviations when things go wrong.

06:13
Speaker 3
So it wasn’t very flexible, and it wasn’t easy to work with fault detection, fault attention, equipment failure, that kind of thing. So they wanted us to introduce something that’s going to help them manage their quality issues, to have better visibility, to have better trending and history, better reporting, managing the recipes and all of that. So that’s when we said, okay, Ignition, we’re already rolling it out throughout the other departments, so this is the logical next step.

06:51
Speaker 1
So at that time, you had enough confidence to take this on.

06:54
Speaker 3
Yeah, so we saw it. It’s not as big as the other projects that we did. So we were comfortable enough to say, we can do this in-house. So we started to scope it out, determine the key requirements, you know, redeveloped the SCADA in Ignition. We found that the existing SCADA didn’t have much. So basically, what we had there was just trending of the furnace temperatures. So, with the foil annealing, we’ve got about 16 furnaces, and the operator continuously has to scroll through the different pages just to see what’s happening on each furnace. So we looked at the oil system, we said, Okay, it doesn’t give enough information. It’s not a fair representation of how the plant looks. So we decided not to actually use it to develop the mosquito, but to redevelop this gear from scratch.

07:51
Speaker 1
Fantastic. I’m so happy you did that because very often the. Because very often the requirement is whatever we had to make it look like that.

08:00
Speaker 3
Exactly.

08:00
Speaker 1
And it is such an amazing opportunity to redo something and bring some innovation in. And, you know, the human change management aspect of it, the familiarity that people had with the previous system, there’s so many constructive ways of taking those people along with the journey on a new system. And if we don’t do that, it will look like that for the next 150 years. So. So we may as well use those projects to do that.

08:25
Speaker 3
And that’s true. Redeveloping gave us an opportunity to introduce situational awareness. And the nice thing with Ignition is that, because it’s so flexible, as you’re doing development, you can actually engage the end users. So you develop a page, you deploy, and immediately the end user via the browser can see what you’re doing, let you know, give you feedback. Okay. This works. This doesn’t work. So this is part of why we really enjoy working with Ignition, because it gives us that flexibility. And throughout the development cycle, we are involved at every step.

09:05
Speaker 1
And also. Also, various PLCs, PLC types, and models.

09:09
Speaker 3
Yes. So the furnaces, like I said, we had about. We have about 16 furnaces. So some of the older ones, we’ve still got the Siemens S7 200 PLCs, so we couldn’t even connect to them directly from the Ignition OPC UA. We had to use, like, CapWare to sort of interface.

09:33
Speaker 1
That’s very old.

09:34
Speaker 3
So, yeah, we still have some of the older PLCs, we have the S7 3 hundreds and we have the S7 1500s as well. So that was part of the challenge because the tech structures were the same. Addressing was different for different people between the different furnaces. But what we did was to sort of develop templates within Ignition and develop the unities, even though within the unities, we had to manage the addressing. But that allowed us to sort of standardise and hide the different devices on the user interface side so that all the furnaces look the same regardless of. Regardless of what’s happening in the background, what the actual address is. But the information flow on the screen, everything looks the same to the end user.

10:24
Speaker 1
That’s the only way to make it scalable. So UDT is short for User-Defined Type. Some people call it User-Defined data types. But UDTs, yeah, that’s. That’s a very good design philosophy. And then you mentioned just the last thing on this is the manual recipes from Excel to a secure database.

10:44
Speaker 3
Yes. So, what we had previously was that our old system used to use the recipes on a CSV file. Part of the problem with that is that users can access the CSV file directly in Excel and manipulate the recipe values. So you don’t actually have control over who’s making changes and all of those. So we moved over to a database, imported all the recipes, and now you need to be logged in with a certain level of access. Then you are able to edit and manage the recipes. And obviously, there’s an audit tray after that. It says, okay, Elvis came in on this day. Change this value on the recipe, and it’s all in the system.

11:29
Speaker 1
Okay, fantastic. And Dion, you had to sign off on this to say, yes, let’s do it confidently. All right. Some of the challenges, as we discussed with Damian and Rajiv, there are always challenges. Some of the highlights or lowlights of the biggest challenges that you had.

11:49
Speaker 3
Yeah, like I said, it’s the legacy system. So it didn’t give us much to work with. So, all the SCADA systems we had to rethink how we’re gonna visualise the fairness. Developing a new overview screen. So, so that from when the operator launches the screen, they are able to see exactly what is happening at the different furnaces. So I don’t have to scroll through 16 different pages for me to get an idea of what’s happening at each furnace. So that to me it added a lot of value at the end of the day because within one screen I’m able to see what’s happening throughout the different furnaces.

12:29
Speaker 3
And then working with the older PLCs, trying to make sure that you get the data right, you know, interface in between the Kepware and also future-proofing because obviously those 200 PLCs, older PLCs, we’re going to have to upgrade at some point, making sure that we prepare the SCADA system for those upgrades as well.

12:49
Speaker 1
Fantastic. And then the results. Improved operator response, enhanced situational awareness. I think we’ve spoken through most of this UDTs, definitely anything else that stands out as a result? Key highlight before we get into the technical stuff.

13:06
Speaker 3
Yes. Because of the new look and feel of the SCADA, the responsiveness of the operators was actually one of the biggest successes. So the guys can see when the furnace door seal, maybe it’s not closed properly, the clamps are not on, the door is not fully closed. So we’ve managed to significantly reduce those furnace deviations and improve the product quality because of that. So we’ve implemented additional checks. So the operators are required to say every four hours they must check the trends so that they can pick up if there’s any deviation within the temperature control zones and things like that. And we have these reminders now that come up to show the operators, okay, you need to do your check now. So all of these small improvements actually add up to give us a much better performance on those furnaces.

14:01
Speaker 1
Fantastic. All right. And then, briefly, the architecture is fairly simple in terms of a single-server kind of architecture. You want to talk us through that?

14:13
Speaker 3
Yes. So the diagram here is fairly simplified because I’m sure we’ve got, yeah, quite a huge installation base already. So it’s bit difficult to put everything in one drawing. Essentially, we have a single central gateway where all our projects reside, and we employ a scaled-out type of architecture. So, like I mentioned, Hulamin has different manufacturing departments. So in each department, we’ve got a tax server that we refer to as the collector. And that tech server will connect to the end devices within that department. So in the foil department, we’ve got that tech server that’s connecting to all 16 furnaces. All 16 PLCs for the furnaces. And then on the tag server, we create the tag providers where all the tags will reside.

15:12
Speaker 3
And then on the main gateway, we simply link a remote tech provider to the tech provider that’s sitting on the collector. Then we can create the projects on the main gateway using the tags on our collector that’s connected to the local PLCs. And we found that kind of architecture is more efficient because you sort of spread the computing load, and not everything is handled by your primary gateway.

15:36
Speaker 1
So using multiple gateways, you’ve created that flexible architecture.

15:39
Speaker 3
Yes, because we’ve got such a distributed plant. So in every department, we just have a tech collector that’s connecting to the PLCs in that area.

15:50
Speaker 1
Okay. All right, so a couple of screwed overviews. People always enjoy seeing the visuals. So this is the overview screen.

15:58
Speaker 3
Yes. So this is the first overview. So on the screen, very ASM, very.

16:05
Speaker 1
Some people call it 50 Shades of Gray. A couple of years ago, when we designed the screen, the question was how many colours do we have? Have 7 million. We’ll use all of them.

16:14
Speaker 3
So.

16:14
Speaker 1
So today it’s a bit more situationally aware.

16:17
Speaker 3
So today we.

16:18
Speaker 1
You remember those?

16:19
Speaker 2
Yeah, I just thought. I interject there. I think our biggest problem, which we thought we were going to have to solve, was the situational awareness. So I’m very situated, I’m very pro. And it is actually amazing.

16:36
Speaker 1
Oh, yes. Maybe just closer.

16:38
Speaker 2
Yeah, it’s actually amazing how the plant adapted to the situation awareness. So.

16:43
Speaker 1
Yeah.

16:44
Speaker 2
Yeah, cool cutting line. They’re the next project.

16:49
Speaker 3
So we try to be very intentional in our use of colour. So what we did here is everything that’s running, you’ll see it in white. And then if it’s in grey, it just means that it stopped. It’s in its natural state. Nothing is happening. And then when something is in red, it means. Means something has gone wrong. So immediately when you’re looking at that screen, you can see the rate there at FA15, you know that something is not, it’s not right in that furnace. And at this time, that furnace was isolated. So the use of colour. We’re trying to be intentional so that, you know, only the important things that need to grab the operator’s attention will grab the operator’s attention. So on this first Overview. We had to split it into two overviews because, like I said, we’ve got 16 furnaces.

17:40
Speaker 3
So to show everything on one screen, it’s a bit much because now it becomes sort of squished in together, but.

17:48
Speaker 1
It’s essentially two pages versus 16 individual pages.

17:51
Speaker 3
Exactly. So immediately on the overview page, you can see which furnaces are running, which furnaces have faults, which have stopped and things like that. And then you can see the individual components. So we’ve got two types of furnaces. The smaller furnaces, which are two-zone furnaces, and you’ve got the bigger furnaces, which are the four zones. And all of this is a template designed. So if you can look at them, the only thing we had to change is that tag, the UDT tag for that template, and then it displays the information for that specific furnace. So the development was actually quite quick and it was us actually reaping the benefits of the early work. Yeah. The decisions that we made from the beginning, we wanted to create standards that can be reusable in different projects.

18:44
Speaker 3
So now, when we get to this stage, we just pulled all those symbols and graphics that we’ve developed previously, put them together into a template, and then you can just duplicate it. So all of them they’re the same. It’s just the tags in the background that are different. And that made the whole development very easy and very quick.

19:03
Speaker 1
Fantastic. So there’s recipe and control.

19:07
Speaker 3
So on the recipe side, we also introduced some new functionalities here for the operators. We try to prevent operators from manually entering recipes as much as we can. So that. Because sometimes you can have guys, you know, mistakenly entering different values or incorrect values, so they have a selection where they will enter a recipe number that they want to use. And then we are connected to our MES system, where we fetch the recipe data and we populate it into the recipe column there. And then part of the improvement that we did with this new system is the. That’s when a new recipe is downloaded, you’ll see the colours in the background change to show that, okay, these are new values.

19:56
Speaker 1
So that’s a visual.

19:57
Speaker 3
Visual operators can visually confirm, okay, my, this is my new recipe, and it’s the one that I’m looking for. And then in this instance, we do have a manual option. So if we have connectivity issues, maybe to our MES system, and the operator is not able to get the recipe, they can come here and then manually select a recipe.

20:18
Speaker 1
If you use this for ad hoc recipe orders.

20:22
Speaker 3
No, this is.

20:23
Speaker 1
Okay. Just.

20:24
Speaker 3
Yeah, okay. And then in cases where you do have some kind of anomalies or deviations, depending on the furnace. Because our furnaces can run for quite a long time. You can have a process that’s running for anything from one day to five days. And that’s why it’s important to catch deviations early. Because if you have some deviations happening and they persist for the full five-day duration, you’re going to have a problem with your quality. So if they do see that something is going wrong, they can intervene, adjust the recipe accordingly and then carry on with the process.

20:59
Speaker 1
And, at the moment, it is user-specific access.

21:03
Speaker 3
Yes. So all of that is what you need next level authorisation to be able to do those kinds of changes.

21:11
Speaker 1
Okay.

21:11
Speaker 3
And then this is an example of one furnace where the furnace is already running. And then you can see the highlighted rows there on the recipe. It’s just showing you the stage in the cycle where it is the temperature set point that we’re targeting and the time duration that we’re supposed to be in that stage. And then we played around with some of the animations there. When the fans are running, you actually see them turning on the screen to show that the fans are running, your heaters are on, your door is closed, your clamps are on and all of that. So it just makes it more visual.

21:48
Speaker 1
And you did this with the old drawing tools?

21:51
Speaker 3
Yes, yes. This was before 8.3, unfortunately. This was before 8.3.

21:57
Speaker 1
Right.  A couple of trends, and I suppose more on the report trend and trends and reporting side.

22:02
Speaker 3
Yes. So this is crucial for the operators. It’s maybe not visible there, but at the top there next to the date selection, you actually have a little icon that says check trends. So the operators are required to on a schedule. Like I said, normally we make it every four hours to go and check the trends, make sure that all the data is still within the required spec and then they will just click that check trend to say okay, confirm that they’ve done the check. And then if four hours go by without them actually checking the trend, a pop-up will come up to remind them that, okay, guys, you need to check the trends for this screen.

22:44
Speaker 3
And then if they still don’t do it for 30 30-minute grace period, then an email will get generated and will be sent to their supervisor to say Can you ask the guys to check the trends?

22:54
Speaker 1
And, did you build that with the workflow, the alarm workflow or just outside of that?

22:59
Speaker 3
No, we build it outside of. Okay, it’s just one of our templates.

23:04
Speaker 1
Okay. So you probably would have seen the new. There’s a new capability for WhatsApp and Telegram, depending on how far you want to go outside of email alerts. And the great thing about WhatsApp as a potential means for an alarm is that you can have collaboration between the teams. You know, an email or a WhatsApp is a single alarm to a single person where a group of people can then discuss that alarm and the potential. So that might be a nice add-on to some further trends. I’m going to go to the alarms and looks.

23:40
Speaker 3
Yes. So this again was, you know, we struggled a lot with the alarms in the previous system. So now we could go into more detail about the different alarms and what they mean and how the operators interact with them. And then you’ll see on the next slide that we’re also pushing the alarms into the database for long-term storage, so that if we need to go back and evaluate what was happening at every stage, we’ve got all of that information.

24:10
Speaker 1
You didn’t create a reset-all button somewhere.

24:14
Speaker 3
Yeah, unfortunately, you still have to have it.

24:20
Speaker 1
The alarm rationalisation is. There’s actually some new functionality available that’ll be worth looking at. But it is critical here. And then of course, naturally, you’ve got the audit logs, which go down to the per-user level as well.

24:32
Speaker 3
Yes. So I mean from the maintenance side, the audit logs and the next slide with the diagnostics, that’s crucial because they are able to see at every stage what the operators have been operating the plant. And then if the operators are having issues, it makes it a lot easier to just fall find. And then the diagnostics. We actually got this from the Exchange. It’s one of the nice tools that you can plug in. And then I’ve adapted it so that it works for our situation. And then you can see all the PLCs that are connected, they’re still healthy, the databases, all of that is still here.

25:11
Speaker 1
It’s a lovely resource and it’s free.

25:13
Speaker 3
Yeah.

25:14
Speaker 1
And I think the other resource of the Ignition Exchange is a repository of different resources that the community have created. It has been vetted by Inductive Automation, but it is available for free. And I think you were also one of the first to use Eskom se Push integration one at some point, I think, in an earlier project. Earlier project, yeah. Okay. But this is a very nice resource, and you’ve made good use of it as well.

25:39
Speaker 3
Yeah.

25:40
Speaker 1
Okay.

25:43
Speaker 3
Yeah. So with this project, we were able to standardise on the SCADA side, and then the next work for us is to do the same on the PLC side, upgrade our old controllers, and then standardise in terms of the PLCs and the tech structures. Yeah. Make it a lot more easier.

26:01
Speaker 1
Okay. And from your perspective, looking ahead, you. You’re just okay to go to. To have the confidence to say yes to the next one.

26:11
Speaker 2
Stepping back, I think from my side, it’s just we’ve got to train more people in the plant. The more people who know about it, the more people want to work with it.

26:19
Speaker 1
Yeah.

26:19
Speaker 2
Makes our job easier. So.

26:22
Speaker 1
Yeah, I. I like how you did the gamification with Inductive University and the projects as well. So, to give you an idea, when you sign up for Inductive University, everybody else with the same domain as you can kind of see each other’s progress. And that’s always fun because then it becomes competitive to see who. Who finishes first? I’m on. I’m on 3%. So if anybody is looking to start soon, let me know, and we can see who does it by the end of the year.

26:45
Speaker 3
You’re still better than Ewan.

26:49
Speaker 1
So, Leon, you’re further than me. I’m on. Yeah, I’m. I’m on 3%.

26:53
Speaker 3
That changed.

26:53
Speaker 2
There were a whole lot of guys in 100. I think you’ve added some more modules. So they dropped down to 80. Yes.

26:57
Speaker 1
Yeah. Yeah. With the new modules that have been added, all of a sudden, you’re on 80. Yeah. So, yeah, but well done, I think. I love how you’ve approached the adoption of a technology in the correct manner around the Inductive University as the departure point, building the confidence out from there with smaller projects. Before attempting a project, as if you had just spoken to. So well done to you guys. Yeah, I think it’s very brave. A lot of companies will look at it and say, Yes, we need to try and save costs, but to replace a control system. A lot of them will look at it and say there’s the risk of doing that, and the potential losses behind that are far greater than what we can see. And in fact, it’s the other way around with you.

27:38
Speaker 3
You, you.

27:39
Speaker 1
You started out as a project that was looking to save costs on licensing, but from what I see here, you’ve seen a lot more benefit than just the licensing.

27:48
Speaker 2
Absolutely. Absolutely. I can’t stress the. The ease of working with. With this product. And you don’t have to buy an extra module; everything is just there.

28:00
Speaker 1
You must just find that you’re being very complimentary to the technology, but essentially, it’s your effort. So well done. But thank you for that, and thank you for sharing.

28:09
Speaker 2
Thank you.

28:09
Speaker 1
I do appreciate it. Any. Any questions over there are immediately answered. Yeah.

28:14
Speaker 3
How did you guys manage project management? People change management, was that a particular focus?

28:21
Speaker 1
So, people management and change management are part of the project.

28:25
Speaker 2
Yeah. So as I said before we. When we actually contract our department to another department, and we work off a project schedule, so we have a. We have weekly meetings with the guys, and we have the updates. Okay. It’s very structured.

28:37
Speaker 1
Okay. And the human change management that would involve your typical. Maybe some manuals training, user training. Yeah.

28:44
Speaker 2
And. And on the. On people’s. We don’t just. We actually involve the area managers at a high level from the start. So we get there by. And so it’s a top down approach, not a bottom up approach.

28:59
Speaker 3
Just to add on what you are saying is that we get the guys involved, they take a lot more ownership. So at. For the foiler example, at the end of the project, all I did was compile a long document saying Okay, this is what’s in the system, this is how it works. And then the supervisor at the Foil fantasies took that document and created a new SOP for the operators. And then that’s how the knowledge transfer and how you get the guys to actually become comfortable using the system is getting them to take ownership and then just making sure that whenever they need something, we are there to support them.

29:39
Speaker 1
Opposed to one day going tada.

29:41
Speaker 3
Yeah, and just saying, in case you miss gator and walking away, that’s it’s not going to work.

29:45
Speaker 1
Okay, thank you. Yep.

29:48
Speaker 3
You mentioned it’s a five-year journey.

29:50
Speaker 2
What’s the.

29:51
Speaker 3
What’s the end goal once you migrate scales, range miles and SOP?

29:57
Speaker 1
Big play long motorcycle.

30:05
Speaker 2
So the five-year goal was that I would take the licenses over five years, and I had to sell them to our directors at that point to say Well, this is what we’re going to save to change what we have existing. Luckily, we are in an ever-changing company, so we are always buying equipment or upgrading, so there’s always going to be the maintenance aspect, and I think probably in about five years’ time, we will go into maintenance mode more than a project mode. And one of our standards now is that if we buy new equipment, Ignition isn’t on the top of the list as a requirement.

30:48
Speaker 1
I think there was one more question if I was it.

30:52
Speaker 2
Okay. All good.

30:54
Speaker 1
Leon, Elvis, thank you so much for sharing it. We appreciate it. Appreciate the time.