Customer Showcase: Clover South Africa

00:12
Speaker 1
Cool.

00:12
Speaker 2
So our first customer story that we wanted to share with you today is not a new one. If you have been a part of our community for the last couple of years, you would remember seeing Clover very early. I would say probably about four years ago. It was about four years ago. So the story of Clover’s journey towards a modern industrial automation platform is quite a special one. So I’m going to ask Deon Korp to join me on stage from Clover. Deon, welcome. Thank you. How are you? You’re welcome. Please. Not. Not on that one, but maybe. Yeah, yeah, let’s do that.

00:47
Speaker 1
Hi.

00:48
Speaker 2
All right, so talk us through the architect. No. So Clovis Jr. In my life start. You have never seen that slide. Well, then we’ve got a problem. So maybe just by way of introduction, if you can share your role, who you are, what you do and yeah, that I think the context will be helpful for people to understand.

01:07
Speaker 1
My name is Deon Korb. I’m the engineering manager for all the electrical and control systems at Clover. Yeah, it’s a lot. What, what else do you want to know? Where did we come from?

01:18
Speaker 2
Kind of day to day role, your business, what do you do? Your team sort of context. Yeah.

01:22
Speaker 1
So we. My team is located centrally and we take care of all the standards and also make sure that projects get rolled out according to the standards. When the budget works, sometimes it gets strong on. But we also support the sites and we have representatives in our team at every site.

01:41
Speaker 2
Yeah. Cool. So I’m going to maybe go to this slide before we get into technical detail. This. I thought this would be a good departure. I hope it is.

01:52
Speaker 1
Yeah. So apologies for people who have seen it before at previous very many years ago.

01:58
Speaker 2
Many, many years ago. So maybe that’s the place to start us to understand why we spoke four years ago. Maybe we can start with that kick-off.

02:06
Speaker 1
Yeah, a little bit of background. When I started working for Clover, we had little factories all over. Obviously, historically, dairy is quite a unstable product. So you processed it at source. So every town had a dairy. And then when you process dairy, then you have a by-product and you got to process a by-product. And later, with advances in technology, we started let’s get more stable. But we still had this legacy, old factories. We came up with a project. Project was called Projects Insila, where we consolidated these factories into four mega factories. And with that, obviously it was a big job. There’s some economy of scale goes with that. Of course, we didn’t just do it out of the goodness of our hearts, there’s proper economic reasons.

02:50
Speaker 2
And together with that, maybe you had to understand, cool, do we have the technology that supports part of those initiatives? And I think you actually spoke through this slide four years ago, which was your automation roadmap that kind of talks through the strategic support required at the top based on those objectives and how it filters down to the technology. So we don’t necessarily have to go through the details of the slide.

03:13
Speaker 1
So obviously some of it’s redacted in that slide, but what we did is we set where we are and in an ideal environment, what do we want to do? So obviously when you’re just coming up with contextual plans and at the time we had very legacy technology, but you see the world evolving around you, walking around, you can do everything on your phone. How cool would it be to stand next to a valve and open it? Instead of shouting open the valve on the left, your left or my left, you just take your own phone and do it day to day. Technologies that can be useful in an environment, just mobile mobility in the. In the centre is one then web based application. Because it’s always an IT disaster to get approval to install this software on a device.

04:01
Speaker 2
We talk about security right now. No, we don’t have to.

04:04
Speaker 1
Security necessary evil. Yeah.

04:08
Speaker 2
So this was actually your slide that we spoke through four years ago. And why I think it was important to highlight is kind of based on these functional requirements that support the technology, which pretty much the ones highlighted in yellow. You kind of identified the challenges and then the requirements from a digital infrastructure. I mean the challenges. The reason is called common challenges, but they’re fairly common. Don’t know if you want to talk through any of those.

04:34
Speaker 1
Yes, very common. So with this big consolidation project, of course we have to do some cost forecasting for budgeting purposes. You know that 1.4 billion rand that we had to spend half of it couldn’t go into software licenses and you know, you fall off your chair when you see what you actually want to do. You want to have an HMI next to every unit machine.

04:56
Speaker 2
Yeah.

04:57
Speaker 1
You add all of that up with certain legacy vendors and it’s scary. So at the time it was a very good opportunity to have a look at. Is the incumbent technology still fit for purpose? What else is out there?

05:10
Speaker 2
And the cost is not necessarily.

05:11
Speaker 1
It was just good governance, actually.

05:12
Speaker 2
Yeah. And the cost is not necessarily just about the cost of the time, but what is the total cost of ownership.

05:17
Speaker 1
Over X period and engineering exercise? If you can speed Matters up. You have to consolidate factories, build new ones. You can’t have a resource dedicated to right click new. You have to have rapid techniques and complexity.

05:35
Speaker 2
I think kind of there’s a segue into complexity with that and I think.

05:39
Speaker 1
The ease of use and I think we discussed this a few years ago, weren’t really scared of complexity in the back end. I think I still use the example of your phone. A phone is not an easy thing. The guys at Apple and Google write the software. It’s difficult, but it’s easy for the end user. So weren’t averse to taking on the complexity if we have to do it internally. But for the application must be easy and that contextual information played in there. We want the operator to just see what’s relevant at the top.

06:13
Speaker 2
And together with that again is obviously resource requirements and very often resource constraints. The ability to support those applications and those systems internally. That’s, that’s a big.

06:24
Speaker 1
Yeah, we have a small team and luckily those components don’t keep us busy for too long.

06:30
Speaker 2
Okay, that’s good.

06:31
Speaker 1
It keeps running.

06:32
Speaker 2
That’s good. Unless your antivirus is not turned off.

06:34
Speaker 1
When you upgrade and especially when your antivirus doesn’t respect the exclusion.

06:40
Speaker 2
All right, cool. So based on your technology requirements, we’re going to go through these very quickly. We have a couple of more visual slides around some of the early conceptual visual ideas, but. So the first requirement was capability. A very simple fit on existing PLC standards and infrastructure. Again, we don’t necessarily want to orphan or isolate any of our existing infrastructure, whether it’s software or hardware. Run a Microsoft’s platform with the latest and greatest versions and it needs to be a robust architecture with a variety of client devices and client types. That’s it.

07:16
Speaker 1
So obviously with the first one, I helped develop some of Clover standards before I actually joined them. And you can’t three years down the line say, yeah, we made a huge mistake, we’re starting over.

07:27
Speaker 2
Especially if you’re the one who did it.

07:29
Speaker 1
Yes, exactly. It’s a little bit of added incentive not to look like an idiot. And so when we decided to move to a new skater platform, of course that must fit on top of what we already did. And also it was as a proof of concept, we had to run it in parallel at the same time. And well, that worked perfectly. Cool.

07:51
Speaker 2
You want to.

07:51
Speaker 1
Sorry, sorry. The rest is obviously fairly straightforward. We have. Yeah. Especially with the security. You know, the IT guys are very averse to change because they want to vet everybody. And part of our challenges is always a massive evaluation of when there was a security update. And we just didn’t want that to be a problem. It hasn’t been actually.

08:09
Speaker 2
Okay, fantastic. But you also have a very good relationship with your IT folks.

08:14
Speaker 1
Yeah. From day to day. Within reason.

08:22
Speaker 2
I’ve only heard very complimentary things. Speaking to them and speaking with you, it’s been very complimentary.

08:28
Speaker 1
Of course, we always have a focus on reliability and getting the job done and they have a focus on security. I think we have a good understanding that. I like to use the car analogy saying, yes, the car has to be safe, but we can’t lock it so well that you end up walking to work.

08:41
Speaker 2
That’s a good point.

08:42
Speaker 1
The security function cannot impede the core functionality.

08:46
Speaker 2
That’s a very good point. And I think some of the what came out in the discussion four years ago around the topic of the collaboration. I’m guessing very soon there’s not even going to be collaboration. There’s going to be one team. But the collaboration between IT and OT is legacy. It tends to inhibit and be a barrier for kind of innovation where more modern IT is see themselves as enablers. So it may be a no, but it’s not just a no, it’s a no. Why and how can we help to make it happen? And you guys still have to get the job done.

09:21
Speaker 1
You can help me to do it safely.

09:23
Speaker 2
Yeah, you guys did that very well within your business. Yeah. So some of on the points of the requirements around capability, these are examples of how you kind of met that criteria around capability. Installed and connected to your controllers in minutes. I think you actually personally downloaded Ignition I think at the time connected a.

09:43
Speaker 1
Couple of things elements I didn’t exist at the time.

09:45
Speaker 2
Didn’t exist. Correct. Yeah. Accessed easily from any client devices and redundancy that works that kind of straightforward.

09:53
Speaker 1
So on that in the original architecture that we decided we’re still running it, we haven’t made a change to it. So all of those research and deployment that we did is still applicable to this day.

10:05
Speaker 2
Okay.

10:06
Speaker 1
Which I think we did a good job and the products are good.

10:10
Speaker 2
Fantastic scalability requirements around that start small and expand. I think Agile was very much from your point of view, it was a requirement from the beginning whether it comes to tags, clients, how you deploy that. Again, tying in with that simple licensing model that does not require a precise forecast. In other words, depending on how you build out that Agile build, the licensing kind of doesn’t change too much based on that. And that kind of makes it a scale.

10:42
Speaker 1
That was actually one of the real frustrating points and which was a big catalyst to going to the new SCADA system was we had a small project to do, but it would have bumped us over a licensing tier. Now it’s very difficult for me to motivate a half a million rand bump in licensing just because we wanted to add one vessel. And those kind of forecasts is very difficult to do.

11:03
Speaker 2
It is, yeah. And then very briefly, I want to get to the sexy stuff. There we go. Criteria met, I think Unlimited tags, unlimited lines. That speaks to Ignition. I think you. I’m guessing it’s. I don’t want to say it. Otherwise I’m biased. I don’t want to say it.

11:23
Speaker 1
It’s obviously within reason and we’ve been sitting on plans to split our biggest gateway into two when the time comes.

11:30
Speaker 2
Okay.

11:31
Speaker 1
But it’s still running half a million tags, 80 prospective clients, about 100 device connections, and it works.

11:39
Speaker 2
Okay.

11:40
Speaker 1
So eventually I’ll have to split it, but I thought it’s going to happen long ago. So that unlimited, if you manage it well, is. It’s fairly big.

11:46
Speaker 2
Okay. You currently have multiple servers. I think you’ve got a redundancy as well. We can maybe share your. No, probably can’t show your architecture. You’ve got multiple servers, redundancy.

11:57
Speaker 1
We did something that goes slightly against Ignition’s official recommendations, but we have a physical server on site that’s our primary gateway and obviously that works for low latency and it’s there and it’s kind of immune to WAN outages. But our redundant doesn’t sit on site. It actually sits in our private data centre in head office.

12:14
Speaker 2
And your Canary is central host. Central enterprise.

12:18
Speaker 1
With the store and forward capabilities of Canary, we’ve never had issues. It pushes data up, it buffers. We’ve had switched it off for a week and then it sends all billion values in a few minutes up. It works.

12:33
Speaker 2
Simplicity is the last one before we look at your sketches. The user interface you very much wanted to. I don’t think you read the handbook, but you very much have had a high performance, situationally aware design in mind. If you are going to modernise, that’s usually a good time to do it. Keeping in mind that you want to make sure you don’t leave behind any of your people that are not familiar with that kind of a design, especially operators. You want to make sure that everybody’s aligned. But that was a very big.

13:04
Speaker 1
It is a jarring effect for them because I believe it’s not green, it’s not good. But yes, that’s something that they got used to.

13:13
Speaker 2
Okay. Reusable assets and templates, I think that’s a given standards and then rapid roll of an instance without re-development. So what you’ve actually done is. And why we don’t have a specific system integrator partner with us on the stage today. I think the way that Deon and the team designed and built it made it easier for multiple system integrated partners to basically receive that configuration for deployment at multiple sites, wherever that may be.

13:40
Speaker 1
Yeah, correct. So we developed the standards ourselves with our internal team and then we partnered with multiple SIs concurrently. And those SI’s who really just embraced it, they loved it. It was rapid rollout, didn’t have to waste time for instantiating and you can actually spend time fine tuning and commissioning the system.

13:59
Speaker 2
Cool. Fantastic. All right. The sexy stuff. I think it’s sexy stuff.

14:03
Speaker 1
My apologies. I didn’t know he’s going to use my screenshots.

14:08
Speaker 2
All right, so we, these are some of the early contextual designs around. You know what do we want this thing to look at? So it’s not final. In fact, we didn’t necessarily want to share the final ones, if that’s okay with everybody, because there were some. Not necessarily ip, but there were some clover. Clover specific bits in there. So Deon agreed that we can share his early conceptual mock ups, if I can call it that, which were actually not sketches, as you can see. We made it sketches, but maybe to talk through it briefly, just to give an understanding of when we’re talking about situationally aware or high performance. A couple of things we had in mind. So the first one is obviously a contextual dashboard, which I think is the bulk of that real estate.

14:53
Speaker 1
I think that’s actually where operators get the most value for when they learn that. Just make sure that the correct thing’s on the screen at the time. So if you’re running a cleaning process, you don’t really care about the totalised value, just it’s completely not applicable. And when you’re running a charging operation, then you don’t really care what the discharging units is at that specific time. And this dashboard, we’ve made it completely editable in runtime. We decided against making it user-based.

15:22
Speaker 2
Maybe just explain editable and runtime.

15:24
Speaker 1
Yeah, you don’t have to launch a designer. You can click on that little pencil there, obviously with the right credentials and put whatever you want. We’ve, we have a handful of pre developed widgets, say at this Stage it might be relevant to see the output of that control valve. Put it on there.

15:42
Speaker 2
Cool.

15:42
Speaker 1
And we can do that. We generally only do that while we’re commissioning. Because that’s when you realize what’s important.

15:47
Speaker 2
Yes.

15:48
Speaker 1
If I need to look at it right now, operator might get some value.

15:51
Speaker 2
In it later which aids a lot in saving time without having to then go back to the designer.

15:56
Speaker 1
Can do it straight from the operator station.

15:58
Speaker 2
Yeah. Cool. You’ve got a number two. Over there is a state diagram with a navigator really just to know what’s happening kind of macro actions.

16:05
Speaker 1
Know where you are, know where you can go to. So that’s not properly illustrated there but those little arrows, if they thin, they’re not applicable. If they thicker than that it is active. And if it’s really bold then it means an operator can command it to there. And then at number three, obviously you can’t really see the detail of it. But that’s where we have our current step enumerations. But we inject live plant values in there. So instead of just saying I’m waiting for this timer that actually, that timer actually times down in there waiting for that temperature can tell you what temperature you’re waiting for and what the current value is. You know, especially a timer, some product going to drain. How long is it going to be?

16:43
Speaker 1
By the time you open the PLC to see what it’s in there, then it’s finished. So now you know, you don’t have to panic. It’s in 5, 4, 3. No, it goes down and then of course at the top there it’s just the different tabs. What we do is you can see those are similar units on the same time. See what’s active, what’s not active.

17:00
Speaker 2
All right. And then this is pretty much a mimic just with pop up details, quick actions. It is number five. I think we can’t see that on the screen.

17:13
Speaker 1
Everybody always wants a mimic. Yeah, it has to be there.

17:16
Speaker 2
And again your high performance type design as opposed to, you know, the kind of more legacy type designs, it’s becoming more popular. You. This was a non negotiable for you.

17:27
Speaker 1
Yeah. And it’s just like obviously you can see some orange there but because that’s just running in a simulation and it’s to show you that it’s in simulation. But it’s just that overarching principle of don’t scream that everything is normal.

17:38
Speaker 2
Yes, yes, exactly. Kind of be aware when attention is drawn to something where it needs to be.

17:45
Speaker 1
Love you.

17:46
Speaker 2
Cool Very nice sketch that you. You draw quite well, Deon. Your lines are very straight.

17:51
Speaker 1
My numbers aren’t.

17:53
Speaker 2
So this is status table auto populating. Maybe explain.

17:58
Speaker 1
That’s always something that you know. You want to see what the status of all the ems are and who’s acquired it and what it’s in. If it’s an error. And we’ve always found that it gets neglected because the plant runs. If you didn’t configure that. But if you need to have some idea of what’s wrong, in an instant it’s not there, then you have an issue. So this is just the capability of actually browsing the tags automatically in Ignition and building all this boring stuff out for you automatically. And what I didn’t show there, if you expand one, it actually shows you what devices form part of that. Ian.

18:28
Speaker 2
Okay, this is a drill down. Drill down capability. Very nice feature. A contextual alarm table. So in other words, not just an alarm table, but a contextual alarm table.

18:39
Speaker 1
Yeah, because you don’t want to see the whole plant. You know, Queensborough is a big plant. It’s got eight factories in the same site.

18:46
Speaker 2
Was your first deployment Queensborough?

18:48
Speaker 1
No, the first deployment was at pe because we had a. We had a process running on our incumbent standard, which we can run parallel. But then before we knew it, Queensborough had surpassed it by a factor of 10. Okay, but there you don’t want to be overwhelmed with alarms that aren’t applicable. If I’m looking at the butter churn, I only want to see what’s happening here. I really don’t care about the cheese.

19:09
Speaker 2
Factory kind of what does that configuration look like behind the scenes?

19:14
Speaker 1
You just give it a starting tag path and it’ll just browse everything under that and you can give it more than one.

19:22
Speaker 2
This is a feedback table.

19:25
Speaker 1
This is something that I should probably revisit because I don’t think it gets used a lot. But we had it auto populating with issues. So if you have an enumeration, instead of permanently have a description not configured, it actually logs to a database and say, please go and configure this description. And also an operator can go and add whatever he wants it. You can go and say when I start up, the temperature control on this valve is not great. Later we might integrate the alarm notification or something. This was actually stolen from the Ignition Exchange though. Yes, with very minimal changes.

20:03
Speaker 2
So maybe my work maybe explain what the Ignition Exchange is.

20:06
Speaker 1
Yeah. So the Ignition Exchange is a repository of user contributed items and a lot of IA stuff too, which is really good inspiration to see. This is something that I need. I wouldn’t. I see it as a very good starting point. Somebody did all the heavy lifting. I won’t necessarily go and download an update three years down the line and expect it to fit, but this almost fit the ball perfectly.

20:30
Speaker 2
This is very powerful.

20:31
Speaker 1
Travis made it.

20:32
Speaker 2
Did Travis make it really? Travis, well done. Not just the evangelist, also the engineer. A couple of other useful resources on the Exchange. There is a machine learning model manager. There is an Eskom-Se-Push integrator which is quite nice. It kind of just pulls the APIs.

20:53
Speaker 1
We don’t have load shedding anymore.

20:54
Speaker 2
We don’t have load shedding. It used to be a very popular one. It was actually Deren from advances that built that one. Deren. It used to be useful. It’s not anymore. Thank you. It’s a very cool API. Kind of essentially just an API call. There is a very ticket, nice ticket management system resources available on the Exchange. So maybe check it out if you’ve never heard of the Exchange. There’s resources that’s available for free. It’s been vetted, it’s been quality checked and it does add a couple of very nice features to your application and makes you look good. Like it made look good.

21:32
Speaker 1
And the final one is just a. An audit table. And I always like having that accessible to everyone. When operators get bored in the night, they click around and they realize, oh, these guys are recording every single click. But it makes everything easier. Retrospective investigations, who did what at what.

21:51
Speaker 2
Time, Especially when there’s a bunch of compliance involved and tracking involved and maybe on the order table, if you can provide some context in terms of what is actually tracked and logged. It is pretty much every click.

22:05
Speaker 1
It is everything. And here we just filter it by what’s useful. But we log everything that has a user action and also it goes through the store and forward so you don’t even lose any data.

22:20
Speaker 2
So this is a screen that we okay to show. This is what if. Kind of an actual example of what it looks like today. Pasteurizer view.

22:29
Speaker 1
Yeah, it’s just got the widgets and something that I quite like is those little bars at the bottom.

22:35
Speaker 2
Yes.

22:36
Speaker 1
And once we train up the operators to see. So that’s just a P&ID a PID controller. So that thin little sliver in the middle is actually the error. So if you look from across the plant and you see just thin lines, you know that every single control loops in control, it doesn’t exactly tell you what the set point and what the output is. But if you see that grow to the top, you’ve got a positive error. Grow to the bottom, it’s minimal. So if you look at it from far, you see that kind of that human eye with that vernier acuity, we can spot a straight line very easily. And that just tells you everything’s cool.

23:10
Speaker 2
I like that. It’s very simple, but so functional. Maybe in terms of design, very much focused around control operations. The there is data store, central data store happening long term data store happening in the canary. What is the kind of. What is the data strategy or the not data strategy, but reporting that’s supported by a data strategy. What has that been like over the last couple of years?

23:34
Speaker 1
So I’ll be dead honest to say that’s our weak point and that is something that we have to put a lot of attention to. So product traceability is a focus area for future. But we first wanted to enable it, not paint ourselves into a corner. And now that we are more than halfway there with a plant, it’s time to jack up that capability.

23:54
Speaker 2
Okay. From a reporting point of view, we.

23:56
Speaker 1
Do use flow software, but it’s always kind of been left to the site. So we don’t have a central strategy. And that’s something that my boss is pushing us towards. If something is important for one plant to see, it’s probably important for all plants to see.

24:07
Speaker 2
So would you say it requires a certain level of kind of digital maturity or fluency even to get to that level of reporting analytics.

24:17
Speaker 1
But I think it also requires a big plant operational knowledge. And maybe that’s why we’ve been shying away from it because we have the technical knowledge, but we don’t always know exactly what’s important for the manager to see.

24:27
Speaker 2
The subject matter experts.

24:29
Speaker 1
Exactly.

24:29
Speaker 2
Okay.

24:30
Speaker 1
And it’s also difficult for a guy to tell you what he wants if he’s never seen a good example of it.

24:35
Speaker 2
Yeah, I think it’s a very cool application. Deon, any. Anything to look forward to. What’s next for you and the team?

24:43
Speaker 1
Well, I’d love to rip and replace. I know you said it’s not a good idea. That’s what I want to do. We sit with quite a large install base of obsolete equipment.

24:53
Speaker 2
Okay.

24:54
Speaker 1
Unfortunately, nobody’s swimming in money at the moment. The economy is a little bit down, especially if you’re in a consumer product. But the next step is to get rid of that old stuff.

25:03
Speaker 2
Okay.

25:04
Speaker 1
And I’D like every single HMI screen to look the same, for an operator to almost be transplantable, to know what these capabilities are here. And not being that situation where, oh, we have to make a change to a product. Is it on the new software? No problem. Is it on the old software? Oh, yes. It’s going to be difficult.

25:19
Speaker 2
Well, it is a journey. I mean, it’s been four years.

25:21
Speaker 1
Your.

25:22
Speaker 2
Your first install was pretty much four years ago. 2020 something.

25:26
Speaker 1
We commissioned the first new plant in about 2020.

25:28
Speaker 2
So maybe that’s a good learning for everybody involved. You know, we’ve got this. This isn’t a journey that happens overnight. It’s a long process. You’re getting to the point now where you’re looking at some of the kind of more data cleansing, contextualization kind of elements to make it information four years in. So it’s a long journey.

25:46
Speaker 1
It is.

25:46
Speaker 2
But I think it’s a fantastic application that we’ve got a Deon. Thank you. We out of time. We have one minute. Does anybody maybe have a question for Deon right now or any comments? Feedback? Cool.

25:58
Speaker 1
Not cool.

25:59
Speaker 2
I think it’s a. It’s an amazing journey and application. Nobody. Okay. If there are any questions. Oh, this may be a question over here. Sorry, do we have a. No. Oh, we don’t. Sorry. They’re all up here.

26:14
Speaker 1
Already placed that.

26:16
Speaker 2
You replace the ignition?

26:20
Speaker 1
We. We didn’t replace anything.

26:22
Speaker 2
Maybe the question. Sorry, did you have an existing scatter up in place and did you replace it with ignition?

26:30
Speaker 1
So we didn’t replace it. Not yet. It is on the roadmap. We just stopped investing in it. So what it was running and looking at. It’s still there, but we didn’t add anything. All of the new. So the. The plants we moved and the plants that we rebuilt fresh was all built on ignition. So we just decided we are actively divesting from it. And also we only have one site that still runs on the legacy historian. The rest we’ve all migrated to feed into our central canary.

26:58
Speaker 2
This was kind of a balance between the new site was the easy one, but the legacy ones doing it over time.

27:03
Speaker 1
Yeah, we just don’t want to add to it because then we just make life more difficult in the future.

27:07
Speaker 2
Good question. Any other questions? There was a question, I think. I thought I saw that. Cool. Deon, thank you very much. Thank you for sharing. It’s very intimidating being up here.

27:20
Speaker 1
Thank you.