Protecting Business Alpha and Securing Enterprise Agents

Julia Nimchinski:

Thank you. Thank you again, and with that, we’re transitioning to our next CXO panel. Welcome to the show, Daniel Davis. We’re going to be talking about protecting Business Alpha and securing enterprise agents. Super excited. Daniel, how are you doing? And yeah, take it away.

Daniel Davis:

Oh, doing great! What an interesting topic that we have here. And how prescient is this, considering some of the things that we, happened yesterday? So, I guess let me preface it by saying you’re a business leader, and the AI discourse is around, you must adopt AI. AI is the future. This is integral to your business. And on the other hand, you’re being told AI is dangerous. And being told the same thing by the same people making the AI sometimes. And you can’t secure it. It’s dangerous. You’re gonna leak your data, you’re gonna lose your secret sauce to how you do your business.

Well, how do you make a decision? How do you approach that situation? How do you think about, on the one hand, are businesses at an existential risk to not adopt AI, But at the same time, Are we at risk to adopt it? And that’s what we’re gonna talk about. We’re gonna talk about that conflict that arises between the needs of the business and the reality of trying to secure that technology. And joining me today to talk about this is Heather Wood and Jamie Brown. Heather, Jamie, welcome. Introduce yourselves.

Heather Wood:

I guess I’ll go first. My name is Heather Wood, and I am the head of global data privacy and protection at Outreach. Ultimately, that means I oversee privacy, AI governance, and data management at a smaller level, and support our business in making decisions around how we utilize AI, both from an Agentic perspective, but also from And a machine learning as well as LLM generative AI perspective.

Daniel Davis:

Great! Great to have you, Heather. Jamie?

Jamie Brown:

Hey, I’m Jamie Brown. I’m the CISO at Vivin. At Vivin, we’re a small startup. We deliver Hero, which is an AI sales teammate that helps sellers. Before, during, and after all the calls? As CISO, I’m in charge of all security activities, as well as governance, risk, and compliance for all of the company, as well as our product and services.

Daniel Davis:

Great to have you today as well, Jamie. And I have to say, when I first met you. I was, I wasn’t sure I really had met you, considering the picture I had seen, versus, that is quite the beard you have there. I am very impressed. I’m sure most of the… I’m sure there’s going to be many questions about just how long it even took to grow such a beard, but I’ll try to stay away from beard questions. We’re here about AI. And I’m Daniel Davis, I’m the co-creator of TrustGraph, which is an open source context orchestration platform that can do much, much more, and is totally free and open source, and is out there on GitHub.

You can find it. But let’s start first. with, I think, some news that happened just yesterday that’s very relevant to this topic. We saw a researcher from Anthropic, I believe his name is Jacob Coxon. publicly resign on Twitter. Oh, I’m assuming he didn’t resign on Twitter. I’m hoping he went through more formal channels, but you never know in Silicon Valley. I’ve seen some strange things. Over safety concerns, which… It’s a bit odd, considering that… Anthropic has been, at times, the most safety-conscious, and at other times, they’ve rang the alarm bells. And he says he resigned because he feels… I… I don’t remember the exact wording, but he felt that there was A bit of recklessness in how the Frontier Labs are approaching AI, and the dangers they present.

And if you’re a business leader, and again, you feel all this pressure to adopt AI, and we’re going to get into what adopting AI even means for an enterprise. What do you… what do you make of news like this? How… what did… what did you think when you saw this news yesterday? And, I’ll start… how about you, Jamie? You’re on the left of my screen, so I’ll go to you first.

Jamie Brown:

No worries. Yeah, I’m not too familiar with this exact story. I definitely heard the headlines. Obviously, there’s always two sides to a story, so I don’t want to comment directly on the story, but essentially, I think that the AI landscape that we’re in is, you know, is it an evolution or a revolution, right? You know, I’ve been in the computer science field for a long time, 20-plus years in cybersecurity. And it’s just… for us, it’s another one of those, right? Going from, you know, hybrid cloud to private cloud, you know, now we’re doing AI, From this technology space perspective, it’s just… we have to apply these security principles to protect the data and reduce risk.

As well as allow the business to succeed, right? From a security sense, we can’t just be a security department of no, we have to enable the business. So it’s always this balance of enforcement and enablement to have success. when it comes to AI, you know, time’s gonna tell, right? We’re gonna know, you know, in hindsight, 20-20, whether, you know, AI has, you know, ruined humanity or made it better. For me, I’m more of an optimist. And I see AI as, you know, not just a business driver, but just as a human driver, as the ability for humanity to grow.

Who knows, right? Some of these AI models, you know, might solve, you know, some of the biggest diseases that we have, or those type of things, right? It’s not just about enabling business. It’s about how can humanity itself increase and be better with this technology.

Heather Wood:

I think on the flip side, so keeping in mind, I… I love AI, but there are certain limitations to it. One of my favorite movie quotes is from Jurassic Park, when Jeff Goldblum says that your scientists spent so much time thinking about what they could do, rather than thinking about what they should do. And I think it’s interesting when you get into a position where you’ve got different folks that are apart of the development of these, AI technologies. that have different views on what’s ethical and what’s not. And this goes down to the human condition.

If you, I was at a conference a few years ago, and the question was asked whether or not what we’re doing with AI is ethical, and the response was, well, it’s legal. So it’s ethical, right? That’s not necessarily true. We know that the laws take time to catch up with technology, and what we’re seeing is we have a lot of technology that’s quite novel in the last couple years that we don’t truly have, you know, a a legal guideline for, and you have various folks then responding with, I’m not comfortable with what we’re building. I’m not comfortable with what this does, and so not everybody’s going to come at it from that same view.

This individual clearly felt that they needed to step away, because what they felt that they were building, they didn’t think was safe or acceptable, but not everybody’s going to have that view. And I think this is one of the challenges that we run into, especially from a compliance perspective, around what we do with information, and whether or not we think it is truly ethical, or if it’s just permissible. And that’s two very different perspectives to come at, you know, an issue with.

Daniel Davis:

you know, so many Jeff Goldblum quotes you could have gone with, and I kind of thought you were gonna go with life finds a way, or maybe the one where Wayne Knight’s character goes like, see, see, see, see, nobody cares? Actually, maybe that’s what I should go with on this one, is that, So I have a bit of a safety engineering background as well in the aerospace world, in addition to cybersecurity. So I’m, you know, very well steeped in that conflict. And kind of exactly what you just talked about, in that well, we can, you know, but we should, and should we?

And how much conflict in an enterprise there is between, whether it’s safety or the cybersecurity organizations, and saying that, yes, we can do this with the product, yes, we can do this with our services, but… you know, should we? Is it a good idea? And, you know, oftentimes it comes down. The business leaders, it comes down to the cold, hard, well, is it legal? does… is this going to affect our compliance posture? And, you know, there’s this big struggle between, well, how do we understand those two? And, You know, I… you know, perhaps that’s a good way of looking at this, is, you know, the risk aspect of that is that I feel this is where the business leaders are likely struggling.

They’re looking at the risk of, what if I don’t? adopt this technology, and I still do want to talk about what it means to adopt this technology, whereas we have experience with What are the risks of… to consumers, customers, enterprises, the organization itself from an information perspective, and at this point, it seems that our, you know, our entire lives are about information and access to it. So. how do you think, or how would you recommend to business leaders, or how are you even doing this internally? I mean, I’m sure you’re interfacing with people in your organizations that are looking at the bottom line, looking at, you know, this is what we can do, and that’s their… that’s their job.

You know, if you’re the CEO or you’re, you know, a P&L owner of a publicly traded company, you have a fiduciary duty to the shareholders to drive value. So, you have that… that is your job. That is what you’re there to do. And… How do you… how are you… how are you recommending to those people in those positions to… weigh these risks, or even… there’s no framework for how to do this. Nobody’s created an AI risk framework. I mean, some people have tried. I know there’s a NIST sort of one, but there’s no widely adopted one that we all look at, so… How do we approach that aspect of the problem?

Heather Wood:

So, I know from our side, and the way in which I, you know, look at this from a strategic perspective, there’s two different approaches that we have to take. So, there’s the approach of how we build the product itself, and then there’s also the approach of how we use AI internally from the perspective of a data controller. So when we look at how we build the product, my team is very fortunate to have a really great relationship with both our chief product officer as well as our Chief Technology Officer. They have been incredible leaders and lean into the compliance by design process.

And when I say compliance by design, what I mean by that is that it’s not just privacy in a silo. speaking with, you know, an engineer, for instance, that’s building something, we come at this from a collaborative approach where we include security, we’re looking at it from a AI best practice and our documented strategy around AI, and how we want to build products that we’re going to release to The market that are built with those compliance pieces in mind, so that when our customer goes to use it. they don’t, you know, they’re not sitting there going, why, you know, why wasn’t compliance considered for this feature or that feature?

And so, all of that is really built around our roadmap strategy, product strategy, and our AI strategy internally being built into that compliance by design process. I have seen this done in other companies. I’ve got friends that work in other places where they do operate in somewhat of a silo. We find it to be much more effective to be operating as a collaborative team. across these different functions, so that, number one, we’re able to, get some synergies around the recommendations that we’re making from a compliance perspective, but also ensuring that our leaders are aware of why we’re making those recommendations.

It’s not just a, you know, the sky is falling, we have to protect everything. There is business sense built into it along with that compliance factor. Internally, we also have an ecosystem AI strategy, and as you’re utilizing AI, you have to have these strategies in mind. Otherwise, it’s just the wild, wild west. Data leakage happens, you’ve got prompt injection, you know, risks that you bring into the fold without even necessarily thinking about it. And there are all these risks that you really do have to be mindful of as you’re setting those strategies, which you then operate against.

The best part about strategies is that if something needs to change. you re-evaluate why, assess the risk that goes with that, and then make those changes. The beauty of decisions is that if you don’t like the decision you’ve made, you can go make a new one. And so that’s the way in which we do it internally. Jamie, do you… is that kind of in line with what you’re doing?

Jamie Brown:

Yeah, certainly, Heather. I definitely appreciate that two-factor of that. From our side, it’s not just the application and service, it’s also how does the average employee use this technology? How are they using it to, you know, improve their day-to-day? But it also comes down to risk, right? So, internally, we look at risk from an employee perspective, understanding what data’s being used for these services, and then when it comes to our product and building that service out as well, it’s understanding how the data’s being used, and, you know, again, like, Daniel, the topic here is Agentic AI, not just AI, so it’s not just how do you chat. with these… with this data, and use this data as part of a chat, but it’s how do these agents use this as part of their workflows?

So you have to really think about not just the first, but also the second and third order consequences of the data and the data architecture, and the workflows associated with that. And as Heather mentioned, it all comes down to risk, it comes down to compliance, it comes down to the business decisions. And working with those decisions as you achieve those objectives.

Heather Wood:

And I think that’s actually… that… that’s a really good point, because when you think about AI, most… I mean, AI’s been around for, what, 30-plus years at this point, but most folks, when they think of AI now, they’re thinking of generative AI, which has a risk of data leakage, as we’ve seen with some companies that accidentally put their proprietary code into ChatGPT when it first came, you know, became public. But then you also have the Agentic AI risk. Most people don’t even think about the foundational models, right, or the traditional machine learning models. Those ones typically didn’t have this risk of data leakage.

With generative AI models, you’ve got a risk of data leakage. With Agentic AI, what you’ve got is agents that are operating based off a set of data, which may or may not be complete and accurate, and then they’re going out and they’re acting on your behalf. If those agents are acting without necessarily having a human in the loop, you have to be very, very thoughtful about the way in which you write those prompts to protect that data, both from being leaked, but also from having an agent that perhaps makes commitments that it, you know, the company’s not prepared to abide by.

We saw a very large airline run into that from a, somewhat Agentic, somewhat generative, but ultimately they had a bot on their website that made commitments that they were not prepared to execute on. And then there was a court case that then required them to do so. And so these are all the things you have to be super mindful of, and I think that That is something that, especially for compliance folks, has a tendency to make them a little nervous. You’ve got to really understand the technology.

I would say that’s probably the biggest thing, is making sure that if you’re supporting making business decisions, you have to understand the technology you’re working with, and then how it is that you combat those risks. anyone, Daniel, to your point, anyone who says, we’re just not going to use it will be left behind. And we see this in evolutions of technology over time, where a company’s just like, you know what, no, we’re not going to use it, we’ll continue without it, and they get left behind because there are good, strong efficiencies to be had with those particular novel technologies.

Instead of fearing them, you have to learn to understand them so that you can operationalize them while minimizing risk at the same time.

Daniel Davis:

And both of you bring up two points that I want to talk about, and I’m debating which one to go with now. You know, Jamie, you bring up that, well, actually, 3 points. Geez, there’s so many I could go with. You make a great point that the terminology, AI, we use that often, but then there’s Agentic AI and agents, and what does that difference mean realistically, for an enterprise, and where are people in that journey, and how they consume this technology now.

The age of AI, wow, I mean, this technology goes back you know, depending on how you look at it, 70 or 80 years or even longer, even the semantic web, if you actually talked to the people that created the semantic web standards in the late 1990s. they did it with AI actually in mind, because they weren’t really calling it AI at the time, they were focusing more on natural language processing, and they thought NLP was going to, in the early 2000s, really be… kind of get there, so to speak, you know, be… Commercially viable, and it… instead, it took about 20 years more to get there, like, where we are now.

But I think the point I want to talk about is something you mentioned, Heather. About the human-in-the-loop problem, and especially because… I think, some people are not aware that there are regulations around this. There was just recently, I believe it was a fine that Uber, a very, very, very large fine that Uber received in Europe for violation of some of the. aspects of their technology in Europe. I know that California has some legislation that they are trying to push through on automated decision-making technologies. I’ve read some earlier drafts of that bill, and I found it very hard to interpret, and I don’t know what the scope of it would be.

How do business leaders… are they thinking about The… using agents that are no longer… potentially have a human in the loop that could be making various decisions about… internal processes, decisions with customers, And what is a decision, even in that flow? How do we think about an agent? Does an agent make a decision like a human does? How are we going to approach this going forward? And will we ever get to a time when there actually isn’t a human in the loop at all?

Heather Wood:

Honestly, I cannot see us getting to a point where there won’t be a human in the loop, at least to some degree. There are a lot of things that you can have an agent do, and it really depends on what your risk acceptance appetite is, it depends on what your use case is, it depends on how good you are and what your governance is around writing commands for agents. There… and it also depends on whether or not you plan to have those agents thoroughly operate autonomously, or if you have the plan to effectively have them operating as a co-pilot or a co-worker.

It really comes down to what is your strategy for your business case, because your strategy won’t be the same across your vendors. I would say that if your strategy is the same across your vendors, you’re probably going to run into some hiccups. And so you really have to be thinking about what is your use case? Is it viable to have an agent do this? Are you trying to build agents on top of agents? Because we all know that has a tendency to result in some hallucination issues. Have you written your commands in such a way that your AI isn’t going to start reaching for answers and doing hallucinations just on that first command?

There’s a lot of… you know, really understanding what it is that your intended output is, and how good your hygiene is for building your agents. It’s part of the reason that we actually look at… we could build our own agents. We have a tendency to focus on technologies that have built them for us, because they’re thinking about that as they’re building it for their niche market. And so these are all things that should be considered by the business, both from a risk perspective, but also the investment opportunity. And, you know, whether or not you can pull a human out probably depends more on the task. than the overarching activities that that AI is meant to perform.

Jamie Brown:

Yeah, it’s interesting when you start to think about this from a cybersecurity perspective, right? You look at programs, you look at scripts, you look at how things have worked before in terms of workflows, and in a way, we’re just moving higher up the stack, we’re moving workflows, you know, further along. than before, or we’re daisy-chaining these workflows, you know, more and more, such that it’s not just prompts now, it’s actually prompts and actions. You know, in programming, it’s the if-then, right? And it’s working through that process. And, again, from a cybersecurity perspective, when it comes to managing that risk and managing that autonomy, it’s not a one-size-fits-all.

It’s a balanced approach. It’s, hey, you know, Heather, you mentioned Copilot, like. Today, when you fly, you know, from a flight perspective, you have a pilot and you have autopilot, right? what’s tomorrow look like? Does tomorrow look like, you know, the planes being driven by a, you know, a drone, somewhere else, right? So, it’s just understanding how this technology is going to evolve, and looking at it from, you know, from our… our seat in terms of SaaS or businesses. Again, it’s… From a security perspective, from a risk perspective, it’s… it’s not one size fits all.

It’s… you have to balance it, from that side.

Heather Wood:

And actually, Jimmy, off the… off the back of your example of the co-pilot, I… that’s another one where, can you imagine you’ve got an AI that’s been trained to fly? Great. Something happens, and the co-pilot doesn’t know what to do, because it only knows what it’s been trained on. humans will always need to be a part of these processes, because there is a certain level of creativity that humans have that you can’t necessarily, at least not currently, train into in AI. They only know what they’ve been trained on, and they don’t have that ability to really get creative And think outside the box, because that’s not how they’re built.

And so you’re always going to have to have a human. Now, do you necessarily have to have as many? No, but that’s the beauty of the co-pilot, is you could probably increase productivity. Let’s say you’ve got a group of people that are focused on doing something, for instance, producing revenue. you give them a co-pilot, they’re able to take their menial tasks, their admin tasks, I’m thinking about sales teams and having… doing things like updating Salesforce. with the outputs of a meeting. Well, if you’ve got a co-pilot that’s able to do that for you, you’re able to focus on the things that your human brain has needed, like relationship development.

And so, that’s where the co-pilot is really, from my perspective, is the optimal use case, because you’re only having it do the things that it can be trained on, and that are easy for it to execute on, but doesn’t necessarily require that creative component that you do need a human brain for.

Daniel Davis:

I’m so glad that you reminded me of those examples with aerospace and pilots, because you know, again, I spent a very, very long time in safety-critical aerospace, and we’ve actually had commercial aircraft that could fly ground-to-ground fully autonomously, since the 1970s. Matter of fact, I’m the one person on a flight that would be, hey, pilot and co-pilot, come take a seat back here and let the autopilot fly the plane, because I know the autopilot can do a much better job than you can. However, I think every other passenger on the aircraft would be horrified at that.

Because of my experience in aerospace, I ended up in autonomous vehicles during that first wave, and that’s also a similar problem in that here you have autonomous systems, which you could say are being controlled by AI. That’s a human function driving a car, so if you have a system that is controlling a car, you could call it AI. You could even call it Agentic, because there is no human in the loop in that system. And Waymo has quite a bit of data, you know, here in San Francisco, that shows that they have fewer accidents and are… I guess, on average, safer than human drivers, yet… the large percentage of the population is very vocally still against Waymo and autonomous vehicles, and will go out of their way to find examples of their mistakes, and even sometimes interrupt cars.

Just generally, not really… Ignoring the data and succumbing to their very human emotional response to the technology. Which brings me to ask… how are we going to overcome that in… with AI and agents in enterprise, where people have fears that this is going to take my job, this is going to cause nuclear war, this is going to cause the end of humanity? This is a problem that we’ve seen, you know, many times in cybersecurity, is that there’s the saber-rattling, there’s the fear-mongering, and then there’s the other side of it, you know, how do I figure out how to invest in this?

So, how do we reconcile where there is We know there’s so many instances in the past where autonomous systems have outperformed humans, and humans are… Still, even when confronted with the data, very uncomfortable. with that data.

Heather Wood:

I think a lot of this goes back to fear of the unknown. That is something… it’s a… it’s a survival instinct, right, from our caveman brain. And so, if you don’t understand something, you have a tendency to fear it. And I think what we’re going to see is, as technology continues to evolve. as we continue to move towards autonomy for these AI technologies, there’s going to have to be A set of people that are gonna have to learn about it, understand it, because how do you overcome fear? you break it down into pieces so that you can digest it, and then understand it.

What comes out of that? Better understanding of the technology. A lot of that, though, is going to be self-propelled or forced on you by your employer, depending on which type of field you’re in. And, so… I think that what we’re going to see is that shift, but very similar to other things in our history, it’s going to take time, and changing hearts and minds, and getting folks to invest. in learning about these technologies to overcome that fear. That’s a… I realize that’s a very sociological, psychological perspective, but that is typically how you get to that, where people go, oh, okay, I think a lot of it will also be dependent on how large companies utilize these technologies.

In my role, I spend a lot of time talking to customers about AI and what it does for them. And there’s two very different mindsets. There is the mindset of, I need to cut costs, so I’m going to replace a team that was doing X with an AI that does that instead, and that’s one, you know. focus. And then there’s the other one, which is we’re going to utilize this AI to beef up the skill sets and the, you know, time that you’re able to save on menial tasks. using the co-pilot method. Those are two… and I’m not saying one is right or wrong, I’m just saying those are two very different ways of viewing AI and how it can support your business.

What we have seen is lots of companies do layoffs, where they are looking to utilize AI to replace humans, and it doesn’t work out so well. And then you see other companies that look at it from a co-pilot perspective. I think IKEA is one of the biggest ones that did this, and the amount of profit that they were able to generate by utilizing these AI agents to do these menial tasks and allow humans to focus on human things. generated billions in extra revenue.

And when you look at that, you know, there’s… there’s a different way of approaching this, and it really does depend on what’s the direction of your board, what’s the direction of your C-suite. what are you trying to accomplish as to which one of these that you are, which one of these strategies, if you will, that you’re looking to implement. But I think that’s also going to play a big part into Folks’ acceptance of this new technology, where it’s not necessarily replacing them, but giving them a boost to be able to be better producers.

Daniel Davis:

Yeah, go ahead, Jamie.

Jamie Brown:

Yeah. No, it totally makes sense, Heather. You know, again, it comes back to the technology and the adoption of technology. You know, there’s the adoption of technology curve, right? So you have, you know, whether you’re bleeding edge to cutting edge, or you’re in the big middle, or you’re at the end in terms of laggards, right? It’s the same type of thing when it comes to AI and Agentic AI. You know, from my seat, we’ve seen We’ve seen tremendous progress in the last 3 years, let alone the last 6 months, let alone the last month, right?

So, I think time is going to be an interesting part of the equation here, where the technology’s shifting very fast, and understanding how long it takes either a company to adopt things, or the individuals to adopt things is going to be interesting. I’m excited about the next 5 years. I haven’t been more excited about, you know, work and understanding this technology space in a long time. And it’s… it’s just… it’s just amazing on the capability that this technology is bringing, the efficiencies that we have.

You know, it’s interesting because some of our… some of our developers they obviously use clogged code, or cursor, or different technologies, and they’re able to do some amazing things, and, you know, I was telling someone about it, and it’s like, they’re not able to do it just because the technology’s enabling them, it’s because they have 15 years’ experience, they know how to shape the technology to get the The actual objective that they’re looking for, and they’re able to do it very, very fast. We had one person that was able to, you know, deliver not just an agent, not just skills, but also a whole workflow associated with that when it comes to SRE.

And he was able to do it in a matter of days and weeks, where before it would have taken a team of teams to deliver this technology. So, that’s the exciting part of the unknown, if you will.

Daniel Davis:

Julia has just mentioned, we have some really interesting questions, and I actually have a question for our audience. How many of you are lawyers? Because I’m getting some very lawyerly questions, but you know. it’s… they’re all very fair. I actually gave a talk, many years ago where I called it the, Better Call Saul, I think, was either still on the air or had just called it… gone off the air.

I called it the Saulification of cybersecurity, and the theme of my talk was how cybersecurity had become a legal function, and it’s interesting that, how many companies, the CISO or whoever is the head of the security or IT, sometimes even answers up through general counsel or legal, and they don’t answer to… I don’t know who you report to, Jamie, but I’ve seen that increasingly happen, where security isn’t in engineering, it’s not in IT, it’s kind of been shifted over to legal. So. And we’ve gotten quite a few questions around that regard. Let’s, let’s dive into some of these questions, because they’re pretty interesting.

The first question was. Oof, boy, how much time do we have on some of these? What legal foundation terms DPA, which, if you’re not familiar with that term, is typically data processing agreement. AI addendum, IP, indemnity, indemnity. That’s a… that’s a big word that gets thrown around a lot, isn’t it? Sometimes I say that’s the only thing you’re really paying for when you go with an enterprise contract. And liability provisions are essential for enterprise deployment. I guess That… I guess there’s two sides to that. That’s… that question is really framed in some regards for, you know, an external product, I think.

But, I guess you also have to think about internally, how you’re deploying it, and you know, Heather, I know you talked a lot about compliance, and I know Jamie, a CISO, I’m sure you own this from a… as a business perspective, as part of your role, so… you know, how are you approaching these problems in terms of what frameworks are you complying with? Are you creating your own? Have you created new DPAs that have AI addendum? Are you approaching these problems differently? Are you having to work more with legal that you did in the past?

I know that’s a lot to chew on.

Heather Wood:

It is, and this one, I mean, you can… this is definitely an attorney, right? So, I have nothing to do with the liability, indemnity, or IP side of things, but I do have a heavy hand in the DPA and AI addendum side. And a lot of this comes down to what is it that, you know, depending on what type of contract we’re looking at here, depend… so, let’s say you’re looking at, I’m a SaaS provider. And so I need to make sure that I’ve got a DPA available for my customer, as well as an AI addendum.

There’s a set of… principles that you’re going to want to have for the commitments that you make, but you also have to ensure that those commitments are something that your CTO and your CPO have also agreed to. And that’s strictly from the development side of things. Now, if you’re looking at this from a vendor side of things. You want to make sure that your relationship around your data is super crisp. Great example, would be, like, data enrichment services. You have to get very, very crisp on, this data belongs to us as the controller. You don’t get to take possession of that, especially if you don’t have, you know, the legal permissions in place to be able to sell slash transfer that data.

This data is data that you’re giving to us. You own that until you give it to us. We’re independent controllers, as an example. Some folks totally go in for joint controllership. Me, personally? I think that’s a nightmare. And so… at all costs avoided. It’s actually part of our internal policy. We do not do that, because it creates a lot of, like, joint liability that you don’t want to have to be responsible for another entity.

When it comes to the AI addendum, you have to be very crisp, especially from, you know, a legal perspective, being able to speak to This is what we do with data, this is how we ensure that we have legal permissions for data, this is how we train our models, this is how we train our traditional models, versus if you’re training LLMs, this is how our Agentic AI works, what it’s doing with your information. All of that needs to be super crisp from your internal folks. Around what it is that they want to do, so that you can redline that contract to fit your needs, and so that there is zero question about what your instructions as the data controller to your vendor R.

And so there’s a lot there, but it’s very specific to, this is the service that you’re purchasing, and these are the things that you need to be mindful of, so it’s also a very tricky question to answer.

Jamie Brown:

Yeah, it is tricky, and I’ll just preference, I’m not a lawyer, but I work with one every day. Yeah, so, you know, here at Vivin, we’re a small startup, but yet, you know, security works with IT every day, security works with legal every day, security is going to work with engineering as well. from a legal side, it is interesting, at least from my side of it. It does introduce some gray space, right? You know, Heather kind of mentioned it in terms of the data, and I see it as a data architecture you know, a workflow and understanding, you know, where… not just where is the data, but how is it being interacted with?

Where’s that data being stored? How is it being protected, and what controls are in place to protect that data. You know, when, you know, 3 years ago, when AI, you know, ChatGPT first came out, everybody was talking about training, training, training, right? You still hear the question when it comes to training, you know, is my data being trained on, those type of things. But now it’s a slightly different question. It’s more in-depth than just training. You know, most of the controls, at least for the enterprises, have, you know. Promised from a legal side, but also, you know, controls are in place to prevent training of your data or of your customers’ data.

But there’s still risk, and, you know, again, from a security stance, it’s always, you know, do we trust but verify? And how do we apply those same principles when it comes to AI or Agentic AI, and building in those controls? Daniel, I don’t know if you want to keep moving on to some of these other questions, but yeah, we can talk about the controls and some of the technologies associated with that as well.

Daniel Davis:

Well, I think there’s so much of the… I was just gonna say, I think there’s so much that we can talk about that I did want to quickly add to this, that we haven’t discussed Because now we’re kind of getting to that third-party risk. Is my data being used for training? How do you approach this, whether you’re using the closed pro- you know, closed proprietary models of OpenAI, Anthropic, Google, etc, versus using open-weight models that you can fully control Yourselves, because, you know, that seems like that’s a very different prospect, but at the same time.

If you just go by what you hear on the news, you have quite a few people playing both sides of this coin, trying to tell you that the open weight models are dangerous, and then at the same time, you have probably the security team telling you that Well, we can… if we deploy these models in our environment, and they’re open weight, and we can evaluate them, well, we can actually very granularly, with a quite high confidence, measure our security posture. So, and I know a lot of enterprises, we talk to a lot that all have… they’ve developed their own API gateways that just kind of is a faucet you can turn on with a blend of all the different kind of models out there.

So. How do you approach this problem, whether you are using the, you know, proprietary models where you are concerned, where what’s happening with my data that goes to them, versus just say, we’ll just do it ourselves and use open weight models?

Jamie Brown:

Yeah, it’s a tough problem, right? You mentioned the geopolitics. I won’t get into too much of that from a geopolitic perspective, but just from a technology sense, in terms of, you know, where’s the data, how’s that data being protected, and what are the risks associated with that? You know, data residency is a real a real requirement on many of our contracts, and understanding, you know, not just, you know, who has access to the data, but where is it actually being resided and protected. That’s a very important thing. So, you know, some of these open source models, you may be able to, you know, ensure they run on your hardware or rented hardware in certain data centers, but again, it comes back to The risk associated with You know, running… certain things that are more unknown than others, right?

Again, we have a third-party risk team that looks at third-party risks, looks at the vendors, does an assessment on that. It’s not that it’s impossible to do that with open source, it’s just… it’s just, you have to get more creative. And you have to understand where the risks are, and weigh those risks from a… development per side, as well as a customer side, as well as a company side. So, again, back to my point earlier, it’s not just one size fits all. It is a calculated risk, and you have to, you know, measure, okay, is the cost-benefit worth the open source versus paying for the proprietary model?

Right now, there’s a big price difference. I think competition It’s going to, you know, relax that price over time. But we’ll see. So Heather, any thoughts from your…

Heather Wood:

Yeah, and I’m gonna go a little bit away from, like, the LLMs specifically, and thinking about flows of data, because this will also go back to that last question that we were talking about. When you’re, all LLMs. And as you’re looking at MCP connections and plugins, as that’s growing, you’re going to have a more complex data architecture and data flow than you’ve ever had in the past. And there is a significant amount of risk that comes with that. As an example, let’s say you have subprocessors connected to your Cloud MCP. And now you’ve connected a different vendor to that same Claude instance.

You then have a risk of data. that has gone into Claude going into a sub-processor system. That’s already a risk that is an issue, right? But then, let’s say your contract with that subprocessor hasn’t restricted them from utilizing data that gets in there. Now they’re using your data to train. But they’re also using your customers, which was inadvertently leaked. There’s a lot of risk that needs to be considered, around where your data is flowing, and I think that looking at it from just the perspective of. these, you know, specialized, proprietary LLMs versus larger ones. Plays into that as well, because you have to be considerate of What does this look like when we start connecting other things to it?

Typically, our LLM providers are not sitting in a vacuum or a silo. It’s not just, I’m going to use Claude, nothing else is connected to it. That doesn’t give you productivity, it doesn’t give you the uplift that you’re looking for. And so all of that then has an impact on your data, as well as, depending on, you know, whether or not you are a provider that stores data, your customers’ data. And those are all things I think we have to be mindful of as we go into this world. When it comes to the proprietary model versus a larger one. theoretically, They commit in contract that they’re not going to utilize your data if you’ve got an enterprise contract.

Theoretically. Again, not a lawyer, not looking at contracts. Well, I do, but these are the things that, you know, they’ve contractually agreed to not have your data utilized in a particular way. That goes back to that question around what’s in the DPA? What’s in the AI addendum? How do we define data ownership? How do we define permitted use? that is restricted enough that it, you know, is defined as what it is that they’re supposed to do on our behalf, but it also defines what they are not allowed to do with our information. All of these play into that.

It’s kind of like a, I mean, it’s an ecosystem, but not necessarily the data ecosystem. That’s one type of ecosystem, but there’s also the contractual agreements that you’ve made, and that your vendors have made to you, that you have made to your customers, that play into that overarching ecosystem as you’re looking at the risk of connecting data from, you know, here to there.

Daniel Davis:

I’m so glad that you mentioned MCP, Heather, because, you know, that was seen as this magical solution that was going to open the door to connectivity and large-scale adoption of agents, and Many people, immediately said, well, wait a minute, we’re now creating a bit of a mesh network that’s, That doesn’t have any gating to who can join the network, which, as you were talking about, when you think about data flows, you now kind of have this one-to-many relationship where you have no control over it. Depending on how you design your agent, it may pull data from one vendor, one source, and then start passing it back and forth to people that You never intended for that to happen, and all of a sudden, now, how do you… how do you manage these… all these relationships?

My question is… we view MCP, or many people view MCP, as this quicker. Easier, lower-effort way of doing these integrations. However, can we mitigate the risk With this data flow problem, and this kind of one-to-many relationship. So that the risks don’t outweigh the ease of being able to make these connections.

Heather Wood:

Yeah, and this is something, you know, we’ve encountered internally. I feel like everybody is looking at this right now. When you look at MCP connections and plugins that are available, you’re then looking at, okay, what permissions does this MCP connection allow? And in a lot of instances, we’re seeing it’s just full enchilada, or my personal favorite, it’s inherited user permissions. Well, a task doesn’t necessarily need to have the same level of access that that user does, and so you have to be super thoughtful about how you set up not just your MCP connections, but also how you want your various LLM architectures.

Set up, keeping in mind that you might have to allow for You know, basically full access. for this agent to do a menial task, is that truly worth it? And you’re then having to look at the trade-offs, but it’s not just for that one MCP connection. Once those connections are put in, it’s like a giant spider web in the background, right? So you can have data flowing from here that you never intended to be over here. Even better, you may not have gotten, for instance, a DPA, because that system wasn’t intended for PI data, but now that data is being leaked into it, and so you have to be very thoughtful about these, and you can’t really be looking at MCP connections, number one, just as a, oh, that’s okay, the user has that access, so the agent should too.

I definitely would not recommend that. But you also need to be looking at, when you’re making these approvals to do these connections, what does that look like from a grander, you know, the broader perspective? So that you’re not doing it, oh, this is only going to connect here, it won’t touch that over there. Because that is, I mean. ultimately, you’re just setting yourself up for failure if you come at it from that perspective, knowing that it’s kind of like a plate of spaghetti. All those noodles are not touching each other.

Jamie Brown:

Yeah. Yeah, it’s interesting. I see MCP as just another layer of APIs, and I don’t think this is where we end up. There’s a number of vendors that we use where, you know, the enterprise version of their tool gives you very limited connect… gives you very limited control over the MCP. And that’s… that’s not what enterprises are asking for. Enterprises want the fine-grained permissions, they want the fine-grained access. And I… I see MCP kind of changing, evolving, how fast that’s gonna happen, unknown, but right now, it’s… it’s not… it’s not… it’s mainly there just for functionality as opposed to… security and what enterprises need.

So there’s definitely a gap from that side. Overall, you know, from our side, you know, it’s… you know, Heather kind of mentioned, you know, it’s… it’s allowing users to have access, it’s allowing users to have access to data. We see it, you know, from a cybersecurity perspective, is, you know, much worse than shadow IT, right? It’s like shadow AI MCP, right? And users just are asking for this access, and they don’t really understand that they have OAuth permissions, and those OAuth permissions have access to you know, not just email or calendar or, you know, Google Drive, or whatever the case is.

The point is that MCP connection is pretty powerful and needs to be looked at from the top down on a continuous basis. It can’t just be, oh, you know, IT just allows it, and there’s no risk associated with it.

Daniel Davis:

I’m glad you brought up that word shadow and risk, because one of the most interesting questions that got proposed for this talk, I think it was actually one of your questions, potentially, Jamie, and I would love to get your thoughts on this, I’m sure you have a lot, is what are the risks that potentially people haven’t even considered yet with deploying agents in their enterprise. What’s that… shadow risk that, as was, you know, somebody like, you know, Rumsfeld, you know, made famous, the unknown unknowns. What is that kind of unknown that potentially people haven’t even considered or are going to stumble into by accident?

Jamie Brown:

Yeah, it’s hard to predict the unknown, but what, you know, from my seat, I do see some companies underestimating the difference between just AI and Agentic AI, and one of the big areas is around memory, and understanding what’s actually in memory, and how does that memory evolve over time. you know, memory’s not just logs, right? Logs, you know, from a cybersecurity perspective, are pretty… you know, they’re dynamic because they happen a lot, but they’re pretty static in terms of what actually goes into a log. But when it comes to agent memory, that’s very dynamic, and that’s a very different set of data than logs, and I think for companies to really understand how does the memory evolve as relates to data classification, as it relates to data protections, is definitely an interesting problem.

You know, again, like, in one sense. You know, a version 1 of the agent just understands, you know, certain data flows, but then version 2 may understand, you know, customer data. And then version 3 may understand, you know, the customer’s customer data, right? And understanding that memory, and how that evolves, and how the data of classification may change over time associated with that memory is… is going to be an interesting thing to watch over time.

Heather Wood:

I think that also applies to the agents and the way in which they utilize that memory, because when you, as an example, you might go in and say, okay, I’m gonna build an agent to do X. and you tell it to do something with something very specific in mind, and probably a very specific data set in mind. But that agent has access to all of your memory, which means that the output may not actually be what you intended, because the memory alters the way in which the agent performs based on a set of data. And these are all things, you know, I… I actually… I had a conversation this morning where someone said, well, maybe we should just turn off all memory.

It’s like, whoa. That is not something that we’re going to be able to do as a business, so we have to make sure that we’ve got mitigations in mind, that we’re thinking through how we ensure that data is protected, because you can’t just say. okay, no data, and that way, all the data’s protected. It completely removes the business viability that you, you know, purchased that product for, or that you were looking at implementing those agents for. And so it is a very delicate balance between these risks as you’re looking at, well, do I just turn off memory, which we all know would be prohibitive to the business?

That means that that person that you were training, your co-pilot, your new co-worker, now has zero memory. Can you imagine? That’d be like Drew Barrymore on 51st Dates every morning with your agent? Oh, lordy. And so, you know, you can’t just remove data and call it a day, so you’d have to think through. what does all of this information over here actually mean if I set up an agent to do XYZ? What does it actually have access to, and how accurate is that data that it has access to? Is it a confluence, where we all know things go to die?

Is it a Confluence page from 6 years ago that nobody ever updated? Or is it something that somebody looked at last week and is still current and factual? What is this agent using in the way of information to then provide its outputs that you’ve requested from it? And that’s the reason that, you know. AI governance programs are so important as you’re looking to utilize these, is that you do have to think about not just what’s in front of you, but the entire ecosystem sitting behind it.

Daniel Davis:

It’s funny that you… when you said confluence is where, you know, data goes to die, I actually have been using, an analogy lately. In a past role, I was researching how many times had we in the company tried to create a cyber risk management program? And going through Confluence, I could find 4 different occasions where 4 people in isolation had tried to create different risk management programs, and that took a huge amount of manual digging through Confluence, and I don’t think anybody was really aware of that history. There’s actually a timeline of people trying to do this, but I do have to ask, quickly, I was taken a little aback by somebody suggesting just turn off the memory.

Do you know, why that… why they made that suggestion? Like, what would they What was their motivation and hoping to achieve by doing that?

Heather Wood:

Yep, the… the thought process behind it was, this eliminates risk. And the.

Daniel Davis:

Well, I guess, yeah. That’s one way to do it.

Heather Wood:

And, you know, this… there’s a… there’s another option, but, you know, it’s very cost prohibitive. And then there’s this option over here, but it relies on manual controls. And these are all things that, you know, we have to be mindful of as we’re looking at risk, especially data leakage going to pla- you know, data going to places it’s not meant to be. presents a significant risk to any business. And so, you know, that was the, well, can we just not have data memory? Whoa! not a recommendation I want to make. And I can actually, I mean, I can see the way that leaders would look at that now, like, wait, I… I just trained my instance of Claude under my name to do all these things.

I’ve put the skills in, I’ve got a project that I’ve been working on for the last 12 weeks. Can you imagine just turning around and being like, alright, you just have to retrain it every morning, copy and paste from this Word doc over here into here? You know, I… Lord help me if anybody ever brought that as a proposed solution. But, you know, these are the things that, as compliance professionals. We have to be mindful of this is the risk it’s presenting. this is the output that I’m trying to get to. How much business risk do we accept?

Where do we accept it? What things can we put into place from a business process perspective? what does the system actually allow us to do? These are all questions that you have to be looking at, and preferably during your review of a system that you’re planning to procure, instead of after the fact during implementation, because One of two things is going to happen. Either implementation is delayed, and no one wants to see that, because you’re paying for something you’re not getting use out of. or implementation proceeds forward without you, and then you have to figure out a way to clean that up, and in some instances, you can’t.

You just have to accept the business risk that comes with implementation prior to risk evaluations. And so these are all things that we have to keep in mind, and part of the reason that you, you know, these are where you put controls up in the business, but… You also, you know, and Jamie, I’m sure that you have to think about this all the time. There’s… this is the risk, and this gets us to, you know, close to zero risk. You’ll never be zero risk. Anybody who thinks they’ve got zero risk is crazy, but gets you close to zero. this is… we do nothing, you’ve got the most amount of risk, how do we determine where in the middle here on the spectrum that we’re going to get based on the business need?

And, you know, business need isn’t just about business need, it’s also about, like, what… And actually, Daniel, I think you mentioned this earlier. You’re looking at what is the risk? What is our legal responsibility? And one thing that you didn’t mention, but I think is worth calling out, what is the revenue impact to the business?

Daniel Davis:

A business…

Heather Wood:

It needs to keep its people, you know, employed, and so that does come in.

Daniel Davis:

Yeah, absolutely, and I think we are coming up again to the end of this panel, and I did have one quick question I wanted to finish with. And I’ll let you lead this one, Jamie. We did talk about how, you know, agents to agents, once you start letting them work without humans in the loop, potentially, hallucinations can propagate and How do we… how do we prepare for when this thing potentially could spiral out and create bad outcomes? How do we have a button we can hit to shut it down? Or how do we even know when to hit the button, and do you have a plan on how to attack this problem?

Jamie Brown:

Yeah, that’s a… that’s a whole other hour of a conversation, Daniel, for sure. Essentially, again.

Daniel Davis:

Well, you got 60 seconds.

Jamie Brown:

Alright, it comes back to the same principles, right? Data availability, data protection, as well as systems engineering, right? So, when it comes to designing the systems, when it comes to understanding the risks of the systems, you have to account for not just if, but when bad things happen, right? In the cybersecurity world, like, hackers aren’t going away, their jobs in some cases are getting easier. And you have to be prepared for, you know, a rainy day, you have to be prepared for a bad day. So, when it comes to Agentic AI, when it comes to those risks.

You have to put guardrails in place. You have to test those guardrails, you know, going back to, you know, the beginning of the conversation, Jurassic Park. You gotta test the fences, you gotta test the controls, right? To make sure the defenses are working the way you think they are, right? Canaries in the coal mine. That’s how cybersecurity does it. So, the point is, you have to plan, and then you have to execute the plan, but you have to trust the plan, and more importantly, to validate the plan. There’s a huge audit function with security, and the audit isn’t just a SOC 2 audit.

The audit is validating the controls are in place, validating the specifications are working accordingly, and that’s one of our major jobs that we do.

Heather Wood:

Wholeheartedly agree, and from here forward, we’re gonna need to have Jurassic Park, pictures up on our LinkedIns.

Julia Nimchinski:

Phenomenal discussion. Thank you so much, Daniel, Heather, and Jamie. What’s the best way for our community to support you?

Daniel Davis:

Well, for myself, as I mentioned, Trustgraph is open source, Trustgraph.ai, you can find it on, GitHub. I all of a sudden blanked on, where’s code stored? You can easily find me on LinkedIn, and all of our links are there. Also, YouTube, YouTube at Trustgraph dot… not Trustgraph AI, that’s actually where most people find us these days.

Julia Nimchinski:

Heather, how about yourself?

Heather Wood:

I would just say, you know, community. To Daniel’s point, LinkedIn, but being able to, you know. toss ideas back and forth. I’m a very green person, if anybody’s ever taken the insights profiles. I love being able to hear from other people. I have my experience, but I am not the end-all, be-all of knowledge, and so I love hearing what other people have done, what challenges they’re facing. You know, other companies might face that challenge before we do. We might face challenges before, you know, another company does. And I think community really helps drive improvements in being able to protect The companies we work for, as well as the people whose data that we hold.

Julia Nimchinski:

Thank you. Jamie?

Jamie Brown:

Yeah, definitely LinkedIn. And the other thing is, you know, being at Vivun, we are a startup, so we definitely appreciate other startups kind of tackling this challenge as well, so we certainly don’t want to do it in a vacuum, and we’re always open to learning new ideas, as well as being design partners and working with other companies to kind of tackle these things together. Because as we started, this isn’t just an us problem, this is a we problem, and we have to figure out how to work through this together. So, thank you.

Julia Nimchinski:

Thank you so much.

Heather Wood:

Thank you.