Skip to content

Developers, get ready for the collision of DevOps and data analytics

The COVID-19 pandemic has accelerated the blending of data analytics and DevOps meaning developers, data scientists, and product managers will need to work more closely together than ever before.

The COVID-19 pandemic has accelerated the blending of data analytics and DevOps meaning developers, data scientists, and product managers will need to work more closely together than ever before. In this episode of TechRepublic’s Dynamic Developer , host and TechRepublic editor-in-chief Bill Detwiler speaks with Michael O’Connell Ph.D., chief analytics officer at TIBCO Software about how data analytics is merging with DevOps, the data science work his company has done helping organizations respond to the COVID-19 pandemic, and what that works tells them about the future of software development and analytics. The following is a transcript of the interview, edited for readability.

Bill Detwiler (00:16): All right. So before we get started talking about data analytics and DevOps, and how the current COVID-19 pandemic is affecting that, give me a little rundown on TIBCO Software. You all specialize in data analytics. Tell me about what you do.

Michael O’Connell (00:34): Absolutely. Yeah, so the TIBCO Connected Intelligence platform, it’s got three pillars. There’s the connect pillar, API-led microservices, real time data integration. The unified pillar is where we virtualize data, we model data, master data, metadata, reference data, and create virtualized views on multiple source systems. Primarily data rest, but real time data sources as well. And then feeding that into the analytics lab, which includes our visual analytics, data science, and streaming analytics. Three pillars to the platform and, most business problems we can cover by combining those building blocks and creating analytic applications.

Bill Detwiler: And you work on everything from retail to the COVID-19 pandemic, all sorts of data science questions, right?

Michael O’Connell: Yeah, absolutely. As chief analytics officer, I am customer facing, it’s an offensive position. But I’ve also got one foot in the product teams where I help drive the input for the product evolution based on what I’m seeing in the field. So yeah, we work heavily in the finance sector, energy manufacturing, retail, consumer goods, telco, travel, transportation, logistics. All of those industries have been affected in various ways by COVID-19. I’m sure we’ll get into that.

SEE: How to become a data scientist: A cheat sheet (TechRepublic)

Chief Analytics Officer: Using data science and analytics to transform businesses

Bill Detwiler: So I’m really interested to hear a little bit more about your role as chief analytics officer. A lot of times, we hear the CXO moniker applied in technical positions these days, because there are so many variants, everything from a CSO, CTO, CIO. So, talk a little bit about what the chief analytics officer role is.

Michael O’Connell, Ph.D., chief analytics officer at TIBCO Software

” data-credit=”Image: TIBCO Software” rel=”noopener noreferrer nofollow”>Michael O'Connell, Ph.D., chief analytics officer at TIBCOMichael O'Connell, Ph.D., chief analytics officer at TIBCO

Michael O’Connell, Ph.D., chief analytics officer at TIBCO Software

Image: TIBCO Software

Michael O’Connell: Yeah. So it’s like I was saying, it’s a one foot in the customer and one in the product. And it’s also building out the ecosystem a little bit. So, customer innovation, community, but, I meet with a lot of customers, all our marquee customers and bigger accounts, help them figure out their digital strategy and so on, and really trying to understand what are the data sources, what are they trying to get done? Where’s their opportunity to create value with digital transformation initiatives that are driven by analytics and data science? And then those sort of learnings, I bubble back to the product teams where I’ve joined at the hip with a lot of the product managers, tell them what I’m seeing, as we figure out the next releases of the product, what’s going to really create, move the needle for our customers and their businesses, create value, with our software. So it really is a fun job, to help our customers use data science and analytics to transform their businesses and then use that to transform our software, to generate that value for our customers.

Bill Detwiler: I think that’s something that’s really interesting. What you said was around helping the customers unlock their data, or understand what they are trying to do with data. Because, when I talked to data analytics folks, one of the things they say is a lot of times, they go out to companies, and the companies have data, or they think they have data, or they want to collect data, but they don’t necessarily know what question they’re trying to answer with the data. And that’s really the first place to start. It’s not, “Let’s collect everything and then figure out how we sort through it later.” “Let’s figure out the problems we’re trying to answer first, and then design a system that helps us collect the data we need to answer that question, and then can help us make the decisions based on that answer.” So, how difficult is it to help companies or, talk a little bit about that role. When you walk into a company and they say, “Look, we want to do X.” How much of a challenge for you is it to help them shape, what they want to do with data?

Michael O’Connell: Yeah. So, like you say, you’re going to focus on the high value business problems where you can really, generate revenue to the bottom line or take out costs, improve productivity, manage risk. One of the big initiatives that the executive leadership team has for the company, and how can analytics play a big role in generating value around those initiatives. That’s kind of the starting point. And in some industries, the value calculations can be ginormous. When you’re thinking about, say energy sector being depressed at the moment, but that’s a time when you can optimize. So how do you get the most out of different production facilities? If you can minimize downtime on a high producing asset, that’s like the value calculation is off the charts.

In the current climate, say in the law 48, at least, when a well starts producing, do you even bother bringing it back up? How’s the strategy different for, law 48 versus offshore? And, the national oil companies run their business very differently. They’re providing energy to their country, which is different than taking a profit out of a shale oil well, which is, less profitable at the moment. So, everybody’s got their own business objective. How do we optimize that with analytics, is the challenge. And in post or in the current COVID world, retail, CPG, these industries are really transforming right in front of our eyes. And analytics is a big deal for optimizing all of the aspects of those businesses, supply chain, all that kind of stuff.

SEE: Feature comparison: Data analytics software, and services (TechRepublic Premium)

COVID-19: Navigating a pandemic with data science 

Bill Detwiler: Well, let’s talk about that, because TIBCO has done some interesting work around COVID-19. And looking at, done some modeling around the virus spread, trying to help your customers, help people be able to reopen, get back in business, figure out how this is going to affect their businesses. Tell me about some of the initiatives that TIBCO has undertaken in these last few months.

Michael O’Connell: Yeah, for sure. Well firstly, it’s important to realize that, the COVID or the virus is hidden from us. Data science and analytics becomes hugely important in finding out, what the hell is going on out there. It’s like, what you see right now happened two to three weeks ago. Or, what you’re looking at right now is going to show up in two, three weeks. It’s kind of like a Twilight Zone episode. You’re predicting the future, but the future is actually right here right now. It’s odd. It’s like this weird time delay. And so, we’ve got, in the work we’ve been doing analyzing the COVID cases worldwide, we’ve got models that are predicting, the reproduction number of the virus, and all these spikes that are occurring in Florida and all that, we knew about those three weeks ago. We knew that was going to happen. But exponential growth is tricky for people to get, especially when it’s hidden.

TIBCO Software COVID-19 Live Report

” data-credit rel=”noopener noreferrer nofollow”>TIBCO COVID-19 Live ReportTIBCO COVID-19 Live Report

TIBCO Software COVID-19 Live Report

It’s like there’s a delay between when you get infected and you’re infectious until you get symptoms and you’re infecting all these other people. So if you think about, two to the seventh, it was 128. Two to the 14th, 16,384. And so you think, “Oh, I’ve only got 128 cases.” I’m making an analogy here obviously, but the way that it is hidden and exponentially growing, all of a sudden, you’ve got these problems on your hand, but we can predict that. We’ve got models that are predicting the reproduction of the virus. There’s lots of problems with the data as well. It’s got a lot of reporting artifacts. Mondays, a lot of cases are reported. The weekend, not so much. There’s, lots of errors and artifacts in the data.

So smoothing that out and fitting models to actually find the signal in a noisy landscape, another important data science contribution. So, in dealing with the data, we’ve got a lot to offer. And then, we mentioned CPG and retail, how do you predict a retail business when traditionally that’s done with same store sales, and there’s no store sales? So it’s getting the eCommerce data, combining it with what limited in-store sales you might have, and then starting to understand, which type of stores might get sales. It’s not going to be indoor malls. It’s not going to be stores that are in tourist-driven areas, there’s little tourism. So how do you start to understand the attributes of the stores and, where revenue might come from and how to maximize the eCommerce revenue?

There’s also, the ad spend media landscape’s changed, as people are moving their ad spend to channels like Hulu, and away that everybody’s watching, and away from more traditional channels. And we are seeing spikes in eCommerce revenue. So, lots of people buying lounge wear and, floral pink pajamas and stuff. But, that’s some of the oddities we’ve found in data mining the sales, but in general, eCommerce sales are going through the roof. And how do you then manage the store re-openings and predict the business, when all of that is, the sands are shifting under our feet?

Blending data science and DevOps to enable digital transformation

Bill Detwiler: And I think something that’s also really interesting that you touched on a little earlier and we talked a little bit about before the show, which was, this pandemic came on quickly when it did come on. Not to talk about, you can debate how soon people should have known or did know about it. But, from a perspective of the shutdown, companies had to react very rapidly to the changing business climate, to a changing workforce. And, you talked about some of the ways that TIBCO, your team, had to quickly spin up some of these efforts. You were talking about how you had data scientists, data analytics folks, really having to rush into DevOps, and kind of try to, “Okay, how can we spin these processes up? How can we spin these services up? How can we spin the analysis up very quickly?”

Bill Detwiler: And vice versa, you have companies that were trying to react really quickly to this pandemic, and DevOps folks saying, “Oh, well we need to integrate data science into our processes, into the apps that we’re building. We have people asking us for tools to help them address these challenges, and we need data in the app.” And that creates this back and forth a little bit between DevOps and data science. Talk a little bit about that, from your perspective. From working there at TIBCO and doing this and what you’re seeing with your customers.

Michael O’Connell: Yeah, it’s been fascinating, the merging of data science and DevOps. So to your point, when we started working with the publicly available data from, Johns Hopkins, Our World in Data and so on. We brought it together, started creating analytic apps, and then we’re like, “Well, a lot of our customers want to see this.” And so then we started getting into the DevOps side, but that’s a whole ‘nother world. And the data scientist, traditionally it does a handshake with somebody else, but no, we had to move quickly. So we started up our own servers. We ended up with, a traditional DevOps, you got a blue server and a green server, you’re rapidly innovating on the green server, the blue server’s in production. You flip them out.

And, we built our own little, inside of our data science team, we built a little DevOps team to actually do that. And then we found that, the Hopkins data and other data was pretty limited. So we started going getting data from all these other places. It’s, amazing number of data sources out there. We’ve got the Google mobility data, we’ve got the COVID tracking data. We’ve got unemployment data in there that some of our customers are asking for. We’ve just got a ton of data now that is in the application from all these multiple sources. So we’re federating that, and, in a Postgres database, putting our analytics apps on top of that, we’re bringing in other data sources through our data virtualization layer, providing data services. It’s become this massive operation with lots of different data and analytics and up into our live hosted app that’s refreshing multiple times per day.

TIBCO COVID-19 Live Report: Business Reopening Evaluation – US

” data-credit rel=”noopener noreferrer nofollow”>TIBCO COVID-19 Live Report: Business Reopening Evaluation - USTIBCO COVID-19 Live Report: Business Reopening Evaluation - US

TIBCO COVID-19 Live Report: Business Reopening Evaluation – US

And then, we got into, we started collaborating with these scientists to stop COVID around how to reopen society. And then our DevOps and engineering team under the office of the CTO got involved. And we built this application, TIBCO GatherSmart, to help bring people back together in a safe way. So that became a focus. We built a control center and a mobile app for people to do symptom tracking. You can sit in the control center, figuring out who’s going to get what survey. They answer the questions, you decide if you’re going to give them a QR code to come into a building based on their self-reported symptoms or hotspot, local hotspots. And so this became a DevOps-led project to create the cloud-based application and the mobile app for doing the surveys, and onboarding the employees and the university students, and so on.

That’s become quite popular too. So we’re about to launch that in July, this TIBCO GatherSmart. But, we wanted to have a bunch of analytics and data science in that, for the control center that the employer is looking out across their collection of employees, and so on. And so then we were the data science fairy dust into that DevOps-led initiative. So we’ve seen both sides of it at TIBCO. Data scientists becoming DevOps savvy people, and DevOps people becoming data scientists, and then the boundaries are blurring at TIBCO, and I’m seeing that, in the rest of the world too. It’s where the fast forward button has been pressed, Bill. It’s crazy.

SEE: 60 ways to get the most value from your big data initiatives (free PDF) (TechRepublic download)

Building data science into the application development process

Bill Detwiler: Yeah. I think that’s one of the things that we’ve talked a lot about is the acceleration. A lot of these digital transformation efforts were already underway, but they’ve been time-compressed because of the COVID pandemic. So what lessons did you learn that maybe other organizations, who are working on applications and systems and products, and their engineers are working on them right now, the product managers are working on the right now. If you want to incorporate analytics into those applications from the get-go, what are your recommendations to them based on your experience?

Michael O’Connell: Yeah, so we found that, certain people on either side of that, fans if you will, have a desire and passion to learn the other side. And it’s, be a learn-it-all, not a know-it-all kind of thing. And so, certain folks on the data science team got fascinated with, “How do I make this, low latency, high throughput, high response. How do I operationalize this stuff?” And, they sort of stepped into those roles and, as a learning experience. And then similarly on the DevOps engineering side of the house, some people, they were like, “Well how do I make this control center app cooler? And how do we bring in data and present, multilayered analytics views for consumers of that, and getting fascinated on the data science side?”

But you do need a bit of both. The data scientist wants to get the DevOps stuff going, needs to be able to go to somebody and ask them, “How do I do this? Can you help me do this?” And vice versa on the engineering side, injecting data scientist’s perspective into that world, and the cross-fertilization back and forth. It’s been really cool and people have gotten to learn a lot outside of their own comfort zone.

Bill Detwiler: Yeah, and what type of tools did you all use internally to facilitate that type of communication? And I guess practices as well. So, is it happening during weekly stand ups, is it happening during the product design phase? How’s that work, and what are some learnings that maybe some things that didn’t work really well or things that, did work really well? So, to make that cross pollination happen?

Michael O’Connell: Yeah, yeah. Well, it sort of started with a Slack channel that we set up around the pandemic. And, back in late February, early March, that Slack channel got really popular. And I think we’ve still got, I don’t know, 500 to a thousand people on the main COVID Slack channel. And that week, I sort of put those people to work as well. Cause as we built data around government interventions at a local level, the people in the channel were all from around the world. And everybody started chipping in, we designed a certain format about the metadata around government interventions, whether it was school closures or shelter in place, or whatever it was, we had a taxonomy then, and people went to their local regions and started filling it in. And we’ve actually used that metadata to annotate all of the analysis, and that’s become quite important as things have, interventions have been lifted, but then in some places put back on, and now you can see the changes in the reproduction number or the hotspots with that metadata, lay it onto it, that’s been literally crowdsource collected around the TIBCO worldwide team.

So the Slack channel has got a lot of enthusiasm going. And as we started to release, the Spotify live report, and the data sources started to get added and everybody was like, “Well, can you put in this country?” And so I said, we started with Hopkins, but now we’ve got most countries in the world where we’ve actually gone out and got the data ourselves from different department of health sites and so on, and assembled pretty broad coverage at a very local level, across worldwide countries. And that’s been driven by that internal enthusiasm around the Slack channel.

TIBCO COVID-19 Live Report: Outbreak Hotspots

” data-credit rel=”noopener noreferrer nofollow”>TIBCO COVID-19 Live Report: Outbreak HotspotsTIBCO COVID-19 Live Report: Outbreak Hotspots

TIBCO COVID-19 Live Report: Outbreak Hotspots

Now, as the projects got stood up, we had daily stand ups, we had weekly leadership team stand ups, and we’ve got that, on the live report analytics app for COVID. And we’ve also got that on the GatherSmart project. And those projects have now merged a little bit, and we’ve been able to cut back some of the daily stand ups, but we’ve got very regular meetings on both the projects and also a group that’s working on the confluence of the two. So Slack’s been important, I guess, the usual standup meetings, and then finding these little target teams of people who are really enthusiastic to learn each other’s world, and collaborate.

Bill Detwiler: Is that something that TIBCO had focused on, before you started addressing the COVID-19 pandemic, which is sort of that cross-functional development. Sort of as part of just normal workforce development, encouraging people to maybe push outside their comfort zone, and also providing them time and the resources to do that. Was that just part of TIBCO’s DNA?

Michael O’Connell: Great question. Because, some years ago, we had this next big thing concept at TIBCO, and AI was one of the next big thing topics. And so we had a lot of our product teams were challenged on, “What are you going to do when it comes to AI?” And we have a philosophy of AI being a foundational element of the dev process. And so, you think about the three things that drive us, cloud native is one, AI as a foundation is another one. And these foundational elements are driven from the top down. So, every product manager has to have a better together strategy of the different TIBCO products working together. They have to have something on, “What’s your AI foundational aspect of this product? What’s the cloud native part, and then what’s the open source, embracement part of it?”

So the three guiding principles are infused through our product. But I guess that the AI part is the bit that, has made the data science groups work together with the integration products, because it’s just been a natural top down led initiative is, let’s bring AI into all the products at a foundational level, that’s been going on for a few years. And that has started some years ago, this collaboration between data science and DevOps. But, like you said, the COVID thing has just accelerated that. We built this GatherSmart app in like six, eight weeks. And the same thing with the live report on COVID. That just was grown from the ground up in a matter of weeks. Everybody’s just working hard and having fun.

SEE: Coronavirus: Critical IT policies and tools every business needs (TechRepublic Premium)

Data science is becoming highly democratized and merging with visual analytics

Bill Detwiler: Yeah. Where do you think we’re going to go from here when it comes to data analytics, and sort of incorporating that into almost everything? We’ve all become, I think, data analytics experts, or not experts, but at least conversant in the topics. I think with the COVID-19 pandemic, with the daily briefings that people get, we’ve learned about terms like flattening the curve, and we’ve learned about exponential growth. So, for someone like yourself or someone like me who was in a previous life was a research scientist, the DB administrator, I worked in social science research.

So I love anything with data and analytics and being able to tell a good story with it. But a lot of people, you can get lost in the data. A lot of people can glance over it. Sometimes data can be used in ways to tell it to support a particular point of view, so people discount it. But I think now, it’s in the mainstream, and there is an effort to make data science and analytics and especially visualizations, more prominent and hopefully society in general, and then hopefully people will make educated decisions based on some of this. Where do you think we’re going from here? So is it, I’m assuming it’s increased acceleration, but what do you see on the horizon for analytics?

Michael O’Connell: Well, the world of data science and visual analytics have just collided and merged. And then they’ve also become highly democratized to your point. A turning point for me was when Governor Cuomo started tweeting out, “We got to get our effective reproduction number down to 0.8 to bring this under control.” I’m like, “Dude, that’s amazing!” I retweeted that, and I was like, I mentioned to you earlier, we’ve had tools in a live report that estimate the reproduction number at a very local level, at the county level and so on. We’ve been tracking the spread of the virus. We saw Michigan before it erupted. Most recently we saw Arizona, Florida, you can see it, we’re predicting the spread. The effective reproduction number is a function of time. But I thought it was a real geeked out thing.

And then Cuomo was tweeting about it, that he’s going to get it down to 0.8. And it’s like, data science is in your living room every night. And it’s just amazing that, you don’t have to spend time building the context of a conversation you want to have with a customer about data science. It’s already there. People are seeing it every night in their living room. So, it’s just been fantastic for me as a data scientist to see how democratize widespread, data science has become, and everybody is becoming a data scientist, it’s awesome.

Bill Detwiler: Yeah. And it’s sad, that it took something as tragic as the COVID-19 pandemic to do that. You and I, people who have been around data and analytics and systems like this for so long, you would be much happier going back to doing retail analysis. It’s just, it is, it’s heartbreaking that it’s something that it’s so tragic, that has brought this to the forefront, but at least it can be a turning point to help people understand how important analytics and data, and science is to sort of making effective decisions. Well Michael, I really appreciate you taking the time to be here with us again. It’s been a great conversation.

Michael O’Connell: Well let’s get back together sometime soon. I mean, we’re only six months into this year, and we’ve had a global pandemic, we’ve had a racial uprising and an economic meltdown. So, Bill let’s get back together later in the year and see where we stand at the end of the year.

Bill Detwiler: Hopefully things will be better, fingers crossed. It can be much better. All right Michael. Thank you again.

Michael O’Connell: Yeah, cheers.

Get more Dynamic Developer