Updated March 5, 2026
0:00 Welcome to Colaberry AI podcast brought to you by Colaberry AI Research Labs and the Carl Foundation. And welcome to the deep dive, where we kinda go deep and try to figure out what's going on in the world of AI and data science. Yeah. You know what I mean? Absolutely. 0:16 And then try to explain it in a way that's interesting and fun and approachable so you can kinda feel like you know what's going on Yeah. Even if you don't work in those fields. Right. Today, we're gonna be diving into a project that was done by Colaberry Inc. They worked with a major agriculture science company Okay. 0:34 To develop a really advanced econometric pricing model, and we've got their technical document. And, I guess our mission today is to kinda see how data science and AI are revolutionizing, pricing strategies and to look at, some of the innovative approaches they're taking, in this case, in agriculture. Right. But I think the ideas apply to a lot of different industries. So, yeah, I'm really interested to kinda unpack this document and see what we can learn. 1:04 Yeah. Me too. So the first thing that struck me is how tricky it must be to get pricing right in the agriculture world. Oh, absolutely. Right. 1:14 I mean, think about it. You've got so many factors that are completely out of your control. Yeah. You've got the weather. Yep. 1:20 You've got, you know, global markets, you know, pests and diseases. Right. All these things can impact, you know, supply and demand of different products. And so trying to set a price that's both fair and profitable in that kind of environment is really a massive challenge. Yeah. 1:36 I guess the old ways of doing things just weren't cutting it anymore. No. They weren't. You know, they they mentioned in the document that, the traditional econometric models, you know, they're good for some things, but they can struggle to adapt to how quickly markets move. Right. 1:55 You know? They're not always great at predicting the real world impact of price changes. Yeah. So, like, in in in simple terms, they're they're not keeping up with the times. Exactly. 2:05 They're a little bit like, using last year's weather for cast to plan a picnic today. Yeah. Like, if there's a sudden heat wave and a bunch of crops get wiped out. Right. Right? 2:15 You can't use, like, an old model to figure out what the price of those crops should be. Exactly. Right. It's gotta be something that can respond to those changes Yeah. In real time. 2:24 Now, they also talk about how, traditional machine learning models can have a hard time with something called elasticity. Yeah. That's a good point. Can you explain what that is in very simple terms? Elasticity is basically how sensitive sales volume is to changes in price. 2:46 Okay. So, like, if something is very elastic Right. If you raise the price even a little bit, people will stop buying it. Exactly. So a good example of that would be, let's say, you know, a certain brand of cereal. 2:59 If the price goes up by, you know, 50¢, people might switch to a different brand because they see it as too expensive. Right. But if something is inelastic, you can raise the price a bit, and people will still buy it. Yeah. Like gasoline or something. 3:14 Exactly. You kinda need it. You need to you need to get to work. You need to, you know Right. Get around. 3:19 So machine learning models aren't always great at factoring that in. Yeah. They're not always great at Right. Directly accounting for that. Okay. 3:25 You know, they're good at finding patterns in data, but elasticity can be tricky for them. So you've got these two approaches. Right? The old econometric models and the machine learning models. Right. 3:36 And they both have strengths and weaknesses. So what did Colaberry do? Well, they did something really interesting. They decided to combine the two. Oh, okay. 3:43 So they they kinda took the best of both worlds. They took the best of both worlds. Okay. They, used machine learning algorithms like XGBoost and gradient boosting, which are very good at prediction. And they combine that with classical econometric methods. 3:58 Okay. So the idea was to create a hybrid model that could both accurately predict how prices would impact sales, but also adapt quickly to changing market conditions. So I guess, like, you know, the machine learning side brings the kind of real time adaptability. Like And then the econometric side brings the kinda economic theory. The economic theory. 4:20 Right. So you get a model that's both accurate and responsive. And that can really, you know, incorporate that elasticity factor that we were talking about. Absolutely. Okay. 4:30 Cool. So another interesting thing they did was they used AI tools to actually gather the information that they needed Right. To build their model. They use what are called text analyzers. Analyzers. 4:40 Yeah. And, basically, these are computer programs that can read through tons of text, like research papers and market reports Okay. And pull up the key information. So I guess in this case, you know, they would feed it a bunch of PDFs Right. About agriculture markets. 4:54 Exactly. And the text analyzer would say, okay. Here are the key trends. Here are the prices. Here's the, you know, the the supply and information. 5:01 Right. And and all that stuff can then get fed into the model? Exactly. It gets fed right into the model. Okay. 5:06 Cool. So once they had this model built, the next step, I guess, is to actually figure out what prices to set. Right. Right. And that's where they use something called automated optimization. 5:18 Okay. And that sounds complicated, but it's actually a pretty simple idea. Basically, they use algorithms to automatically search for the best possible prices. Okay. So instead of, like, a person having to sit there and Yeah. 5:30 You know, tweak the prices and see what happens, the algorithm can just kinda figure it out on its own. It can figure it out on its own. Right. And one specific technique they mentioned is called particle swarm optimization. Particle swarm optimization. 5:42 Yeah. And it's kinda like, imagine a flock of birds searching for food. Okay. They all communicate with each other, and they kind of collectively figure out where the best food source is. These algorithms work in a similar way. 5:54 Okay. They have these multiple particles that explore different price options, and they learn from each other to find the best solutions. Oh, okay. So it's like a whole bunch of little kind of virtual birds Virtual birds. Birds. 6:05 All working together to find the best price. Exactly. That's really cool. Now, of course, when you're dealing with all this data, you have to make sure that it's handled responsibly. Absolutely. 6:14 Data security and privacy are paramount. So how did they approach that? They they had a really robust process for data handling. They made sure that all the data was collected securely, either from the client's databases or through, you know, secure transfer methods, and they always made sure to comply with data privacy regulations. Okay. 6:36 So, you know, they were very careful about protecting sensitive information. That makes sense. And they also had to have a really sophisticated way of managing and analyzing all that data. Right. So what did they use for that? 6:48 Well, they they used a combination of different tools. For storage, they used cloud platforms like AWS s three buckets. AWS s three? Yeah. Okay. 6:58 And for large scale data queries, they used Google BigQuery. Okay. They also used a tool called TIBCO Spotfire for data visualization. Okay. And then for actually processing the data, they used a mix of SQL, Python, and R Okay. 7:17 And Spark as well for big data workflows. So So it's like a whole ecosystem. There's a whole ecosystem of tools. Of tools all working together. Right. 7:25 Now one thing they highlight in the document is the innovation through automation and the cloud. Yeah. Can you unpack that a little bit? Like Sure. What makes what they did so innovative? 7:40 Well, I think that the key innovation here is the automated pricing mechanism. Okay. Right? So as we talked about before, they use these gradient free optimization algorithms to automatically find the best prices without needing a human to constantly intervene. Right. 7:55 And that's a huge step forward from traditional pricing methods, which often relied on manual adjustments. Right. And then the cloud aspect is also crucial for innovation because it allows for scalability. Okay. Right? 8:08 They can easily expand the system to accommodate more data, more products, more markets Right. Without having to invest in a lot of expensive hardware. Right. It's all kind of virtual. It's all virtual. 8:20 Yeah. Okay. Cool. So we've talked about the high level stuff, but I'm curious about the actual technology stack they used. Yeah. 8:27 Like, what specific tools did they mention? Okay. Well, for cloud services, as we mentioned, they used AWS and Google Cloud Platform Okay. GCP. GCP. 8:38 For their analytics and development, they used Domino, which is a platform that helps manage the different components of their models and makes it easier to update and improve them Okay. For visualization and interactive tools. They use Dash by Plotly. Dash by Plotly. Yeah. 8:54 And that allows them to create these interactive dashboards where people can explore the data and see how different scenarios might play out. Okay. And then for the machine learning algorithms themselves, they used XGBoost, gradient boosting, and particle swarm optimization. Okay. And then for programming, they used Python and R Okay. 9:15 Which are kind of the go to languages for data science. Right. And they also use Jupyter notebooks for development. Okay. Cool. 9:24 So that's a lot of different pieces. Can you kinda give us a sense of how it all fits together? Like, how does the data actually flow through the system? So imagine it like this. You start with data ingestion, which is basically bringing the data in from various sources, and that's done securely as we talked about. 9:41 Okay. And then the data goes through processing, often using Spark to handle the large volumes. Okay. And then it gets stored in the cloud, either in AWS s three or Google BigQuery. Then the data scientists, they develop their models in Jupyter Notebooks, which is a really great environment for experimentation and collaboration. 10:00 Okay. And then once a model is ready, it gets deployed using Domino, which ensures that it can scale to meet the demands of the business. K. And then finally, the insights and recommendations are presented through those interactive dashboards that we talked about. So it's like a whole kind of pipeline. 10:16 It's a pipeline Right. From raw data to insights and actions. Yeah. Okay. Cool. 10:21 Now one thing I found really interesting in the document was they talked about some unexpected agricultural insights Yeah. That came out of this project. That's right. And and these weren't even directly related to pricing. No. 10:33 They weren't. But they kinda show how powerful data science can be. They do. Right. So one of the things they found was that they could identify just five key hyperspectral wavelength bands. 10:44 Five. Yeah. Out of how many? Out of hundreds? Out of hundreds. 10:47 Yeah. Wow. That are the best indicators of plant health and stress. Okay. So, traditionally, you would have to measure, you know, hundreds of these wavelengths, which is very expensive and time consuming. 10:59 Right. But now they've shown that you can just focus on these five Yep. And get a really good understanding of how the plant is doing. Okay. So that has huge implications for, like, building cheaper sensors. 11:12 Exactly. You can build sensors that are much simpler and more affordable. Yeah. Okay. Another interesting insight was that they were able to categorize different corn hybrids Right. 11:23 Based on their drought tolerance. Their drought tolerance. Yeah. Yeah. Using data analysis. 11:28 Right. And this can help plant breeders make better decisions about which crops to grow Absolutely. Especially in areas that are prone to drought. Especially with climate change, you know, becoming a bigger and bigger issue. Right. 11:40 This is gonna be increasingly important. So they're not just making crops cheaper. They're helping make them stronger. They're helping make them stronger. Yeah. 11:47 Okay. And then finally, this project also contributed to the development of new multispectral devices Right. For real time crop monitoring. Yeah. So I guess, like, sensors Sensors that can measure different wavelengths of light and give you information about the health of the crop. 12:04 Okay. Cool. So, you know, a lot of different benefits came out of this project Yeah. Beyond just the pricing. It really highlights the power of data science to drive innovation in agriculture. 12:15 So if you had to kinda sum up the key outcomes and the value of this work, what would you say? Well, I think the the biggest outcome is that they were able to significantly reduce the number of hyperspectral bands needed to assess plant health, which as we discussed, leads to more cost effective sensors. And I think more broadly, this project demonstrates how data science can empower agricultural scientists and breeders to make more informed decisions, which ultimately leads to more sustainable and profitable farming. Yeah. So it's not just about, you know, making more money. 12:53 It's about It's about doing things in a smarter way. Doing things in a smarter way. Yeah. Yeah. Okay. 12:59 Cool. So as we wrap up here, what do you think our listeners should take away from this? I think the biggest takeaway is that by combining different analytical techniques, like economic modeling and machine learning, and by leveraging data automation, you can achieve some pretty amazing results. Yeah. And it's not just limited to agriculture. 13:21 No. Not at all. Right. These same principles can be applied to all sorts of industries. Yeah. 13:26 Like, imagine using these techniques in health care Health care. Absolutely. Energy Got it. Or finance Or finance. The possibilities are endless. 13:34 The possibilities are endless. Yeah. Okay. Well, this has been a really fascinating deep dive. It has. 13:38 I feel like I've learned a lot Me too. About how data science is being used to revolutionize agriculture. Yeah. So thank you for listening in. Be sure to subscribe and follow Callberry on social media, and check out our website, www.callberry.ai, for more insights like this.
