Challenges in utilizing open and public data in an AgTech startup

Note: This is part 2 of 2 of our series on open and public data in agriculture.  Part 1 focused on other stakeholders’ thoughts on the subject and was originally published on


As with many industries, the potential of open and public data in agriculture is mouthwatering.  Promises of higher yields for the farmer, reduced system waste across the value chain, and greater transparency for the end-consumer are all hopeful results of this powerful data source.

But similar to many other industries, the potential of using open and public data in agriculture is far greater than its current reality with several challenges impeding its efficient adoption.  These challenges can be especially trying for an agricultural start-up looking for a jump-start to its product development.

As the recipient of a €100,000 grant from the Open Data Institute for our use of open and public data in agriculture, we feel that Farm Dog is in a unique position to share our thoughts on what these challenges are.

Challenge 1: Low resolution of open and public data

Agriculture is a hyperlocal industry.  Growing conditions can be different not only from field to field but also within a field itself.  This means that high-level agronomic information is rarely translatable into actionable insights for an individual farmer.

Let’s take an example from England.  The Department for Environment, Food, and Rural Affairs (DEFRA) has done a notable job of opening its data to the public (by June 2016, it will have released 8,000 agriculture-related datasets into the public domain!).  But can a farmer who grows tomatoes in Southport, England leverage this data?  To be even more concrete, can the available data be used to benchmark his phosphate fertilizer use to others, an important comparison to gauge one’s farming practices?

The answer is unfortunately no.  DEFRA’s relevant dataset does not provide the necessary resolution for our farmer.  DEFRA provides that all tilled crops in England use 29 kilograms per hectare of phosphates on average as well as specific data for 6 crop types (winterLow resolution of open and public data wheat, spring barley, winter barley, potatoes, oilseed rape, and sugar beet).  Our farmer, though, cannot utilize this data as tomatoes have quite different requirements than the crops for which data is available.

And when you realize that there are over 50 types of vegetables grown in England with a production value of ~£1.2 billion in 2014 alone, you beginto understand the shortcomings of today’s existing open and public data sources.

The tomato-phosphate scenario is not simply a convenient example.  The issue replicates itself across datasets and geographies.  Many open and public datasets are great for developing general analyses, but when it comes to helping out individual farmers, it is limited.

Challenge 2: Integrating open data with MyData

Given the low resolution of open data and the hyperlocal qualities of data in agriculture, open data’s value is multiplied when combined with MyData.  MyData is a type of open data that is collected about an individual and is then shared back with the individual (note that it is not necessarily open and public).

Open and public data_Challenge 2
Open data is not enough

Imagine a scenario in which a farmer in Texas has a network of in-field sensors collecting soil moisture, electrical conductivity, and soil temperature in real-time from his sweet onion field (state vegetable of Texas!).  This data is then relayed back to the farmer on a private dashboard.  That is an example of MyData.

So what is the challenge?  The challenge is that the full value of MyData requires integration with open and public data sources.  Taking our Texas farmer example, we would want to combine (1) the sensor data with (2) the farmer’s soil type data that we receive from the Natural Resources Conservation Service and (3) optimal growing conditions that we receive from the Texas A&M AgriLife Extension Service.   If we do so successfully, we can, in theory, calculate future irrigation and fertilizer requirements.

But each dataset was developed for its own initial purposes, with various levels of resolution, and differing data structures.  As such, a significant amount of clean-up, re-structuring, and logic must be applied to all the data sources in order to squeeze out their full value.

Challenge 3: Data ownership and privacy questions are potential inhibitors to mass adoption

Remember five years ago when we were all up in arms over how Google and Facebook were using our data?  Well, we have reached that stage now in agriculture, strongly in part due to the use of MyData.

There are two main reasons why farmers are worried about data privacy:

  1. Data ownership – each season’s worth of data that is collected increases the value of the farmer’s data by orders of magnitude. So what happens when a farmer wants to switch between data companies and needs to retain his historical data?  Does he have the right to transfer his data to the new provider or does the data remain with his old business partner?


  1. Data sharing – unlike Google or Facebook using your personal data to show you advertisements for your favorite band, a data company sharing a farmer’s data with third parties can have negative consequences.

Can a data company share a farmer’s data with competitors who can then improve their growing practices presumably at the farmer’s expense?  Farmer’s Business Network and FarmLink seem to have found the happy-medium among farmers

What about trading on the farmer’s data in the commodities market?  Farmobile has a unique solution for this issue

Or, at worst, openly sharing farmer data with regulatory agencies?

Data ownership and privacy

These are all open questions that are now being taken up by the agriculture industry in the United States.  Farm Dog sits on the Farm Bureau’s Data Privacy Advisory Board, an industry initiative to develop data ownership and privacy standards to answer just such questions (we will delve into the details in a future blog post).

It is important to understand that data privacy issues permeate the agriculture industry, even if you start with open data, and that there are initiatives underway to resolve them.


Open and public data is a tremendous asset for anyone looking to build solutions in the agriculture space.  In isolation, however, the value of the current offerings of open and public data is limited.  This data should be combined with additional data from the farmer in order to realize its full potential and to help us increase yields, reduce waste, and improve transparency.

To find out more about open and public data, we recommend:

The Open Data Institute

McKinsey & Company: Open Data: Unlocking innovation and performance with liquid information

To search open and public data sources in agriculture, we recommend:

USDA Open Data Catalog

DEFRA open data sets


Posts created 3

Leave a Reply

Your email address will not be published. Required fields are marked *

Related Posts

Begin typing your search term above and press enter to search. Press ESC to cancel.

Back To Top