Evaluation of the Initial Thematic Output from a Continuous Change-Detection Algorithm for Use in Automated Operational Land-Change Mapping by the U.S. Geological Survey

Bruce Pengra; Alisa L. Gallant; Zhe Zhu; Devendra Dahal

doi:10.3390/rs8100811

Evaluation of the Initial Thematic Output from a Continuous Change-Detection Algorithm for Use in Automated Operational Land-Change Mapping by the U.S. Geological Survey

Pengra, Bruce;Gallant, Alisa L.;Zhu, Zhe;Dahal, Devendra 2016-10-01 00:00:00 remote sensing Article Evaluation of the Initial Thematic Output from a Continuous Change-Detection Algorithm for Use in Automated Operational Land-Change Mapping by the U.S. Geological Survey 1 , 2 3 , † 1 Bruce Pengra *, Alisa L. Gallant , Zhe Zhu and Devendra Dahal SGT Inc., Contractor to the U.S. Geological Survey, Earth Resources Observation and Science Center, 47914 252nd St., Sioux Falls, SD 57198, USA; [email protected] U.S. Geological Survey, Earth Resources Observation and Science Center, 47914 252nd St., Sioux Falls, SD 57198, USA; [email protected] Inuteq., Contractor to the U.S. Geological Survey, Earth Resources Observation and Science Center, 47914 252nd St., Sioux Falls, SD 57198, USA; [email protected] * Correspondence: [email protected]; Tel.: +1-605-594-6865 † Current address: Department of Geosciences, MS 1053, Science Building 125, Texas Tech University, Lubbock, TX 79409, USA; [email protected]. Academic Editors: Parth Sarathi Roy and Prasad S. Thenkabail Received: 3 May 2016; Accepted: 19 September 2016; Published: 1 October 2016 Abstract: The U.S. Geological Survey (USGS) has begun the development of operational, 30-m resolution annual thematic land cover data to meet the needs of a variety of land cover data users. The Continuous Change Detection and Classiﬁcation (CCDC) algorithm is being evaluated as the likely methodology following early trials. Data for training and testing of CCDC thematic maps have been provided by the USGS Land Cover Trends (LC Trends) project, which offers sample-based, manually classiﬁed thematic land cover data at 2755 probabilistically located sample blocks across the conterminous United States. These samples represent a high quality, well distributed source of data to train the Random Forest classiﬁer invoked by CCDC. We evaluated the suitability of LC Trends data to train the classiﬁer by assessing the agreement of annual land cover maps output from CCDC with output from the LC Trends project within 14 Landsat path/row locations across the conterminous United States. We used a small subset of circa 2000 data from the LC Trends project to train the classiﬁer, reserving the remaining Trends data from 2000, and incorporating LC Trends data from 1992, to evaluate measures of agreement across time, space, and thematic classes, and to characterize disagreement. Overall agreement ranged from 75% to 98% across the path/rows, and results were largely consistent across time. Land cover types that were well represented in the training data tended to have higher rates of agreement between LC Trends and CCDC outputs. Characteristics of disagreement are being used to improve the use of LC Trends data as a continued source of training information for operational production of annual land cover maps. Keywords: Continuous Change Detection and Classiﬁcation; USGS Land Cover Trends; training data; Landsat; high-resolution imagery; land cover mapping 1. Introduction Mapping land cover and monitoring land cover change are important for a variety of societal and scientiﬁc purposes, including land management, natural resource management, ecological studies, sustainable development, climate modeling, urban planning, habitat monitoring, and many others [1–5]. The U.S Geological Survey (USGS) is moving forward with a Land Change Monitoring, Remote Sens. 2016, 8, 811; doi:10.3390/rs8100811 www.mdpi.com/journal/remotesensing Remote Sens. 2016, 8, 811 2 of 33 Assessment, and Projection (LCMAP) initiative to develop an expanded operational capacity for land cover mapping and monitoring to support these needs. One goal of LCMAP is to provide high temporal and moderate spatial resolution land cover and land change products, including annual thematic land cover at 30 m resolution (30 30 m pixels). The Continuous Change Detection and Classiﬁcation (CCDC) algorithm [6] was developed to support continuous monitoring with Landsat data to take advantage of the multi-decadal Landsat archive housed by the USGS and is expected to play a central role in LCMAP mapping and monitoring activities. CCDC will be used to generate annual thematic land cover maps, with a class legend generally based on Anderson Level 1 categories of land cover previously adopted by the USGS Land Cover Trends project [7]. The USGS Land Cover LC Trends project plays an integral role in the development of the current capability for continuous monitoring by providing a reliable, consistent land cover product and related change assessments [8–10]. LC Trends data were generated through manual interpretation and were developed for the nominal years of 1973, 1980, 1986, 1992, and 2000 [7]. These data offer a basis for both training and initial testing of CCDC land cover classiﬁcation. Our objective was to conduct a comparison between output from the CCDC algorithm and LC Trends data [11] to determine whether CCDC can produce comparable maps. We selected 14 Landsat path/row locations within the conterminous United States to capture a wide range of land cover types and mapping challenges. CCDC currently operates on Landsat data collected since the operation of the Thematic Mapper sensor [6], launched in 1982, so we compared CCDC map output with LC Trends data for 1992 and 2000. Our goal was to determine if results were consistent across time, space, and thematic classes and gain insight into the use of LC Trends data for eventual operational training of annual land cover maps with CCDC. The results we present do not provide a statistical description of error in the CCDC Land Cover maps; they provide levels of agreement with maps generated from the LC Trends project and characterize features associated with common categories of disagreement between LC Trends and CCDC maps. The aim of our assessment is to inform reﬁnement of the CCDC algorithm’s approach for deriving annual land cover maps and provide internal information on data inputs and work ﬂows. 2. Materials and Methods 2.1. Data 2.1.1. Land Cover Trends The LC Trends project used analyst interpretation of Landsat data for sample blocks of 10 km 10 km (for most parts of the United States) or 20 km 20 km (for a few areas) to characterize land cover and land cover change [7]. Land cover map dates were nominally 1973, 1980, 1986, 1992 and 2000, with actual dates of imagery varying for some samples because of clouds, poor data quality, or availability. The classiﬁcation system used 11 classes representing a mix of land cover and land use types that “paralleled” the Anderson Level I classiﬁcation system (Table 1) [12,13]. A national set of 2755 blocks was selected using a probabilistic sampling design with stratiﬁcation based on 1999-era Level III ecoregions deﬁned by the U.S. Environmental Protection Agency [8,14,15]. The LC Trends project mapped land use/land cover for 1992 as a baseline from which successive dates of land cover were mapped both forward and backward in time [16]. This initial baseline was created by starting with the National Land Cover Dataset [17] for 1992, collapsing the more detailed classes to the LC Trends class legend, then manually editing to improve local accuracy [16]. Changes from the 1992 baseline were identiﬁed and delineated manually using Landsat data, aided by aerial photographs and other ancillary data, to produce the land cover for successive dates. A minimum mapping unit of 60 60 m was used. Analysts conducted group reviews of each other ’s work for every sample [7,16]. Remote Sens. 2016, 8, 811 3 of 33 Table 1. Land use/land cover classes and descriptions used by the U.S. Geological Survey, Land Cover Trends project. Land Cover Class Description Open water Areas persistently covered with water, such as streams, canals, lakes, reservoirs, bays, and oceans. Areas of intensive use where much of the land is covered with structures or anthropogenic impervious surfaces (residential, commercial, industrial, roads, etc.) or less-intensive use where Developed (urban or the land cover matrix includes both vegetation and structures (low-density residential, otherwise built-up) recreational facilities, cemeteries, utility corridors, etc.), including any land functionally related to urban or built-up environments (parks, golf courses, etc.). Land in either a vegetated or an unvegetated state used for the production of food and ﬁber, Agriculture (cropland including cultivated and uncultivated croplands, haylands, pasture, orchards, vineyards, and and pasture) conﬁned livestock operations. Note that forest plantations are considered forests regardless of their use for wood products. Non-developed land where the tree cover density is >10%. Note that cleared forest land Forest and woodland (i.e., clearcuts) is mapped according to current cover (e.g., mechanically disturbed or grassland/shrubland). Non-developed land where cover by grasses, forbs, and/or shrubs predominates and tree-cover Grassland/shrubland density is <10%. Land where water saturation is the determining factor in soil characteristics, vegetation types, Wetland and animal communities. Wetlands can contain both water and vegetated cover. Areas with extractive mining activities that have a signiﬁcant surface expression, including Mines and quarries mining buildings, quarry pits, overburden, leach, evaporative features, tailings, or other related components. Land composed of soils, sand, or rocks where <10% of the area is vegetated. Does not include Barren land in transition recently cleared by disturbance. Land in an altered, often unvegetated transitional state caused by disturbance from mechanical Mechanically disturbed means, as by forest clearcutting, earthmoving, scraping, chaining, reservoir drawdown, and other similar human-induced changes. Land in an altered, often unvegetated transitional state caused by disturbance from Non-mechanically disturbed non-mechanical means, as by ﬁre, wind, ﬂood, animals, and other similar phenomena. Land where the accumulation of snow and ice does not completely melt during summer Snow and ice (e.g., alpine glaciers and snowﬁelds). Whereas LC Trends maps provided an established USGS land cover product, we were not aware of any quantitative assessment of its accuracy. We conducted an accuracy assessment for LC Trends blocks in 5 of our 14 path/rows that overlapped high numbers of blocks (112 of 186 blocks) and represented a range of landscape settings. Included in the assessment were path/rows 23/37, 27/27, 28/33, 43/34 and 46/27. Based on an assumed accuracy rate of 95% with a target standard error of 0.025, a 300 point sample size met criteria for estimating overall accuracy [18]. Sample locations were randomly selected from within all LC Trends pixels in the ﬁve path/rows. Analysts manually interpreted high resolution imagery available in Google Earth™, assigning a primary land cover label based on LC Trends class deﬁnitions. Secondary labels were assigned for pixels where mixed cover types made class assignment ambiguous. These reference data showed LC Trends to have an overall accuracy of 91% based on the primary label, and 99% accuracy when both the primary and secondary labels were considered. This conﬁrmed expectations that LC Trends data were of high accuracy and appropriate for use as a source of training data for CCDC. 2.1.2. CCDC Annual Land Cover We acquired time series Landsat data processed to surface reﬂectance [19,20] from the USGS Earth Resources Observation and Science (EROS) Center ’s Science Processing Architecture (ESPA) data system [21]. We included all archived data from Landsat 4, Landsat 5, Landsat 7, and Landsat 8 scenes with processing to the L1T standard [22] that had more than 20% clear observations (no cloud, cloud shadow, or snow). Clouds, cloud shadows, and snow were screened initially by ESPA with the Fmask algorithm (speciﬁcally, the CFmask implementation) [23,24], then further screened via a multitemporal cloud, cloud shadow, and snow detection algorithm called Tmask [25]. Remote Sens. 2016, 8, 811 4 of 33 Remote Sens. 2016, 8, 811 4 of 33 The CCDC algorithm uses all available Landsat data to estimate time series models and applies the models to predict future observations [6,26]. If the values of new observations are not within The CCDC algorithm uses all available Landsat data to estimate time series models and applies the predicted range for six consecutive observations, a break in the time series is ﬂagged and a new the models to predict future observations [6,26]. If the values of new observations are not within the time series model will be estimated when sufﬁcient observations are available. The time series predicted range for six consecutive observations, a break in the time series is flagged and a new models are composed of harmonic models [27,28] that capture annual cycles, seasonality, and a slope time series model will be estimated when sufficient observations are available. The time series models are composed of harmonic models [27,28] that capture annual cycles, seasonality, and a component. The breaks found in the time series provide change information, such as caused by land slope component. The breaks found in the time series provide change information, such as caused cover conversion. The coefﬁcients that deﬁne the time series cycles and slope, along with the root by land cover conversion. The coefficients that define the time series cycles and slope, along with mean square errors (RMSE), are used as inputs to a land cover classiﬁer (Figure 1). CCDC uses the the root mean square errors (RMSE), are used as inputs to a land cover classifier (Figure 1). CCDC Random Forest classiﬁer [29] to derive decision tree models that are used to generate land cover maps. uses the Random Forest classifier [29] to derive decision tree models that are used to generate land The time-series approach used by CCDC means that model trajectories can be “consulted” at any given cover maps. The time-series approach used by CCDC means that model trajectories can be time within the time series period to generate a map of land cover. “consulted” at any given time within the time series period to generate a map of land cover. Figure 1. Example of time series models estimated for all Landsat bands for Forest and Developed Figure 1. Example of time series models estimated for all Landsat bands for Forest and Developed (residential) land cover classes. During the transition between classes, where Continuous Change (residential) land cover classes. During the transition between classes, where Continuous Change Detection Detection and Classification (CCDC) did not fit a model, land cover was labeled “Disturbed”. and Classification (CCDC) did not fit a model, land cover was labeled “Disturbed”. BT = brightness BT = brightness temperature; ETM = Enhanced Thematic Mapper Plus; NIR = near-infrared; temperature; ETM = Enhanced Thematic Mapper Plus; NIR = near-infrared; OLS = ordinary least OLS = ordinary least squares; SR = surface reflectance; SWIR1 = shortwave infrared 1; squares; SR = surface reﬂectance; SWIR1 = shortwave infrared 1; SWIR2 = shortwave infrared 2; SWIR2 = shortwave infrared 2; TIRS = Thermal Infrared Sensor; TM = Thematic Mapper. TIRS = Thermal Infrared Sensor; TM = Thematic Mapper. As the CCDC algorithm is capable of providing land cover maps at any given date, we selected a fixed day of the year (1 July) to provide annual CCDC land cover maps for our assessment. Note As the CCDC algorithm is capable of providing land cover maps at any given date, we selected that following a break in the time series (six consecutive observations not within the predicted a ﬁxed day of the year (1 July) to provide annual CCDC land cover maps for our assessment. Note range), observations may fluctuate such that CCDC is unable to initiate a new time series model. that following a break in the time series (six consecutive observations not within the predicted range), During this time period we label the pixel as “Disturbed” (Figure 1). Thus, the annual land cover at observations may ﬂuctuate such that CCDC is unable to initiate a new time series model. During the 14 path/row locations was labeled as Disturbed whenever a pixel was unable to initiate a time this time period we label the pixel as “Disturbed” (Figure 1). Thus, the annual land cover at the series model over the 1 July anniversary date of the map. We used LC Trends data from 2000 as the pool to extract training data for the classifier. We 14 path/row locations was labeled as Disturbed whenever a pixel was unable to initiate a time series extracted training data from LC Trends blocks based on criteria developed from an analysis of best model over the 1 July anniversary date of the map. practices [30]. That analysis found that a total of 20,000 pixels distributed across classes in We used LC Trends data from 2000 as the pool to extract training data for the classiﬁer. proportion to the LC Trends class distribution was optimal, with a minimum of 600 pixels and a We extracted training data from LC Trends blocks based on criteria developed from an analysis maximum of 8000 pixels required for each class (note, if the total number of pixels for a given class of best practices [30]. That analysis found that a total of 20,000 pixels distributed across classes was less than 600, we extracted all available pixels). Selection of training pixels within each class in proportion to the LC Trends class distribution was optimal, with a minimum of 600 pixels and was random from all available pixels in that class. We also incorporated eight ancillary datasets for a maximum of 8000 pixels required for each class (note, if the total number of pixels for a given training and classification: digital elevation data and derivatives (aspect, slope, and position index), class was less than 600, we extracted all available pixels). Selection of training pixels within each a Wetland Potential Index (an index map generated for the 2006 U.S. National Land Cover Database class [3 was 1], N random ational fr Wet om lands all available Inventory [ pixels 32], and in that Soil S class. urvey Geo We also graincorporated phic [SSURGO] h eight ydric ancillary soils map datasets s [33]), and probability of cloud, snow, and water occurrence. The latter probabilities were derived for training and classiﬁcation: digital elevation data and derivatives (aspect, slope, and position from Fmask statistics and represented the percent of cloud (or snow or water) observations from all index), a Wetland Potential Index (an index map generated for the 2006 U.S. National Land Cover available historical observations in the Landsat archive. Database [31], National Wetlands Inventory [32], and Soil Survey Geographic [SSURGO] hydric soils maps [33]), and probability of cloud, snow, and water occurrence. The latter probabilities were derived Remote Sens. 2016, 8, 811 5 of 33 from Fmask statistics and represented the percent of cloud (or snow or water) observations from all available Remote Sens historical . 2016, 8observations , 811 in the Landsat archive. 5 of 33 The CCDC workﬂow creates multiple products, but the thematic land cover product was the only The CCDC workflow creates multiple products, but the thematic land cover product was the type we used for the current analysis. only type we used for the current analysis. 2.2. Methods of Land Cover Product Comparison 2.2. Methods of Land Cover Product Comparison The area of comparison was deﬁned by the footprint of LC Trends blocks in each of the The area of comparison was defined by the footprint of LC Trends blocks in each of the 14 14 path/rows (Figure 2). We re-projected the data from the LC Trends blocks from Albers Equal path/rows (Figure 2). We re-projected the data from the LC Trends blocks from Albers Equal Area, Area, NAD83, to the Universal Transverse Mercator system, WGS84, to correspond with the Landsat NAD83, to the Universal Transverse Mercator system, WGS84, to correspond with the Landsat time time series data. It was likely that the re-projection resulted in some degradation in spatial ﬁdelity; series data. It was likely that the re-projection resulted in some degradation in spatial fidelity; however however, visual exam , visual examination ination showed this to be showed this to be of of minor minor concern, given the concern, given the 60 60 × 60 m minimum 60 m minimum mapping unit applied by the LC Trends project and the level of spatial generalization inherent in mapping unit applied by the LC Trends project and the level of spatial generalization inherent in the the (manually delineated) LC Trends data. (manually delineated) LC Trends data. Figure 2. Land Cover (LC) Trends sample blocks (dot symbols) within the 14 path/row test locations. Figure 2. Land Cover (LC) Trends sample blocks (dot symbols) within the 14 path/row test locations. We created a map of per-pixel agreement for the area covered by LC Trends blocks within each W ofe the 14 created test a pa map th/rows, of perma -pixel tching the yea agreement r of for CCD the C l area and cover coveredoutput to the yea by LC Trends blocks r of source da within each ta of used for the LC Trends samples. Output layers were used to associate categories of disagreement the 14 test path/rows, matching the year of CCDC land cover output to the year of source data used for with specific locations and to characterize the conditions (land cover characteristics and data the LC Trends samples. Output layers were used to associate categories of disagreement with speciﬁc characteristics) typical of the main categories of confusion. We constructed a set of confusion locations and to characterize the conditions (land cover characteristics and data characteristics) typical matrices for all LC Trends blocks in each of the 14 path/rows and for each date of comparison of the main categories of confusion. We constructed a set of confusion matrices for all LC Trends blocks (Tables S1–S30). These were aggregated across path/rows into error matrices representing all pixels in each of the 14 path/rows and for each date of comparison (Tables S1–S30). These were aggregated in all path/rows for each of the nominal dates of comparison (Tables S31 and S32). These summary across path/rows into error matrices representing all pixels in all path/rows for each of the nominal error matrices, primarily the one for the nominal 2000 data comparison (Table 2), were the basis for dates of comparison (Tables S31 and S32). These summary error matrices, primarily the one for the identifying categories of disagreement covering the largest aerial extent as a fraction of the entire nominal 2000 data comparison (Table 2), were the basis for identifying categories of disagreement area of study and/or covering the largest area as a fraction of the respective LC Trends and CCDC covering classes to which the pi the largest aerial xels extent belonged. as a fraction of the entire area of study and/or covering the largest For each category of disagreement, we ranked the degree of concentration within the path/row area as a fraction of the respective LC Trends and CCDC classes to which the pixels belonged. locations and the relative rate of occurrence per the area covered by LC Trends blocks. We For each category of disagreement, we ranked the degree of concentration within the path/row developed vector layers delineating the areas of disagreement for the path/rows with the largest locations and the relative rate of occurrence per the area covered by LC Trends blocks. We developed concentrations or highest rates per area of comparison. We displayed the vector layers in Google vector layers delineating the areas of disagreement for the path/rows with the largest concentrations Earth™ and overlaid them on Landsat time series images and the thematic classifications of CCDC or highest rates per area of comparison. We displayed the vector layers in Google Earth™ and and LC Trends. We evaluated the land cover associated with each category of disagreement to overlaid them on Landsat time series images and the thematic classiﬁcations of CCDC and LC Trends. identify patterns of occurrence that might be used to improve the CCDC annual land cover accuracy. We evaluated the land cover associated with each category of disagreement to identify patterns of occurrence that might be used to improve the CCDC annual land cover accuracy. Remote Sens. 2016, 8, 811 6 of 33 Table 2. Confusion matrix for all pixels from all path/rows for the 2000 period. Circa 2000 Land Cover Trends Water Developed Disturbed Mining Barren Forest Grass/Shrub Agriculture Wetlands Ice & Snow Total Agreement Water 462,323 4832 1301 615 6992 11,638 8910 11,907 11,395 0 519,913 89% Developed 4652 705,417 22,002 4237 2245 113,665 35,972 128,682 5009 0 1,021,881 69% Disturbed 3137 9223 17,088 1298 969 30,110 162,546 107,305 7734 52 339,462 5% Mining 747 14,105 2829 26,515 1423 5251 4611 13,700 595 0 69,776 38% Barren 4274 833 51 20 120,469 11,994 24,645 2495 2176 9701 176,658 68% Circa 2000 CCDC Forest 19,043 118,687 169,391 2278 14,835 5,781,165 273,434 161,377 133,544 227 6,673,981 87% Grass/Shrub 7402 28,907 45,205 883 19,596 205,283 3,958,903 292,673 19,373 2451 4,580,676 86% Agriculture 15,242 98,464 9168 1801 437 116,467 161,455 5,891,790 55,527 0 6,350,351 93% Wetlands 13,611 2102 11,446 155 1522 112,575 25,212 28,568 554,469 0 749,660 74% Ice & Snow 5 0 0 0 5863 106 136 0 0 60,573 66,683 91% 530,436 982,570 278,481 37,802 174,351 6,388,254 4,655,824 6,638,497 789,822 73,004 overall 85.5% 87% 72% 6% 70% 69% 90% 85% 89% 70% 83% agreement Remote Sens. 2016, 8, 811 7 of 38 Remote Sens. 2016, 8, 811 7 of 33 3. Results 3. Results 3.1. Class Distribution 3.1. Class Distribution The distribution of classes was very similar for LC Trends and CCDC output when aggregated across the 14 path/rows (Figure 3). The largest differences in absolute area were in the Agriculture The distribution of classes was very similar for LC Trends and CCDC output when aggregated and Forest Classes. The LC Trends data resulted in just over 1.4% more of the map labeled as across the 14 path/rows (Figure 3). The largest differences in absolute area were in the Agriculture and Agriculture and just under 1.4% less of the map labeled as Forest than did CCDC. The area mapped Forest Classes. The LC Trends data resulted in just over 1.4% more of the map labeled as Agriculture per class by CCDC and LC Trends differed substantially more for most classes when compared and just under 1.4% less of the map labeled as Forest than did CCDC. The area mapped per class by within individual path/rows (Figure 4). This difference was most obvious for the Disturbed class in CCDC and LC Trends differed substantially more for most classes when compared within individual Arizona (36/38) and California (43/34) and for smaller classes such as Mining and Barren. However, path/rows (Figure 4). This difference was most obvious for the Disturbed class in Arizona (36/38) and some large differences in area mapped for more common classes occurred as well. For example, California (43/34) and for smaller classes such as Mining and Barren. However, some large differences CCDC mapped 6.7% more Forest in Washington (46/27) and 6.4% less Forest in Minnesota (27/27) in area mapped for more common classes occurred as well. For example, CCDC mapped 6.7% more than did LC Trends. LC Trends mapped 7.9% more Agriculture than CCDC in California (43/34) Forest in Washington (46/27) and 6.4% less Forest in Minnesota (27/27) than did LC Trends. LC Trends and 5.8% more in Kansas (28/33). mapped 7.9% more Agriculture than CCDC in California (43/34) and 5.8% more in Kansas (28/33). Figure 3. Class distribution for CCDC and LC Trends output over the total area of comparison. Figure 3. Class distribution for CCDC and LC Trends output over the total area of comparison. 3.2. Per-Pixel Agreement 3.2. Per-Pixel Agreement 3.2.1. Summary of Per-Pixel Agreement 3.2.1. Summary of Per-Pixel Agreement The summary confusion matrix for the circa 2000 data comparison showed 85.5% overall The summary confusion matrix for the circa 2000 data comparison showed 85.5% overall agreement between CCDC and LC Trends land cover (Table 2). Agreement between the largest agreement between CCDC and LC Trends land cover (Table 2). Agreement between the largest classes (Forest, Agriculture, and Grass/Shrub) ranged from a low of 85% producer’s agreement for classes (Forest, Agriculture, and Grass/Shrub) ranged from a low of 85% producer ’s agreement for the Grass/Shrub class to a high of 93% user’s agreement for the Agriculture class. The smaller and the Grass/Shrub class to a high of 93% user ’s agreement for the Agriculture class. The smaller and generally more fragmented Developed and Wetland classes showed 69% user’s and 72% producers generally more fragmented Developed and Wetland classes showed 69% user ’s and 72% producers agreement for the Developed class and 74% user’s and 70% producer’s agreement for the Wetland agreement for the Developed class and 74% user ’s and 70% producer ’s agreement for the Wetland class. The Barren class, which accounted for approximately 1% of the mapped area in both CCDC class. The Barren class, which accounted for approximately 1% of the mapped area in both CCDC and and LC Trends maps, had 68% user’s and 69% producer’s agreement. Mining, by far the smallest LC Trends maps, had 68% user ’s and 69% producer ’s agreement. Mining, by far the smallest class class in either classification (0.3% of area with CCDC and 0.2% of area with LC Trends), had low in either classiﬁcation (0.3% of area with CCDC and 0.2% of area with LC Trends), had low user ’s user’s agreement at 38%, but 70% producer’s agreement. Class agreement for Disturbed area was agreement at 38%, but 70% producer ’s agreement. Class agreement for Disturbed area was extremely extremely low (6.1% producer’s agreement, 5.0% user’s agreement). low (6.1% producer ’s agreement, 5.0% user ’s agreement). Remote Sens. 2016, 8, 811 8 of 33 Remote Sens. 2016, 8, 811 8 of 33 Figure 4. Comparison of class area mapped in each path/row by CCDC and LC Trends. Graph y-axes Figure 4. Comparison of class area mapped in each path/row by CCDC and LC Trends. Graph y- are scaled per class size to best display detail of comparison. axes are scaled per class size to best display detail of comparison. 3.2.2. Per-Pixel Agreement by Path/Row Overall agreement at individual path/row test locations across all years of comparison ranged from 75% to 98% (Table 3). For the circa 2000 period only two of the 14 path/rows had overall Remote Sens. 2016, 8, 811 9 of 33 agreement below 80%, 27/27 in Northern Minnesota (79.4%) and 16/40 in Florida (75.1%). Agreement was very similar for the 1992 and 2000 periods at most path/row locations. The two path/rows with the largest difference in agreement between the 1992 and 2000 periods were 36/38 in Arizona (97.6% overall agreement in 1992 dropped to 82.0% in 2000) and 43/34 in California (78.9% agreement in 1992 rose to 84.8% in 2000). Most of the difference was accounted for by disagreement in the Disturbed class; in both cases CCDC classiﬁed very large areas of Disturbed land in one of the two periods under comparison, but LC Trends did not. Error matrices for the four path/rows with the highest numbers of LC Trends blocks (Table 4) account for 53% of the total area across all 14 path/rows and provide representative examples of the variation of class distribution, overall accuracy, and class accuracy across all test sites. 3.2.3. Per-Pixel Agreement by Land Cover Class Forest Class Agreement The CCDC and LC Trends Forest classes had 90.5% producer ’s and 86.6% user ’s agreement overall. CCDC mapped 1.4% more forest area than did LC Trends. CCDC mapped less forest area than LC Trends in only two locations, Kansas and Arizona (Table 5), the two path/rows with the lowest Forest class producer ’s agreement, 60.5% and 54.7%, respectively. Forest class agreement varied widely across the test path/rows (Table 5). The two locations with the highest Forest class agreement for both 1992 and 2000 were New Hampshire/Vermont (13/29) and South Dakota (33/29). Tree cover occurred as largely unbroken expanses of mixed forest species (New Hampshire/Vermont) or managed, predominantly conifer, national forest (South Dakota). The eight LC Trends blocks in New Hampshire/Vermont had 97.4% producer ’s and 96.2% user ’s agreement between CCDC Forest and LC Trends Forest classes in 2000, and 97.7% producer ’s and 96.4% user ’s agreement in 1992. With just a single forested LC Trends block, South Dakota results showed 98.8% producer ’s and 93.3% user ’s agreement in 2000 and 98.8% producer ’s and 93.0% user ’s agreement in 1992. LC Trends blocks in the Washington path/row covered 18.7% of the total study area and 35% of the LC Trends Forest class. The LC Trends and CCDC Forest classes in this path/row exhibited 91.9% producer ’s and 86.2% user ’s agreement. The majority of LC Trends Forest pixels with which CCDC disagreed were classiﬁed as Developed by CCDC, accounting for 55.5% of the Forest class producer ’s disagreement. In most cases, high resolution images showed these areas to be associated with low-intensity development, where tree cover mixed with some houses and roads had been generalized to the Forest class by LC Trends interpreters. Of the roughly 14% of CCDC Forest pixels in Washington that disagreed with LC Trends pixels, 30.6% had been classiﬁed as Grass/Shrub and 27.9% had been classiﬁed as Developed by LC Trends. Most cases of the former occurred in forest harvest footprints, where early stages of forest regeneration were classiﬁed as Grass/Shrub by LC Trends. Cases of the latter were where land cover generally occurred as fragmented clusters of tree cover within a larger context of low-intensity development. The fragmentation of land cover created a high proportion of mixed pixels and edge pixels, where minor misregistration was likely a contributing factor to disagreement. Nevertheless, the majority of disagreement pixels were actually covered by trees, but often were generalized to the Developed class by LC Trends interpreters and mapped as Forest by CCDC. The Minnesota path/row (27/27) was dominated by tree cover, with much of it in woody wetland. The LC Trends and CCDC Forest classes here had 88.0% producer ’s agreement and 82.7% user ’s agreement. CCDC disagreement with LC Trends Forest pixels predominantly (82.6% of the time) occurred where CCDC classiﬁed the pixels as Wetland. LC Trends disagreed with CCDC Forest pixels approximately 17% of the time, often (15.6% of the time) labeling those pixels as Grass/Shrub or Wetland (40.2%). Remote Sens. 2016, 8, 811 10 of 33 Table 3. Overall agreement per path/row for 1992 and 2000 time periods. AR = Arkansas; AZ = Arizona; CA = California; CO = Colorado; FL = Florida; IL = Illinois; IN = Indiana; KS = Kansas; MN = Minnesota; MS = Mississippi; MO = Montana; ND = North Dakota; NH = New Hampshire; SD = South Dakota; VT = Vermont; WA = Washington. Location 13/29 NH/VT 16/40 FL 22/33 IL/IN 23/37 AR/MS 27/27 MN 28/33 KS 31/27 ND 33/29 SD 34/33 CO 35/32 CO 36/38 AZ 39/26 MT 43/34 CA 46/27 WA Overall agreement 2000 93.5% 75.1% 87.8% 86.4% 79.4% 86.2% 89.0% 93.9% 89.9% 89.8% 82.1% 91.7% 84.8% 80.4% Overall agreement 1992 93.8% 76.9% 87.7% 87.4% 79.2% 85.8% 87.7% 89.9% 90.0% 89.3% 97.6% 89.5% 78.9% 80.3% Table 4. Confusion matrices for the four path/row locations with the most total area of LC Trends data available for comparison. Table values are numbers of pixels. Washington 46/27 Trends/CCDC Agreement Circa 2000 Land Cover Trends Water Developed Disturbed Mining Barren Forest Grass/Shrub Agriculture Wetlands Ice & Snow Total Agreement Water 199,996 1947 3 127 6810 2565 7 1074 1838 0 214,367 93% Developed 2789 406,045 15,638 3503 2147 99,507 12,121 50,405 2971 0 595,126 68% Disturbed 107 2758 5782 486 43 4047 522 1032 146 52 14,975 39% Mining 335 8606 2189 7155 1293 1995 311 707 121 0 22,712 32% Barren 775 504 0 17 51,056 6582 4616 12 105 9701 73,368 70% Circa 2000 CCDC 6 6 Forest 7654 91,646 71,903 1154 12,690 2 10 100,308 19,146 23,476 227 2 10 86% Grass/Shrub 26 2217 38,632 112 9728 49,568 95,688 355 213 2451 198,990 48% Agriculture 1895 32,531 1355 155 414 8105 1394 200,775 3969 0 250,593 80% Wetlands 3577 1068 177 21 1364 6695 741 3777 25,504 0 42,924 59% Ice & Snow 5 0 0 0 5863 106 136 0 0 60,573 66,683 91% Total 217,159 547,322 135,679 12,730 91,408 2 10 215,844 277,283 58,343 73,004 Overall 80.4% Agreement Agreement 92% 74% 4% 56% 56% 92% 44% 72% 44% 83% California 43/34 Trends/CCDC Agreement Circa 2000 Land Cover Trends Water Developed Disturbed Mining Barren Forest Grass/Shrub Agriculture Wetlands Total Agreement Water 13,606 1405 1247 430 0 180 1537 2519 302 21,226 64% Developed 459 170,339 675 286 0 340 9954 44,284 121 226,458 75% Disturbed 156 4158 907 522 0 736 36,406 86,812 799 130,496 1% Mining 64 40 29 4817 0 569 1039 1337 53 7948 61% Circa 2000 CCDC Barren 33 0 22 0 2643 1203 3536 0 0 7437 36% Forest 186 362 13,228 52 60 275,941 27,007 218 12 317,066 87% Grass/Shrub 620 15,837 1124 470 72 31,394 518,106 53,435 1555 622,613 83% Agriculture 2099 31,677 1207 998 0 1775 28,236 1,379,735 1365 1,447,092 95% Wetlands 696 213 3 37 0 237 5766 2503 5590 15,045 37% Total 17,919 224,031 18,442 7612 2775 312,375 631,587 1,570,843 9797 Overall 84.8% Agreement 76% 76% 5% 63% 95% 88% 82% 88% 57% Agreement Remote Sens. 2016, 8, 811 11 of 33 Table 4. Cont. Kansas 28/33 Trends/CCDC Agreement Circa 2000 Land Cover Trends Water Developed Disturbed Mining Barren Forest Grass/Shrub Agriculture Wetlands Total Agreement Water 22,263 392 0 9 65 537 3190 1519 592 28,567 78% Developed 131 16,170 0 8 5 375 1552 5216 34 23,491 69% Disturbed 947 55 0 103 488 53 379 3839 337 6201 0% Mining 59 1024 0 1712 0 98 1902 6127 80 11,002 16% Circa 2000 CCDC Barren 2372 22 0 0 1564 52 157 1266 620 6053 26% Forest 289 758 4 6 91,109 16,012 19,539 1085 128,802 71% Grass/Shrub 3756 2756 32 161 21 28,292 1,088,300 159,808 718 1,283,844 85% Agriculture 1828 5994 15 93 3 25,175 82,423 1,215,580 1446 1,332,557 91% Wetlands 755 47 0 0 20 4920 329 2178 5853 14,102 42% Overall Total 32,400 27,218 51 2092 2166 150,611 1,194,244 1,415,072 10,765 86.2% Agreement 69% 59% 0% 82% 72% 60% 91% 86% 54% Agreement Minnesota 27/27 Trends/CCDC Agreement Circa 2000 Land Cover Trends Water Developed Disturbed Mining Barren Forest Grass/Shrub Agriculture Wetlands Total Agreement Water 77,559 104 15 30 0 1831 34 6 1649 81,228 95% Developed 162 4021 553 294 0 2485 416 1368 106 9405 43% Disturbed 59 391 1247 14 0 219 123 74 503 2630 47% Mining 74 93 101 7310 0 517 254 126 11 8486 86% Barren 7 4 0 2 65 6 1 1 3 89 73% Circa 2000 CCDC Forest 4105 1564 47,315 838 0 695,183 22,649 10,375 58,469 8 10 83% Grass/Shrub 3 3 2979 0 0 1957 6278 591 603 12,414 51% Agriculture 58 276 1252 147 0 9491 10,553 63,340 5229 90,346 70% Wetlands 3529 97 9217 97 0 78,168 8427 5115 283,024 73% 4 10 Total 85,556 6553 62,679 8732 65 789,857 48,735 80,996 349,597 Overall 79.4% Agreement 91% 61% 2% 84% 100% 88% 13% 78% 81% Agreement Remote Sens. 2016, 8, 811 12 of 33 Table 5. (a) Forest pixels mapped by LC Trends distributed across CCDC classes for each path/row location and (b) Forest pixels mapped by CCDC distributed across LC Trends classes for each path/row location. AR = Arkansas; AZ = Arizona; CA = California; CO = Colorado; FL = Florida; IL = Illinois; IN = Indiana; KS = Kansas; MN = Minnesota; MS = Mississippi; MO = Montana; ND = North Dakota; NH = New Hampshire; SD = South Dakota; VT = Vermont; WA = Washington. FOREST (a) Trends Forest Pixels/CCDC Classes NH/VT FL IL/IN AR/MS MN KS ND SD CO CO AZ MT CA WA path/row 13/29 16/40 22/33 23/37 27/27 28/33 31/27 33/29 34/33 35/32 36/38 39/26 43/34 46/27 Total Dist. Water 1508 26 3700 860 1831 537 28 0 232 169 2 0 180 2565 11,638 0.2% Developed 4118 2410 2216 741 2485 375 62 0 1058 86 267 0 340 99,507 113,665 1.8% Disturbed 106 632 74 2205 219 53 1 28 18 5 21,986 0 736 4047 30,110 0.5% Mining 1371 3 265 404 517 98 0 0 28 0 1 0 569 1995 5251 0.1% Barren 2154 0 258 774 6 52 0 0 838 37 90 0 1203 6582 11,994 0.2% Forest 940,857 34,988 268,587 352,455 695,183 91,109 5405 100,139 572,227 362,365 38,500 31 275,941 2,043,378 5,781,165 90.5% Grass/Shrub 4988 567 1207 201 1957 28,292 259 1024 36,342 39,980 9501 3 31,394 49,568 205,283 3.2% Agriculture 4843 1166 41,492 22,570 9491 25,175 1035 94 608 87 22 4 1,775 8,105 116,467 1.8% Wetland 5884 4506 6942 3833 78,168 4920 41 62 366 921 0 0 237 6,695 112,575 1.8% Ice & Snow 0 0 0 0 0 0 0 0 0 0 0 0 0 106 106 0.0% Total 965,829 44,298 324,741 384,043 789,857 150,611 6831 101,347 611,717 403,650 70,369 38 312,375 2,222,548 6,388,254 “Producer’s” 97.4% 79.0% 82.7% 91.8% 88.0% 60.5% 79.1% 98.8% 93.5% 89.8% 54.7% 81.6% 88.3% 91.9% 90.5% All p/r (b) CCDC Forest Pixels/Trends Classes Path/Row Water Dev Disturbed Mining Barren Forest Grass/Shrub Ag Wetland Ice & Snow Total “User’s” NH/VT 13/29 1619 8338 7185 1 570 940,857 4797 7269 7370 0 978,006 96.2% FL 16/40 290 1686 7996 13 0 34,988 6310 1218 7005 0 59,506 58.8% IL/IN 22/33 2868 9465 76 208 0 268,587 720 63,014 8875 0 353,813 75.9% AR/MS 23/37 1631 1077 20,914 3 5 352,455 482 35,609 24,344 0 436,520 80.7% MN 27/27 4105 1564 47,315 838 0 695,183 22,649 10,375 58,469 0 840,498 82.7% KS 28/33 289 758 4 6 91,109 16,012 19,539 1085 0 128,802 70.7% ND 31/27 154 67 0 0 0 5405 806 4541 674 0 11,647 46.4% SD 33/29 0 0 694 0 0 100,139 6530 1 1 0 107,365 93.3% CO 34/33 211 3720 8 3 1432 572,227 33,929 388 1395 0 613,313 93.3% CO 35/32 36 0 68 0 78 362,365 41,215 53 838 0 404,653 89.5% AZ 36/38 0 4 0 0 0 38,500 12,669 0 0 0 51,173 75.2% MT 39/26 0 0 0 0 0 31 6 0 0 37 83.8% CA 43/34 186 362 13,228 52 60 275,941 27,007 218 12 0 317,066 87.0% WA 46/27 7654 91,646 71,903 1154 12,690 2,043,378 100,308 19,146 23,476 227 2,371,582 86.2% Total 19,043 118,687 169,391 2278 14,835 5,781,165 273,434 161,377 133,544 227 6,673,981 86.6% Dist. 0.3% 1.8% 2.5% 0.0% 0.2% 86.6% 4.1% 2.4% 2.0% 0.0% All p/r Remote Sens. 2016, 8, 811 13 of 33 Remote Sens. 2016, 8, 811 13 of 38 Forest class agreement was less strong in Kansas (28/33; 60.5% producer ’s and 70.7% user ’s agreement for circa 2000) and Illinois/Indiana (22/33; 83.0% producer ’s and 75.9% user ’s agreement Forest class agreement was less strong in Kansas (28/33; 60.5% producer’s and 70.7% user’s for circa 2000). In both locations forest occurrence was more fragmented, resulting in a much higher agreement for circa 2000) and Illinois/Indiana (22/33; 83.0% producer’s and 75.9% user’s agreement proportion of edge pixels relative to interior area in forest stands (Figure 5). Almost all of the Forest for circa 2000). In both locations forest occurrence was more fragmented, resulting in a much higher class disagreement in these two path/rows occurred along boundaries between Forest and Agriculture proportion of edge pixels relative to interior area in forest stands (Figure 5). Almost all of the Forest or Forclass disagre est and Grass/Shr ement in ub. th Some ese two of the pat disagr h/rows oc eement curred along in these cases boundar was friom es between mixed pixels, Forest and containing Agriculture or Forest and Grass/Shrub. Some of the disagreement in these cases was from mixed some fraction of Agriculture or Grass/Shrub land cover. This was especially common in the Kansas pixels, containing some fraction of Agriculture or Grass/Shrub land cover. This was especially path/row, where forest occurred as long linear features coinciding with the moister, ﬁre-protected common in the Kansas path/row, where forest occurred as long linear features coinciding with the topography along stream courses. The large fraction of edge pixels would have increased the impact moister, fire-protected topography along stream courses. The large fraction of edge pixels would from image misregistration—and we observed some apparent minor image misregistration. Frequently, have increased the impact from image misregistration—and we observed some apparent minor the 60 m minimum mapping unit and greater generalization in the LC Trends data resulted in pixels image misregistration. Frequently, the 60 m minimum mapping unit and greater generalization in classiﬁed as Agriculture or Grass/Shrub where available high-resolution imagery revealed the actual the LC Trends data resulted in pixels classified as Agriculture or Grass/Shrub where available high- land cover to be tree cover. In some cases, disagreement appeared to result from misclassiﬁcation in resolution imagery revealed the actual land cover to be tree cover. In some cases, disagreement the LC Trends blocks. appeared to result from misclassification in the LC Trends blocks. Figure 5. Example of the fragmented tree cover typical of LC Trends blocks in the Illinois/Indiana Figure 5. Example of the fragmented tree cover typical of LC Trends blocks in the Illinois/Indiana (22/33) study location (upper left corner: 39.40966°N. lat., −87.04695°W. long.). Overlain vector layers (22/33) study location (upper left corner: 39.40966 N. lat., 87.04695 W. long.). Overlain vector layers identify locations of areas of Forest/Agriculture confusion, generally occurring at patch edges. identify locations of areas of Forest/Agriculture confusion, generally occurring at patch edges. Generally, even path/rows with only limited forest area provided meaningful information regarding the performance of the CCDC classification process in different landscapes. Forest Generally, even path/rows with only limited forest area provided meaningful information disagreement in Arizona (36/38), for example, occurred primarily as confusion between regarding the performance of the CCDC classiﬁcation process in different landscapes. Forest Grass/Shrub and Forest classes in the drier oak and juniper forested areas in the southernmost LC disagreement in Arizona (36/38), for example, occurred primarily as confusion between Grass/Shrub Trends block. Tree density and forest structure there varied, often along gradients of topography and Forest classes in the drier oak and juniper forested areas in the southernmost LC Trends block. and elevation and along drainages. The spatial transitions between Forest and Grass/Shrub classes Tree density and forest structure there varied, often along gradients of topography and elevation and are often gradual, and it was in these areas that disagreement with LC Trends classes tended to be along drainages. The spatial transitions between Forest and Grass/Shrub classes are often gradual, concentrated. and it was in these areas that disagreement with LC Trends classes tended to be concentrated. Forest class agreement was high at the sites in Colorado (34/33 and 35/32) and California For (43/est 34), wit class h 93 agr .5% p eement roducwas er’s and high 93at .3% user’ the sites s agre inem Colorado ent for 34(34/33 /33, 89.8and % pro 35/32) ducer’s and and 8 California 9.6% user’s agreement for 35/32, and 88.3% producer’s and 87.0% user’s agreement for 43/34. In both (43/34), with 93.5% producer ’s and 93.3% user ’s agreement for 34/33, 89.8% producer ’s and 89.6% Colorado path/rows over 90% of the disagreement was with the CCDC Grass/Shrub class, where user ’s agreement for 35/32, and 88.3% producer ’s and 87.0% user ’s agreement for 43/34. In both Colorado path/rows over 90% of the disagreement was with the CCDC Grass/Shrub class, where disagreement tended to be in areas where gradual transitions from denser to more diffuse tree cover Remote Sens. 2016, 8, 811 14 of 33 created ambiguous spatial boundaries between classes. This often occurred in locations of high relief, often where the difference between tree height and shrub height was slight. In California (43/34) 86.2% of the LC Trends Forest confusion was with CCDC Grass/Shrub class, and in these cases the circumstances were generally similar to those observed for the Colorado sites; the coarser minimum mapping unit and tendency toward generalization in the LC Trends data contributed to the disagreement with the CCDC results. Likewise, when LC Trends pixels disagreed with CCDC Forest pixels it was because LC Trends interpreters had mapped those pixels as Grass/Shrub 65.7% of the time; in these cases, the LC Trends minimum mapping unit and areal generalization accounted for more of the confusion than did the spatially transitional nature of forest cover in 43/34. Agriculture Class Agreement The CCDC and LC Trends Agriculture classes had good overall agreement (88.8% producer ’s and 92.8% user ’s agreement), with the main categories of disagreement being: (1) LC Trends Agriculture confused with CCDC Grass/Shrub; (2) CCDC Agriculture confused with LC Trends Grass/Shrub; and (3) LC Trends Agriculture confused with CCDC Forest. Five of the 14 path/row locations contained 86.4% of the area LC Trends interpreters had classiﬁed as Agriculture. Results for these ﬁve locations ranged from a low of 85.9% producer ’s and 91.2% user ’s agreement in the Kansas path/row (28/33) to a high of 93.2% producer ’s and 94.1% user ’s agreement in North Dakota (31/27) (Table 6a,b). Across all 14 path/rows, 23.7% of the area classiﬁed as agriculture by LC Trends was located in California (43/34), where we observed 87.8% producer ’s and 95.3% user ’s agreement. The single largest category of confusion was where LC Trends interpreters mapped pixels as Agriculture and CCDC mapped them as Grass/Shrub (Table 6a), accounting for 1.4% of the entire study area. The inverse case, where LC Trends interpreters mapped pixels as Grass/Shrub and CCDC mapped them as Agriculture, made up 0.8% of the study area (Table 6b). Both directions of confusion were concentrated in a few path/rows and were heavily concentrated in the Kansas location (28/33), which represented 54.6% of the confusion between LC Trends Agriculture and CCDC Grass/Shrub and 51.1% of the confusion between LC Trends Grass/Shrub and CCDC Agriculture. For the former case, examination of high resolution imagery indicated the actual land cover was Grass/Shrub approximately 80% of the time and ranged in use from rangeland to lightly managed pasture/hayland, with many patches being difﬁcult or impossible to distinguish from rangeland (Figure 6). Other areas where LC Trends labeled pixels as Agriculture and CCDC labeled them as Grass/Shrub occurred in California (23.7%), North Dakota (11.6%), and Montana (8.2%). The overwhelming majority of the confusion across all locations occurred in areas of Grass/Shrub, based on high resolution imagery in TM Google Earth , with some cases showing evidence of haying. A lesser factor contributing to cases of low producer ’s agreement in the Agriculture category was where CCDC labeled pixels as Forest that had been classiﬁed as Agriculture by LC Trends (see previous details in second paragraph under “Forest Class Agreement”). This confusion was heavily concentrated in a few path/rows, including Illinois/Indiana (22/33), Arkansas/Mississippi (23/37), and Kansas (28/33). Confusion between the LC Trends Agriculture class and CCDC Developed class, as well as the inverse, made up 1.1% of the total area of the study, with the latter representing the smaller fraction. Most of this disagreement occurred in the suburban fringes, where developed and agricultural lands were fragmented and intermingled with low-intensity development. These two categories of confusion were heavily concentrated in Washington (46/27: 39.2% of LC Trends Agriculture confused with CCDC Developed and 33.0% of the inverse case) and California (43/34: 34.4% of LC Trends Agriculture confused with CCDC Developed and 32.2% of the inverse case). LC Trends map generalization in these settings often labeled ﬁelds on the order of 250–500 m 250–500 m, which are used for hay and pasture, as Developed. Alternatively, LC Trends generalized buildings and roads in agricultural settings to be labeled as Agriculture, whereas CCDC separated those buildings and roads into the Developed class. Remote Sens. 2016, 8, 811 15 of 33 Table 6. (a) LC Trends circa 2000 Agriculture pixels distributed across CCDC classes for each path/row location and (b) CCDC circa 2000 Agriculture pixels distributed across LC Trends classes for each path/row location. AR = Arkansas; AZ = Arizona; CA = California; CO = Colorado; FL = Florida; IL = Illinois; IN = Indiana; KS = Kansas; MN = Minnesota; MS = Mississippi; MO = Montana; ND = North Dakota; NH = New Hampshire; SD = South Dakota; VT = Vermont; WA = Washington. AGRICULTURE (a) Trends Agriculture Pixels/CCDC Classes NH/VT FL IL/IN AR/MS MN KS ND SD CO CO AZ MT CA WA path/row 13/29 16/40 22/33 23/37 27/27 28/33 31/27 33/29 34/33 35/32 36/38 39/26 43/34 46/27 Total Dist. Water 2 111 3709 2055 6 1519 860 21 26 4 1 0 2519 1074 11,907 0.2% Developed 1342 2898 13,357 4391 1368 5216 3704 0 1096 232 208 181 44,284 50,405 128,682 1.9% Disturbed 8 164 2043 4742 74 3839 1583 13 557 187 1297 4954 86,812 1032 107,305 1.6% Mining 283 12 3831 464 126 6127 754 0 51 0 8 0 1337 707 13,700 0.2% Barren 16 0 197 946 1 1266 0 2 1 54 0 0 12 2495 0.0% Forest 7269 1218 63,014 35,609 10,375 19,539 4541 1 388 53 0 6 218 19,146 161,377 2.4% Grass/Shrub 173 401 4925 1869 591 159,808 34,003 4732 3073 4153 1077 24,078 53,435 355 292,673 4.4% Agriculture 17,042 16,428 1,173,954 657,549 63,340 1,215,580 719,378 8906 47,227 10,349 2,891 378,636 1,379,735 200,775 5,891,790 88.8% Wetland 59 238 4442 2841 5115 2178 6576 0 11 201 0 627 2503 3777 28,568 0.4% Ice & Snow 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0.0% Total 26,194 21,470 1,269,472 710,466 80,996 1,415,072 771,399 13,675 52,429 15,180 5536 408,482 1,570,843 277,283 6,638,497 “Producer’s” 65.1% 76.5% 92.5% 92.6% 78.2% 85.9% 93.3% 65.1% 90.1% 68.2% 52.2% 92.7% 87.8% 72.4% 88.8% All p/r (b) CCDC Agriculture Pixels/Trends Classes Path/Row Water Dev Disturbed Mining Barren Forest Grass/Shrub Ag Wetland Ice & Snow Total “User’s” NH/VT 13/29 35 1316 172 0 0 4843 65 17,042 113 0 23,586 72.3% FL 16/40 122 1936 622 6 0 1166 1181 16,428 550 0 22,011 74.6% IL/IN 22/33 3405 18,420 173 376 0 41,492 1380 1,173,954 8405 0 1,247,605 94.1% AR/MS 23/37 4617 5411 4372 7 7 22,570 1446 657,549 17,762 0 713,741 92.1% MN 27/27 58 276 1252 147 0 9491 10,553 63,340 5229 0 90,346 70.1% KS 28/33 1828 5994 15 93 3 25,175 82,423 1,215,580 1446 0 1,332,557 91.2% ND 31/27 1162 499 0 16 0 1035 19,354 719,378 15,383 0 756,827 95.1% SD 33/29 0 0 0 0 1 94 2061 8906 0 0 11,062 80.5% CO 34/33 14 320 0 3 5 608 1180 47,227 678 0 50,035 94.4% CO 35/32 5 1 0 0 1 87 1823 10,349 289 0 12,555 82.4% AZ 36/38 1 70 0 0 6 22 405 2891 0 0 3395 85.2% MT 39/26 1 13 0 0 0 4 9954 378,636 338 0 388,946 97.3% CA 43/34 2099 31,677 1207 998 0 1775 28,236 1,379,735 1365 0 1,447,092 95.3% WA 46/27 1895 32,531 1355 155 414 8105 1394 200,775 3969 0 250,593 80.1% Total 15,242 98,464 9168 1801 437 116,467 161,455 5,891,790 55,527 0 6,350,351 92.8% Dist. 0.2% 1.6% 0.1% 0.0% 0.0% 1.8% 2.5% 92.8% 0.9% 0.0% All p/r Remote Sens. 2016, 8, 811 16 of 33 Remote Sens. 2016, 8, 811 16 of 38 Figure 6. The red polygons delineate areas of pasture mapped as Agriculture by LC Trends and Figure 6. The red polygons delineate areas of pasture mapped as Agriculture by LC Trends and Grass/Shrub by CCDC. Grass/Shrub by CCDC. Grass/Shrub Class Agreement Grass/Shrub Class Agreement The Grass/Shrub class covered the third largest extent across the study area after Forest and The Grass/Shrub class covered the third largest extent across the study area after Forest and Agriculture, accounting for 22.7% of the LC Trends classification and 22.3% of the CCDC classification. Agriculture, accounting for 22.7% of the LC Trends classiﬁcation and 22.3% of the CCDC classiﬁcation. The Grass/Shrub class had slightly lower agreement than either Forest or Agriculture at 85.0% The Grass/Shrub class had slightly lower agreement than either Forest or Agriculture at 85.0% producer’s and 86.4% user’s agreement. For eight of the 10 path/row test areas where more than 1% producer ’s and 86.4% user ’s agreement. For eight of the 10 path/row test areas where more than 1% of of the area was classified as Grass/Shrub by LC Trends, we observed 82.0% to 94.5% producer’s the area was classiﬁed as Grass/Shrub by LC Trends, we observed 82.0% to 94.5% producer ’s agreement agreement and 74.3% to 98.4% user’s agreement (Table 7a,b). The remaining two path/rows and 74.3% to 98.4% user ’s agreement (Table 7a,b). The remaining two path/rows exhibited low exhibited low Grass/Shrub class agreement, including 12.9% producer’s and 50.6% user’s agreement Grass/Shrub class agreement, including 12.9% producer ’s and 50.6% user ’s agreement in Minnesota in Minnesota (27/27) and 44.3% user’s and 48.7% producer’s agreement in Washington (46/27). Both (27/27) and 44.3% user ’s and 48.7% producer ’s agreement in Washington (46/27). Both LC Trends and LC Trends and CCDC Grass/Shrub classes were most often confused with the counterpart CCDC classiGrass/Shr fication’s Agr ubiclasses culture cl wer ass e(r most efer to often the sconfused econd para with graph the un counterpart der “Agricultu classiﬁcation’s re Class Agree Agr men icultur t”). e class (refer Confusion be to the second tween the Gr paragraph ass/under Shrub an “Agricultur d Forest cla e s Class ses account Agreement”). ed for 2.3% of the entire area of comparison (1.3% LC Trends Grass/Shrub confused with CCDC Forest and 1.0% LC Trends Forest Confusion between the Grass/Shrub and Forest classes accounted for 2.3% of the entire area of confused with CCDC Grass/Shrub). In Washington (46/27) and Minnesota (27/27), 83.5% and 53.4%, comparison (1.3% LC Trends Grass/Shrub confused with CCDC Forest and 1.0% LC Trends Forest respectively, of LC Trends Grass/Shrub class disagreement was where CCDC had classified pixels confused with CCDC Grass/Shrub). In Washington (46/27) and Minnesota (27/27), 83.5% and 53.4%, as Forest. In both locations this disagreement was associated with regenerating forest following respectively, of LC Trends Grass/Shrub class disagreement was where CCDC had classiﬁed pixels as timber harvest. These patches were classified as Forest by CCDC typically within a year of harvest, Forest. In both locations this disagreement was associated with regenerating forest following timber but were considered Grass/Shrub by LC Trends, typically for seven or more years following harvest. harvest. These patches were classiﬁed as Forest by CCDC typically within a year of harvest, but were Grass/Shrub class agreement was higher in the two Colorado path/rows, including 88.7% considered Grass/Shrub by LC Trends, typically for seven or more years following harvest. producer’s and 86.8% user’s agreement in 34/33 and 91.2% producer’s and 86.8% user’s agreement Grass/Shrub class agreement was higher in the two Colorado path/rows, including 88.7% in 35/32. As before, the largest fraction of disagreement was confusion between the Forest and producer ’s and 86.8% user ’s agreement in 34/33 and 91.2% producer ’s and 86.8% user ’s agreement Remote Sens. 2016, 8, 811 17 of 33 in 35/32. As before, the largest fraction of disagreement was confusion between the Forest and Grass/Shrub classes. Areas where LC Trends interpreters classiﬁed pixels as Grass/Shrub and CCDC classiﬁed them as Forest accounted for 74.7% (34/33) and 82.9% (35/32) of the producer ’s disagreement. User ’s disagreement was also predominantly the result of confusion between Grass/Shrub and Forest; areas where CCDC had classiﬁed pixels as Grass/Shrub and LC Trends interpreters classiﬁed them as Forest accounted for 67.3% of the disagreement in 34/33 and 84.6% of the disagreement in 35/32 (see second-to-last paragraph under “Forest Class Agreement”). Developed Class Agreement The Developed class covered 4.8% of the LC Trends pixels and 5.0% of the CCDC pixels in the study area. The Developed classes generally occurred in more complex, fragmented land cover mosaics and had 71.8% producer ’s agreement and 69.0% user ’s agreement. The majority of confusion was with the Forest and Agriculture classes and was concentrated in a few path/row locations (Table 8). Confusion of Developed land with Forest was heavily concentrated in the Washington location (46/27), with 77.2% of cases of LC Trends Developed pixels classiﬁed as Forest by CCDC and 87.5% of cases of the CCDC Developed pixels classiﬁed as Forest by LC Trends. The generalization of land cover features for the LC Trends classiﬁcation often included pixels of pure tree canopy that occurred in the very complex land cover mosaic of low-intensity development around the Puget Sound area of 46/27. CCDC generally classiﬁed these pixels as Forest. The inverse disagreement, where CCDC Developed pixels were classiﬁed as Forest by LC Trends, likewise was often associated with the generalization of the LC Trends classiﬁcation, which had included areas of developed land within larger tracts of Forest. A similar fraction of the Developed/Forest confusion was not caused by the LC Trends generalization, but from mixed pixels in the very fragmented land cover around the Puget Sound. The fragmented land cover mosaic also increased opportunities for image misregistration to contribute to class confusion. The Agriculture class also accounted for a large fraction of the confusion between the CCDC and LC Trends Developed classes. CCDC classiﬁed 10.0% of LC Trends Developed pixels as Agriculture; conversely, LC Trends interpreters classiﬁed 12.6% of the CCDC Developed pixels as Agriculture. In both cases, the vast majority of that confusion was distributed across the Washington (46/27), California (43/34), and Illinois/Indiana (22/33) locations. Pixels where LC Trends interpreters had classiﬁed the land as Agriculture and CCDC had classiﬁed it as Developed were most often associated with land cover that, if deﬁned strictly by cover as opposed to land use or a mixed deﬁnition, was Developed, such as roads, clusters of farm buildings, low intensity residential development and a few commercial/industrial sites. In the California location there were also some cases of bare farmland being classiﬁed as Developed by CCDC. Wetland Class Agreement LC Trends and CCDC mapped 3.8% and 3.6% of the entire study extent as Wetland, respectively; with 70.2% producer ’s and 74.0% user ’s class agreement. CCDC classiﬁed 16.9% of LC Trends Wetland pixels as Forest, 7.0% as Agriculture, 2.5% as Grass/Shrub, and 1.4% as water (Table 9a). Of the pixels classiﬁed as Wetland by CCDC, LC Trends interpreters classiﬁed 15.0% as Forest, 3.8% as Agriculture, and 3.4% as Grass/Shrub (Table 9b). Where CCDC disagreed with the LC Trends Wetland class, it labeled those pixels Forest 57% of the time. Where LC Trends disagreed with the CCDC Wetland class it labeled those pixels Forest 58% of the time. Most Wetland confusion occurred in the path/rows with the most Wetland area (Florida, Illinois/Indiana, Arkansas/Mississippi, Minnesota, and Washington). Minnesota (27/27) had 44.3% of LC Trends and 51.7% of the CCDC Wetland pixels for the entire study extent. The vast majority of wetlands within this path/row were forested and, consequently, most of the disagreement was between Forest and Wetland classes. CCDC mapped 19% of LC Trends Wetland pixels in 27/27 to other classes, primarily forest (87.8%). LC Trends pixels disagreed with 27% of the CCDC Wetland pixels in 27/27, mapping 74.7% of them as Forest. Remote Sens. 2016, 8, 811 18 of 33 Table 7. (a) LC Trends circa 2000 Grass/Shrub pixels distributed across CCDC classes for each path/row location and (b) CCDC circa 2000 Grass/Shrub pixels distributed across LC Trends classes for each path/row location. AR = Arkansas; AZ = Arizona; CA = California; CO = Colorado; FL = Florida; IL = Illinois; IN = Indiana; KS = Kansas; MN = Minnesota; MS = Mississippi; MO = Montana; ND = North Dakota; NH = New Hampshire; SD = South Dakota; VT = Vermont; WA = Washington. GRASS/SHRUB (a) Trends Grass/Shrub Pixels/CCDC Classes NH/VT FL IL/IN AR/MS MN KS ND SD CO CO AZ MT CA WA path/row 13/29 16/40 22/33 23/37 27/27 28/33 31/27 33/29 34/33 35/32 36/38 39/26 43/34 46/27 Total Dist. Water 3 25 63 19 34 3190 2191 1572 51 205 11 2 1537 7 8910 0.2% Developed 75 716 293 14 416 1552 509 0 1148 604 8249 321 9954 12,121 35,972 0.8% Disturbed 15 974 6 31 123 379 92 2979 1534 446 117,790 1249 36,406 522 162,546 3.5% Mining 50 6 114 4 254 1902 283 0 628 18 2 0 1039 311 4611 0.1% Barren 98 0 0 0 1 157 0 7420 3849 2779 2189 0 3536 4616 24,645 0.5% Forest 4797 6310 720 482 22,649 16,012 806 6530 33,929 41,215 12,669 0 27,007 100,308 273,434 5.9% Grass/Shrub 4033 5173 2438 6480 6278 1,088,300 122,322 363,773 354,855 513,857 776,649 100,951 518,106 95,688 3,958,903 85.0% Agriculture 65 1181 1380 1446 10,553 82,423 19,354 2061 1180 1823 405 9954 28,236 1394 161,455 3.5% Wetland 117 495 42 87 8,427 329 2221 443 3076 2600 0 868 5766 741 25,212 0.5% Ice & Snow 0 0 0 0 0 0 0 0 0 0 0 0 0 136 136 0.0% Total 9253 14,880 5056 8563 48,735 1,194,244 147,778 384,778 400,250 563,547 917,964 113,345 631,587 215,844 4,655,824 “Producer’s” 43.6% 34.8% 48.2% 75.7% 12.9% 91.1% 82.8% 94.5% 88.7% 91.2% 84.6% 89.1% 82.0% 44.3% 85.0% All p/r (b) CCDC Grass/Shrub Pixels/Trends Classes path/row Water Dev Disturbed Mining Barren Forest Grass/Shrub Ag Wetland Ice & Snow Total “User’s” NH/VT 13/29 2 334 1178 6 4988 4033 173 46 0 10,760 37.5% FL 16/40 41 133 748 0 0 567 5173 401 677 0 7740 66.8% IL/IN 22/33 175 424 12 26 0 1207 2438 4925 362 0 9569 25.5% AR/MS 23/37 15 4 394 201 6480 1869 655 0 9618 67.4% MN 27/27 3 3 2979 0 0 1957 6278 591 603 0 12,414 50.6% KS 28/33 3756 2756 32 161 21 28,292 1,088,300 159,808 718 0 1,283,844 84.8% ND 31/27 2382 327 5 4 0 259 122,322 34,003 5395 0 164,697 74.3% SD 33/29 119 0 67 0 5838 1024 363,773 4732 2 0 375,555 96.9% CO 34/33 182 5248 1 81 3149 36,342 354,855 3073 5931 0 408,862 86.8% CO 35/32 10 15 33 0 712 39,980 513,857 4153 2355 0 561,115 91.6% AZ 36/38 47 1581 0 29 70 9501 776,649 1077 0 0 788,954 98.4% MT 39/26 24 28 0 0 0 3 100,951 24,078 861 0 125,945 80.2% CA 43/34 620 15,837 1124 470 72 31,394 518,106 53,435 1555 0 622,613 83.2% WA 46/27 26 2217 38,632 112 9728 49,568 95,688 355 213 0 196,539 48.7% Total 7402 28,907 45,205 883 19,596 205,283 3,958,903 292,673 19,373 0 4,578,225 86.5% Dist. 0.2% 0.6% 1.0% 0.0% 0.4% 4.5% 86.5% 6.4% 0.4% 0.0% All p/r Remote Sens. 2016, 8, 811 19 of 33 Table 8. (a) LC Trends Developed pixels distributed across CCDC classes for each path/row location and (b) CCDC Developed pixels distributed across LC Trends classes for each path/row. AR = Arkansas; AZ = Arizona; CA = California; CO = Colorado; FL = Florida; IL = Illinois; IN = Indiana; KS = Kansas; MN = Minnesota; MS = Mississippi; MO = Montana; ND = North Dakota; NH = New Hampshire; SD = South Dakota; VT = Vermont; WA = Washington. DEVELOPED (a) Trends Developed Pixels/CCDC Classes NH/VT FL IL/IN AR/MS MN KS ND SD CO CO AZ MT CA WA path/row 13/29 16/40 22/33 23/37 27/27 28/33 31/27 33/29 34/33 35/32 36/38 39/26 43/34 46/27 Total Dist. Water 15 227 605 71 104 392 37 0 23 0 6 0 1405 1947 4832 0.5% Developed 6795 32,393 44,060 5214 4021 16,170 3294 0 12,553 234 3471 828 170,339 406,045 705,417 71.8% Disturbed 23 283 205 97 391 55 10 0 24 0 1218 1 4158 2758 9223 0.9% Mining 1140 72 2553 117 93 1024 16 0 439 0 5 0 40 8606 14,105 1.4% Barren 68 0 10 20 4 22 0 0 1 0 204 0 0 504 833 0.1% Forest 8338 1686 9465 1077 1564 758 67 0 3720 0 4 0 362 91,646 118,687 12.1% Grass/Shrub 334 133 424 4 3 2756 327 0 5248 15 1581 28 15,837 2217 28,907 2.9% Agriculture 1316 1936 18,420 5411 276 5994 499 0 320 1 70 13 31,677 32,531 98,464 10.0% Wetland 81 466 78 18 97 47 31 0 2 1 0 0 213 1068 2102 0.2% Ice & Snow 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0.0% Total 18,110 37,196 75,820 12,029 6553 27,218 4281 0 22,330 251 6559 870 224,031 547,322 982,570 “Producer’s” 37.5% 87.1% 58.1% 43.3% 61.4% 59.4% 76.9% na 56.2% 93.2% 52.9% 95.2% 76.0% 74.2% 71.8% All p/r (b) CCDC Developed Pixels/Trends Classes Path/Row Water Dev Disturbed Mining Barren Forest Grass/Shrub Ag Wetland Ice & Snow Total “User’s” NH/VT 13/29 41 6795 176 3 7 4118 75 1342 160 0 12,717 53.4% FL 16/40 559 32,393 1151 25 0 2410 716 2898 893 0 41,045 78.9% IL/IN 22/33 229 44,060 76 55 0 2216 293 13,357 301 0 60,587 72.7% AR/MS 23/37 162 5214 3729 1 1 741 14 4391 111 0 14,364 36.3% MN 27/27 162 4021 553 294 0 2485 416 1368 106 0 9405 42.8% KS 28/33 131 16,170 0 8 5 375 1552 5216 34 0 23,491 68.8% ND 31/27 23 3294 4 55 0 62 509 3704 130 0 7781 42.3% SD 33/29 0 0 0 0 0 0 0 0 0 0 0 na CO 34/33 11 12,553 0 2 2 1058 1148 1096 12 0 15,882 79.0% CO 35/32 0 234 0 0 0 86 604 232 113 0 1269 18.4% AZ 36/38 26 3471 0 5 83 267 8249 208 0 0 12,309 28.2% MT 39/26 60 828 0 0 0 0 321 181 57 0 1447 57.2% CA 43/34 459 170,339 675 286 0 340 9954 44,284 121 0 226,458 75.2% WA 46/27 2789 406,045 15,638 3503 2147 99,507 12,121 50,405 2971 0 595,126 68.2% Total 4652 705,417 22,002 4237 2245 113,665 35,972 128,682 5009 0 1,021,881 69.0% Dist. 0.5% 69.0% 2.2% 0.4% 0.2% 11.1% 3.5% 12.6% 0.5% 0.0% All p/r Remote Sens. 2016, 8, 811 20 of 33 Table 9. (a) LC Trends Wetland pixels distributed across CCDC classes for each path/row location and (b) CCDC Wetland pixels distributed across LC Trends classes for each path/row. AR = Arkansas; AZ = Arizona; CA = California; CO = Colorado; FL = Florida; IL = Illinois; IN = Indiana; KS = Kansas; MN = Minnesota; MS = Mississippi; MO = Montana; ND = North Dakota; NH = New Hampshire; SD = South Dakota; VT = Vermont; WA = Washington. WETLAND (a) Trends Wetland Pixels across CCDC Classes NH/VT FL IL/IN AR/MS MN KS ND SD CO CO AZ MT CA WA path/row 13/29 16/40 22/33 23/37 27/27 28/33 31/27 33/29 34/33 35/32 36/38 39/26 43/34 46/27 Total Dist. Water 804 276 1521 1132 1649 592 2710 0 85 486 0 0 302 1838 11,395 1.4% Developed 160 893 301 111 106 34 130 0 12 113 0 57 121 2971 5009 0.6% Disturbed 17 3056 49 469 503 337 186 0 2,105 9 0 58 799 146 7734 1.0% Mining 209 1 7 56 11 80 4 0 52 1 0 0 53 121 595 0.1% Barren 156 0 165 792 3 620 0 0 0 335 0 0 0 105 2176 0.3% Forest 7370 7005 8875 24,344 58,469 1085 674 1 1395 838 0 0 12 23,476 133,544 16.9% Grass/Shrub 46 677 362 655 603 718 5395 2 5931 2355 0 861 1555 213 19,373 2.5% Agriculture 113 550 8405 17,762 5229 1446 15,383 0 678 289 0 338 1365 3969 55,527 7.0% Wetland 18,188 33,424 47,659 84,884 283,024 5853 17,425 160 25,948 5463 0 1347 5590 25,504 554,469 70.2% Ice & Snow 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0.0% Total 27,063 45,882 67,344 130,205 349,597 10,765 41,907 163 36,206 9889 0 2661 9797 58,343 789,822 “Producer’s” 67.2% 72.8% 70.8% 65.2% 81.0% 54.4% 41.6% 98.2% 71.7% 55.2% na 50.6% 57.1% 43.7% 70.2% All p/r (b) CCDC Wetland Pixels/Trends Classes Path/Row Water Dev Disturbed Mining Barren Forest Grass/Shrub Ag Wetland Ice & Snow Total “User’s” NH/VT 13/29 881 81 271 9 5884 117 59 18,188 0 25,490 71.4% FL 16/40 499 466 1,571 0 0 4506 495 238 33,424 0 41,199 81.1% IL/IN 22/33 775 78 0 0 1 6942 42 4442 47,659 0 59,939 79.5% AR/MS 23/37 876 18 205 3833 87 2841 84,884 0 92,744 91.5% MN 27/27 3529 97 9,217 97 0 78,168 8427 5115 283,024 0 387,674 73.0% KS 28/33 755 47 0 0 20 4920 329 2178 5853 0 14,102 41.5% ND 31/27 1943 31 0 0 0 41 2221 6576 17,425 0 28,237 61.7% SD 33/29 0 0 0 0 0 62 443 0 160 0 665 24.1% CO 34/33 27 2 2 0 0 366 3076 11 25,948 0 29,432 88.2% CO 35/32 47 1 0 0 128 921 2600 201 5463 0 9361 58.4% AZ 36/38 0 0 0 0 0 0 0 0 0 0 0 na MT 39/26 6 0 0 0 0 0 868 627 1347 0 2848 47.3% CA 43/34 696 213 3 37 0 237 5766 2503 5590 0 15,045 37.2% WA 46/27 3577 1068 177 21 1364 6695 741 3777 25,504 0 42,924 59.4% Total 13,611 2102 11,446 155 1522 112,575 25,212 28,568 554,469 0 749,660 74.0% Dist. 1.8% 0.3% 1.5% 0.0% 0.2% 15.0% 3.4% 3.8% 74.0% 0.0% All p/r Remote Sens. 2016, 8, 811 21 of 33 Remote Sens. 2016, 8, x FOR PEER REVIEW 21 of 38 Much of the confusion between LC Trends Wetland and CCDC Forest pixels occurred along river Much of the confusion between LC Trends Wetland and CCDC Forest pixels occurred along river channels and other elongated features, as well as boundaries between wetland and forest classes where channels and other elongated features, as well as boundaries between wetland and forest classes where mixed pixels and minor misregistration may have been factors. This was less often the case with CCDC mixed pixels and minor misregistration may have been factors. This was less often the case with CCDC Wetland confusion with LC Trends Forest pixels, where areas of disagreement were clearly associated Wetland confusion with LC Trends Forest pixels, where areas of disagreement were clearly associated with a different interpretation of patches of land cover. In many cases it was difﬁcult to distinguish with a different interpretation of patches of land cover. In many cases it was difficult to distinguish conclusively between forest and forested wetland using visual interpretation of Landsat and high conclusively between forest and forested wetland using visual interpretation of Landsat and high resolution imagery in Google Earth™. Consequently, it was difﬁcult to determine with certainty what resolution imagery in Google Earth™. Consequently, it was difficult to determine with certainty what the the true land cover class should have been. We consulted data from the National Wetlands Inventory true land cover class should have been. We consulted data from the National Wetlands Inventory (NWI) (NWI) [32,34] for additional information on the occurrence of wetlands in the landscape (Figure 7). [32,34] for additional information on the occurrence of wetlands in the landscape (Figure 7). For pixels For pixels labeled by LC Trends as Wetland and by CCDC as Forest, the NWI favored the CCDC Forest labeled by LC Trends as Wetland and by CCDC as Forest, the NWI favored the CCDC Forest classification classiﬁcation 58.4% of the time (i.e., NWI did not classify these pixels as wetland). For pixels classiﬁed 58.4% of the time (i.e., NWI did not classify these pixels as wetland). For pixels classified by LC Trends as by LC Trends as Forest and by CCDC as Wetland, NWI again favored the CCDC interpretation, 60.3% Forest and by CCDC as Wetland, NWI again favored the CCDC interpretation, 60.3% of the time. of the time. Figure 7. Comparison of CCDC and LC Trends Wetland classes with National Wetlands Inventory (NWI) Figure 7. Comparison of CCDC and LC Trends Wetland classes with National Wetlands Inventory data and high resolution imagery. (NWI) data and high resolution imagery. The other concentration of wetlands and Wetland class disagreement was in Arkansas/Mississippi The other concentration of wetlands and Wetland class disagreement was in Arkansas/Mississippi (23/37), accounting for 16.5% of the LC Trends Wetland area and 12.4% of the CCDC Wetland area. (23/37), accounting for 16.5% of the LC Trends Wetland area and 12.4% of the CCDC Wetland area. Producer’s agreement was only 65.2%, with CCDC labeling 18.7% of LC Trends Wetland as Forest and Producer ’s agreement was only 65.2%, with CCDC labeling 18.7% of LC Trends Wetland as Forest and 13.6% of LC Trends Wetland as Agriculture (Table 9). The Wetland class user’s agreement was 91.5%, 13.6% of LC Trends Wetland as Agriculture (Table 9). The Wetland class user ’s agreement was 91.5%, with LC Trends labeling 4.1% of CCDC Wetland pixels as Forest and 3.1% as Agriculture. NWI data with LC Trends labeling 4.1% of CCDC Wetland pixels as Forest and 3.1% as Agriculture. NWI data favored the CCDC interpretation 72.4% of the time in pixels identified by LC Trends as Wetland and by favored the CCDC interpretation 72.4% of the time in pixels identiﬁed by LC Trends as Wetland and Remote Sens. 2016, 8, 811 22 of 33 Remote Sens. 2016, 8, x FOR PEER REVIEW 22 of 33 by CCDC as Forest. NWI favored the CCDC interpretation 67.8% of the time in pixels labeled by LC Trends as Wetland and by CCDC as Agriculture. CCDC as Forest. NWI favored the CCDC interpretation 67.8% of the time in pixels labeled by LC Trends as Wetland and by CCDC as Agriculture. Water Class Agreement Water Class Agreement The Water class had 87.2% producer ’s and 88.9% user ’s agreement. This class accounted for only 2.6% of the LC Trends mapped area and 2.5% of the CCDC mapped area. Five path/rows had less The Water class had 87.2% producer’s and 88.9% user’s agreement. This class accounted for only than 1% of area mapped as water by either CCDC or LC Trends. In path/rows with greater than 1% 2.6% of the LC Trends mapped area and 2.5% of the CCDC mapped area. Five path/rows had less than 1% water, low agreement was concentrated in Illinois/Indiana (22/33), Arkansas (23/37), and Kansas of area mapped as water by either CCDC or LC Trends. In path/rows with greater than 1% water, low (28/33) (Table 10). agreement was concentrated in Illinois/Indiana (22/33), Arkansas (23/37), and Kansas (28/33) (Table 10). The Washington location (46/27) had 41% of the total water mapped by LC Trends or CCDC and The Washington location (46/27) had 41% of the total water mapped by LC Trends or CCDC and had had the largest concentration (25%) of all disagreement, despite a producer ’s agreement of 92.1% and the largest concentration (25%) of all disagreement, despite a producer’s agreement of 92.1% and a user’s a user ’s agreement of 93.3% (Table 10). CCDC disagreement with LC Trends Water was split primarily agreement of 93.3% (Table 10). CCDC disagreement with LC Trends Water was split primarily among among Forest (44.6%), Wetland (20.8%), and Agriculture (11.0%). All three types of disagreement were Forest (44.6%), Wetland (20.8%), and Agriculture (11.0%). All three types of disagreement were concentrated along stream courses (Figure 8) and, to a lesser extent, along small inland water bodies concentrated along stream courses (Figure 8) and, to a lesser extent, along small inland water bodies and and coastal margins of the Puget Sound. Image registration was likely a contributing factor, as well as coastal margins of the Puget Sound. Image registration was likely a contributing factor, as well as mixed mixed pixels, minimum mapping unit, changing water levels, and, perhaps, shifts in water courses in pixels, minimum mapping unit, changing water levels, and, perhaps, shifts in water courses in some some streams. streams. LC Trends interpreters classiﬁed only 6.7% of CCDC Water pixels as other land cover types in the LC Trends interpreters classified only 6.7% of CCDC Water pixels as other land cover types in the Washington path/row, most often as Barren. This confusion occurred almost exclusively at edges of Washington path/row, most often as Barren. This confusion occurred almost exclusively at edges of water water bodies, often within the intertidal zone of the Puget Sound itself, and along stream channels bodies, often within the intertidal zone of the Puget Sound itself, and along stream channels where water where water level variation and changing sandbars and shorelines were likely contributing factors. level variation and changing sandbars and shorelines were likely contributing factors. Figure 8. Green polygons correspond with pixels LC Trends interpreters had classified as Forest and CCDC Figure 8. Green polygons correspond with pixels LC Trends interpreters had classiﬁed as Forest and classified as Water; blue polygons correspond with pixels LC Trends interpreters classified as Water and CCDC classiﬁed as Water; blue polygons correspond with pixels LC Trends interpreters classiﬁed as CCDC classified as Forest. Water and CCDC classiﬁed as Forest. Remote Sens. 2016, 8, 811 23 of 33 Table 10. (a) LC Trends Water pixels distributed across CCDC classes for each path/row location and (b) CCDC Water pixels distributed across LC Trends classes for each path/row. AR = Arkansas; AZ = Arizona; CA = California; CO = Colorado; FL = Florida; IL = Illinois; IN = Indiana; KS = Kansas; MN = Minnesota; MS = Mississippi; MO = Montana; ND = North Dakota; NH = New Hampshire; SD = South Dakota; VT = Vermont; WA = Washington. WATER (a) Trends Water/CCDC Classes NH/VT FL IL/IN AR/MS MN KS ND SD CO CO AZ MT CA WA path/row 13/29 16/40 22/33 23/37 27/27 28/33 31/27 33/29 34/33 35/32 36/38 39/26 43/34 46/27 Total Dist. Water 47,892 41,807 16,232 18,176 77,559 22,263 18,915 1188 3374 1248 57 10 13,606 199,996 462,323 87.2% Developed 41 559 229 162 162 131 23 0 11 0 26 60 459 2789 4652 0.9% Disturbed 2 131 291 1,194 59 947 39 168 0 8 35 0 156 107 3,137 0.6% Mining 16 0 148 44 74 59 1 0 6 0 0 0 64 335 747 0.1% Barren 56 0 529 459 7 2372 0 2 0 22 19 0 33 775 4274 0.8% Forest 1619 290 2868 1631 4105 289 154 0 211 36 0 0 186 7654 19,043 3.6% Grass/Shrub 2 41 175 15 3 3756 2382 119 182 10 47 24 620 26 7402 1.4% Agriculture 35 122 3405 4617 58 1828 1162 0 14 5 1 1 2099 1895 15,242 2.9% Wetland 881 499 775 876 3529 755 1943 0 27 47 0 6 696 3577 13,611 2.6% Ice & Snow 0 0 0 0 0 0 0 0 0 0 0 0 0 5 5 0.0% Total 50,544 43,449 24,652 27,174 85,556 32,400 24,619 1477 3825 1376 185 101 17,919 217,159 530,436 “Producer’s” 94.8% 96.2% 65.8% 66.9% 90.7% 68.7% 76.8% 80.4% 88.2% 90.7% 30.8% 9.9% 75.9% 92.1% 87.2% All p/r (b) CCDC Water Pixels/Trends Classes Path/Row Water Dev Disturbed Mining Barren Forest Grass/Shrub Ag Wetland Ice & Snow Total “User’s” NH/VT 13/29 47,892 15 6 0 0 1508 3 2 804 0 50,230 95.3% FL 16/40 41,807 227 5 0 0 26 25 111 276 0 42,477 98.4% IL/IN 22/33 16,232 605 4 9 11 3700 63 3709 1521 0 25,854 62.8% AR/MS 23/37 18,176 71 21 10 0 860 19 2055 1132 0 22,344 81.3% MN 27/27 77,559 104 15 30 0 1831 34 6 1649 0 81,228 95.5% KS 28/33 22,263 392 0 9 65 537 3190 1519 592 0 28,567 77.9% ND 31/27 18,915 37 0 0 0 28 2191 860 2710 0 24,741 76.5% SD 33/29 1188 0 0 0 92 0 1572 21 0 0 2873 41.4% CO 34/33 3374 23 0 0 1 232 51 26 85 0 3792 89.0% CO 35/32 1248 0 0 0 13 169 205 4 486 0 2125 58.7% AZ 36/38 57 6 0 0 0 2 11 1 0 0 77 74.0% MT 39/26 10 0 0 0 0 2 0 0 0 12 83.3% CA 43/34 13,606 1405 1247 430 0 180 1537 2519 302 0 21,226 64.1% WA 46/27 199,996 1947 3 127 6810 2565 7 1074 1838 0 214,367 93.3% Total 462,323 4832 1301 615 6992 11,638 8910 11,907 11,395 0 519,913 88.9% Dist. 88.9% 0.9% 0.3% 0.1% 1.3% 2.2% 1.7% 2.3% 2.2% 0.0% All p/r Remote Sens. 2016, 8, 811 24 of 33 The Illinois/Indiana path/row (22/33) had less than 5% of the total area mapped in water by both LC Trends and CCDC and had only 65.8% producer ’s agreement and 62.8% user ’s agreement. The Kansas location (28/33) had slightly more water (5.5% of CCDC total water and 6.1% of LC Trends total water), with a producer ’s agreement of 68.7% and a user ’s agreement of 77.9%. In both locations the vast majority of disagreement occurred along stream courses and shorelines of small water bodies. In these cases, LC Trends generalization and larger minimum mapping unit appeared to account for much of the confusion, and image registration may have contributed as well. In a few cases, LC Trends interpreters had classiﬁed agricultural ﬁelds as Water. In Kansas (28/33), confusion between LC Trends Water and CCDC Barren pixels occurred along the channel of the Kansas River, where some areas mapped as exposed sediment by CCDC were included in the LC Trends water class. The closest date of high resolution data in Google Earth™ (16 February 2002) agreed roughly as often with CCDC as with LC Trends. Different dates of high resolution data conﬁrmed the variability of the exposed sediment with water levels, erosion, and deposition. Barren Class Agreement The Barren class covered less than 1% of the entire mapped area in either classiﬁcation and had only 69.1% producer ’s agreement and 68.2% user ’s agreement—among the lowest across classes. The disagreement was heavily concentrated in Washington (46/27), South Dakota (33/29), and Colorado (35/32) (Table 11). Confusion between Barren and Forest was heavily concentrated in the Washington path/row, which had 54.9% of all LC Trends Forest pixels that had been identiﬁed as Barren by CCDC and 85.5% of all LC Trends Barren pixels that had been identiﬁed as Forest by CCDC. Pixels that LC Trends interpreters classiﬁed as Barren, but CCDC called Forest, occurred mostly in extreme terrain at or near treeline and snowline in the LC Trends blocks falling in the northern Cascades. In about half these cases, the pixels represented a mix of cover types, with some combination of bare rock, bare soil, terrain shadow, ground vegetation, and trees. Roughly a third of these pixels were where LC Trends interpreters had misclassiﬁed or generalized tree cover as Barren. At slightly lower elevations, there were cases of pixels identiﬁed by LC Trends as Barren and by CCDC as Forest that occurred along streams where mixed pixels were likely and where LC Trends interpreters had classiﬁed the bare soil and rock of the streams quite liberally, in some instances overlapping obvious tree cover. The inverse confusion, pixels classiﬁed by LC Trends interpreters as Forest and by CCDC as Barren, was highly concentrated in four sample blocks located at high elevations. Most of this confusion occurred in lightly to moderately vegetated rocky areas with few, if any, trees. Confusion between LC Trends Barren and CCDC Grass/Shrub classes occurred almost exclusively in the high-elevation terrain in just ﬁve sample blocks. Most of these pixels were lightly vegetated, with varying mixes of rock and soil in the pixel in most cases. Some additional Barren class disagreement occurred in South Dakota (33/29) and Colorado (34/33). In South Dakota the confusion was almost entirely between the Barren and Grass/Shrub classes. Most pixels were lightly to moderately vegetated, with components of exposed soil and rock. The generalization of LC Trends data appeared to have been a factor in many cases of confusion. In Colorado the majority of the disagreement was between the LC Trends Barren and CCDC Grass/Shrub classes. In most cases where LC Trends interpreters classiﬁed pixels as Barren and CCDC classiﬁed them as Grass/Shrub, the land cover was lightly vegetated, sometimes with scattered trees and rocky understory. Many of the disagreement pixels were at the boundaries of CCDC class patches. Remote Sens. 2016, 8, 811 25 of 33 Table 11. (a) LC Trends Barren pixels distributed across CCDC classes for each path/row location and (b) CCDC Barren pixels distributed across LC Trends classes for each path/row. AR = Arkansas; AZ = Arizona; CA = California; CO = Colorado; FL = Florida; IL = Illinois; IN = Indiana; KS = Kansas; MN = Minnesota; MS = Mississippi; MO = Montana; ND = North Dakota; NH = New Hampshire; SD = South Dakota; VT = Vermont; WA = Washington. BARREN (a) Trends Barren Pixels/Trends Classes NH/VT FL IL/IN AR/MS MN KS ND SD CO CO AZ MT CA WA path/row 13/29 16/40 22/33 23/37 27/27 28/33 31/27 33/29 34/33 35/32 36/38 39/26 43/34 46/27 Total Dist. Water 0 0 11 0 0 65 0 92 1 13 0 0 0 6810 6992 4.0% Developed 7 0 0 1 0 5 0 0 2 0 83 0 0 2147 2245 1.3% Disturbed 0 0 1 1 0 488 0 39 1 8 388 0 0 43 969 0.6% Mining 59 0 1 2 0 0 0 0 68 0 0 0 0 1293 1423 0.8% Barren 1826 0 159 562 65 1564 0 46,169 12,185 3275 965 0 2643 51,056 120,469 69.1% Forest 570 0 0 5 0 0 0 1432 78 0 0 60 12,690 14,835 8.5% Grass/Shrub 6 0 0 0 0 21 0 5838 3149 712 70 0 72 9728 19,596 11.2% Agriculture 0 0 0 7 0 3 0 1 5 1 6 0 0 414 437 0.3% Wetland 9 0 1 0 0 20 0 0 0 128 0 0 0 1364 1522 0.9% Ice & Snow 0 0 0 0 0 0 0 0 0 0 0 0 0 5863 5863 3.4% Total 2477 0 173 578 65 2166 0 52,139 16,843 4215 1512 0 2,775 91,408 174,351 “Producer’s” 9.0% na 91.9% 97.2% 100.0% 72.2% na 88.5% 72.3% 77.7% 63.8% na 95.2% 55.9% 69.1% All p/r (b) CCDC Barren Pixels/Trends Classes Path/Row Water Dev Disturbed Mining Barren Forest Grass/Shrub Ag Wetland Ice & Snow Total “User’s” NH/VT 13/29 56 68 18 1826 2154 98 16 156 0 4392 41.6% FL 16/40 0 0 0 0 0 0 0 0 0 0 0 na IL/IN 22/33 529 10 0 0 159 258 0 197 165 0 1318 12.1% AR/MS 23/37 459 20 11 1 562 774 0 946 792 0 3565 15.8% MN 27/27 7 4 0 2 65 6 1 1 3 0 89 73.0% KS 28/33 2372 22 0 0 1564 52 157 1266 620 0 6053 25.8% ND 31/27 0 0 0 0 0 0 0 0 0 0 0 na SD 33/29 2 0 0 0 46,169 0 7420 2 0 0 53,593 86.1% CO 34/33 0 1 0 0 12,185 838 3849 0 0 0 16,873 72.2% CO 35/32 22 0 0 0 3275 37 2779 1 335 0 6449 50.8% AZ 36/38 19 204 0 0 965 90 2189 54 0 0 3521 27.4% MT 39/26 0 0 0 0 0 0 0 0 0 0 0 na CA 43/34 33 0 22 0 2643 1203 3536 0 0 0 7437 35.5% WA 46/27 775 504 0 17 51,056 6582 4616 12 105 9701 73,368 69.6% Total 4274 833 51 20 120,469 11,994 24,645 2495 2176 9701 176,658 68.2% Dist. 2.4% 0.5% 0.0% 0.0% 68.2% 6.8% 14.0% 1.4% 1.2% 5.5% All p/r Remote Sens. 2016, 8, 811 26 of 33 Mining and Ice-Snow Classes Agreement The Mining class accounted for only 0.2% of the LC Trends mapped area and 0.3% of the CCDC mapped area. Mining class producer ’s agreement across all path/rows was 70.1%. However, CCDC mapped 85% more mining pixels than did LC Trends, and user ’s agreement across all path/rows was only 38.0%. Confusion between developed and mining classes was the largest category of disagreement. Over 20% of the CCDC Mining pixels were classiﬁed as Developed by LC Trends, and 11.2 % of the LC Trends Mining pixels were classiﬁed as Developed by CCDC. Another 19.6% of the CCDC Mining pixels were mapped as Agriculture by LC Trends. The lowest rate of producer ’s agreement (56.2%) was in Washington (46/27) (Table 12a), which accounted for 83% of the confusion between LC Trends Mining pixels and CCDC Developed pixels. The Mining class user ’s agreement for 46/27 was only 31.5% (Table 12b), with over 60% of all confusion between CCDC Mining pixels and LC Trends Developed pixels occurring in that path/row. User ’s agreement for Mining was below 50% for all path/rows having more than 125 pixels of the CCDC Mining class, with the exceptions of California (43/34) (60.6%) and Minnesota (27/27) (86.1%). The Ice and Snow class accounted for only 0.4% of the LC Trends mapped area and 0.3% of the CCDC mapped area and only occurred in Washington (46/27). Producer ’s agreement was 83% and user ’s agreement was 90.8%, with confusion between the Ice and Snow and Barren classes accounting for most disagreement. Disturbed Class Agreement LC Trends and CCDC Disturbed classes agreed in only a small minority of cases (6.1% producer ’s agreement and 5.0% user ’s agreement). When all path/rows were summarized together, LC Trends interpreters mapped 21.9% more area as disturbed than did CCDC. Summarized by path/row the differences appear extreme (Figure 4 and Table 13a,b). In the Washington path/row, LC Trends interpreters mapped nine times more area as Disturbed, with LC Trends identifying 3.5% of area as Disturbed and CCDC identifying only 0.39% as Disturbed. In the California path/row, CCDC mapped seven times more area as Disturbed than did LC Trends. For Arizona, CCDC mapped 14.2% of the map as disturbed, but LC Trends did not identify any area as disturbed. Areas where LC Trends mapped Disturbed and CCDC disagreed were heavily concentrated, with 85% of the cases occurring in just three path/rows (Washington—46/27, Minnesota—27/27, and Arkansas—23/37). Almost all of this disagreement occurred in forest harvest footprints, which in many cases were several years old. Areas where CCDC mapped Disturbed and LC Trends disagreed were also highly concentrated, with over 85% occurring in just two path/rows (Arizona—36/38 and California—43/34). In Arizona 83% of this was where LC Trends had mapped Grass/Shrub and CCDC mapped Disturbed. In California most of the LC Map disagreement with CCDC Disturbed was classiﬁed as Agriculture (67%). Remote Sens. 2016, 8, 811 27 of 33 Table 12. (a) LC Trends Mining pixels distributed across CCDC classes for each path/row location and (b) CCDC Mining pixels distributed across LC Trends classes for each path/row. AR = Arkansas; AZ = Arizona; CA = California; CO = Colorado; FL = Florida; IL = Illinois; IN = Indiana; KS = Kansas; MN = Minnesota; MS = Mississippi; MO = Montana; ND = North Dakota; NH = New Hampshire; SD = South Dakota; VT = Vermont; WA = Washington. MINING (a) Trends Mining Pixels/CCDC Classes NH/VT FL IL/IN AR/MS MN KS ND SD CO CO AZ MT CA WA path/row 13/29 16/40 22/33 23/37 27/27 28/33 31/27 33/29 34/33 35/32 36/38 39/26 43/34 46/27 Total Dist. Water 0 0 9 10 30 9 0 0 0 0 0 0 430 127 615 1.6% Developed 3 25 55 1 294 8 55 0 2 0 5 0 286 3503 4237 11.2% Disturbed 0 3 135 1 14 103 29 0 5 0 0 0 522 486 1298 3.4% Mining 595 215 2822 379 7310 1712 364 0 981 57 108 0 4817 7155 26,515 70.1% Barren 0 0 0 1 2 0 0 0 0 0 0 0 0 17 20 0.1% Forest 1 13 208 3 838 6 0 0 3 0 0 0 52 1154 2278 6.0% Grass/Shrub 0 0 26 0 0 161 4 0 81 0 29 0 470 112 883 2.3% Agriculture 0 6 376 7 147 93 16 0 3 0 0 0 998 155 1801 4.8% Wetland 0 0 0 0 97 0 0 0 0 0 0 0 37 21 155 0.4% Ice & Snow 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0.0% Total 599 262 3631 402 8732 2092 468 0 1075 57 142 0 7612 12,730 37,802 “Producer’s” 99.3% 82.1% 77.7% 94.3% 83.7% 81.8% 77.8% NA 91.3% 100.0% 76.1% NA 63.3% 56.2% 70.1% All p/r (b) CCDC Mining Pixels/Trends Classes Path/Row Water Dev Disturbed Mining Barren Forest Grass/Shrub Ag Wetland Ice & Snow Total “User’s” NH/VT 13/29 16 1140 46 595 59 1371 50 283 209 0 3769 15.8% FL 16/40 0 72 129 215 0 3 6 12 1 0 438 49.1% IL/IN 22/33 148 2553 76 2822 1 265 114 3831 7 0 9817 28.7% AR/MS 23/37 44 117 259 379 2 404 4 464 56 0 1729 21.9% MN 27/27 74 93 101 7310 0 517 254 126 11 0 8486 86.1% KS 28/33 59 1024 0 1712 0 98 1902 6127 80 0 11,002 15.6% ND 31/27 1 16 0 364 0 0 283 754 4 0 1422 25.6% SD 33/29 0 0 0 0 0 0 0 0 0 0 0 na CO 34/33 6 439 0 981 68 28 628 51 52 0 2253 43.5% CO 35/32 0 0 0 57 0 0 18 0 1 0 76 75.0% AZ 36/38 0 5 0 108 0 1 2 8 0 0 124 87.1% MT 39/26 0 0 0 0 0 0 0 0 0 0 0 na CA 43/34 64 40 29 4817 0 569 1039 1337 53 0 7948 60.6% WA 46/27 335 8606 2189 7155 1293 1995 311 707 121 0 22,712 31.5% Total 747 14,105 2829 26,515 1423 5251 4611 13,700 595 0 69,776 38.0% Dist. 1.1% 20.2% 4.1% 38.0% 2.0% 7.5% 6.6% 19.6% 0.9% na All p/r Remote Sens. 2016, 8, 811 28 of 33 Table 13. (a) LC Trends Disturbed pixels distributed across CCDC classes for each path/row location and (b) CCDC Disturbed pixels distributed across LC Trends classes for each path/row. AR = Arkansas; AZ = Arizona; CA = California; CO = Colorado; FL = Florida; IL = Illinois; IN = Indiana; KS = Kansas; MN = Minnesota; MS = Mississippi; MO = Montana; ND = North Dakota; NH = New Hampshire; SD = South Dakota; VT = Vermont; WA = Washington. DISTURBED (a) Trends Disturbed Pixels/CCDC Classes NH/VT FL IL/IN AR/MS MN KS ND SD CO CO AZ MT CA WA path/row 13/29 16/40 22/33 23/37 27/27 28/33 31/27 33/29 34/33 35/32 36/38 39/26 43/34 46/27 Total Dist. Water 6 5 4 21 15 0 0 0 0 0 0 0 1247 3 1301 0.5% Developed 176 1151 76 3729 553 0 4 0 0 0 0 0 675 15,638 22,002 7.9% Disturbed 125 1828 70 7124 1247 0 1 4 0 0 0 0 907 5782 17,088 6.1% Mining 46 129 76 259 101 0 0 0 0 0 0 0 29 2189 2829 1.0% Barren 18 0 0 11 0 0 0 0 0 0 0 0 22 0 51 0.0% Forest 7185 7996 76 20,914 47,315 4 0 694 8 68 0 0 13,228 71,903 169,391 60.8% Grass/Shrub 1178 748 12 394 2979 32 5 67 1 33 0 0 1124 38,632 45,205 16.2% Agriculture 172 622 173 4372 1252 15 0 0 0 0 0 0 1207 1355 9168 3.3% Wetland 271 1,571 0 205 9217 0 0 0 2 0 0 0 3 177 11,446 4.1% Ice & Snow 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0.0% Total 9177 14,050 487 37,029 62,679 51 10 765 11 101 0 0 18,442 135,679 278,481 “Producer’s” 1.4% 13.0% 14.4% 19.2% 2.0% 0.0% 10.0% 0.5% 0.0% 0.0% na na 4.9% 4.3% 6.1% All p/r (b) CCDC Disturbed Pixels/Trends Classes path/row Water Dev Disturbed Mining Barren Forest Grass/Shrub Ag Wetland Ice & Snow Total “User’s” NH/VT 13/29 2 23 125 106 15 8 17 0 296 42.2% FL 16/40 131 283 1,828 3 0 632 974 164 3056 0 7,071 25.9% IL/IN 22/33 291 205 70 135 1 74 6 2043 49 0 2,874 2.4% AR/MS 23/37 1194 97 7124 1 1 2,205 31 4742 469 0 15,864 44.9% MN 27/27 59 391 1247 14 0 219 123 74 503 0 2,630 47.4% KS 28/33 947 55 0 103 488 53 379 3839 337 0 6,201 0.0% ND 31/27 39 10 1 29 0 1 92 1583 186 0 1,941 0.1% SD 33/29 168 0 4 0 39 28 2979 13 0 0 3,231 0.1% CO 34/33 0 24 0 5 1 18 1534 557 2105 0 4,244 0.0% CO 35/32 8 0 0 0 8 5 446 187 9 0 663 0.0% AZ 36/38 35 1218 0 0 388 21,986 117,790 1297 0 0 142,714 0.0% MT 39/26 0 1 0 0 0 0 1249 4954 58 0 6,262 0.0% CA 43/34 156 4158 907 522 0 736 36,406 86,812 799 0 130,496 0.7% WA 46/27 107 2758 5782 486 43 4047 522 1032 146 52 14,975 38.6% Total 3137 9223 17,088 1298 969 30,110 162,546 107,305 7734 52 339,462 5.0% Dist. 0.9% 2.7% 5.0% 0.4% 0.3% 8.9% 47.9% 31.6% 2.3% 0.0% All p/r Remote Sens. 2016, 8, 811 29 of 33 4. Discussion We undertook this analysis to assess the performance of a continuous change-detection algorithm for mapping thematic land cover across a variety of landscape settings and to evaluate the efﬁcacy of applying data from a national study of land cover, LC Trends, to train the classiﬁer in preparation for operational continuous monitoring of land change. We subjectively selected 14 path/row areas that offered different types of challenges for mapping land cover and made opportunistic use of an existing, high-quality land cover dataset to characterize results. We intended to beneﬁt both from the actual analysis as well as from gaining familiarity with the workﬂow that will be needed to support eventual operations. The number of sample blocks (and therefore area of comparison) available for each of the 14 test path/rows varied greatly because the samples were originally selected based on ecoregion strata for the LC Trends project, rather than on Landsat path/rows. For example, the Florida path/row (16/40) had only two sample blocks, accounting for only 1.1% of the total area in this study, but the Washington path/row (46/27) had 36 sample blocks and represented 18.7% of the study area. Our results do not provide a statistical description of either error in the CCDC land cover or of the relation between Land Cover LC Trends and CCDC annual land cover outside of the areas compared. The results will, however, feed back into understanding the suitability of LC Trends data as a source for training data and the types of confusion that may be introduced by the LC Trends classiﬁcation scheme, minimum mapping unit, level of mapping generalization, and contrasting interpretation approach. We found good consistency in map results across time periods for all but two (Arizona and California) study areas (Table 3). Rates of agreement between LC Trends and CCDC maps varied geographically (ranging from 75% to 94% in 2000 and 77% to 98% in 1992), but nine of 14 path/rows had rates exceeding 86% agreement (Table 3). At the class level, we observed that Forest, Agriculture, Grass/Shrub, and Water had the highest rates of agreement (all >87% for both producer ’s and user ’s agreement) between LC Trends and CCDC maps pooled across study areas, but typically showed greater rates of agreement in study areas where the classes occupied appreciable portions of the landscape within the LC Trends blocks. The most important ﬁnding was that CCDC’s automated, efﬁcient, and repeatable approach was able to reproduce results obtained through the LC Trends project’s lengthy manual image interpretation process 86% of the time without any post-classiﬁcation reﬁnement while using the existing LC Trends dataset to guide the selection of training data. We initially questioned the suitability of the LC Trends data for training CCDC’s classiﬁer, as LC Trends data followed a classiﬁcation scheme that included some land use characteristics in its class deﬁnitions, and the LC Trends methods relied on analyst interpretation and a 60 60 m minimum mapping unit (the combination of which resulted in spatial generalization of land cover patches). However, the good overall agreement of CCDC annual land cover with LC Trends maps across the 14 path/rows we studied suggests that the national set of LC Trends data can provide an adequate source of training information to enable CCDC to generate wall-to-wall thematic land cover for the conterminous United States for the 1985 to current Landsat record. We also note that in areas of disagreement, ancillary information indicated that CCDC often made the better choice of class labels. Conversely, where classes only made up a small portion of the LC Trends blocks in a given path/row we observed several issues. For training, a minimum of 600 pixels for any class was found to provide the best classiﬁcation result [30]. For several of the 14 path/rows the limited number of LC Trends blocks available did not provide class populations of 600 pixels for one or more classes (for example, see Table 10a for Montana and Arizona). Furthermore, where classes were represented in small fragmented patches, registration and spatial generalization were observed to be a potential problem due to selection of some pixels representing other-than-intended classes within the training data. These small patches also impacted the CCDC/LC Trends comparison process, sometimes measuring disagreement that was the result of misregistration rather than misclassiﬁcation. The effect of misregistration error on training data selection, which had resulted from reprojection of the LC Remote Sens. 2016, 8, 811 30 of 33 Trends blocks from Albers to UTM, will be eliminated in operational LCMAP classiﬁcation when the input Landsat data will be processed to the USGS Albers/NAD83 grid, the native projection of the LC Trends blocks. To address the underrepresentation of small classes in the training data, we have since expanded the collection of training data to LC Trends blocks beyond the area being classiﬁed. Preliminary results suggest that this helps in meeting the 600 pixel minimum and better represents landscape variability near the edges of the area being classiﬁed. Collection of training pixels from within LC Trends blocks available in a window including and surrounding the area being classiﬁed has so far produced superior classiﬁcation for the Puget Sound, our initial test ground. Another key ﬁnding was the incompatibility of the methods by which LC Trends and CCDC interpreted the Disturbed class. For example, LC Trends land cover was mapped at intervals of 6 to 8 years, and interpreters showed a strong tendency to label pixels as disturbed long after the actual event had occurred to make certain the disturbance was recorded. CCDC only labeled disturbance for the brief intervals in which a time series model could not be ﬁtted following an abrupt change in land cover, and only if this interval overlapped the deﬁned anniversary date of 1 July. This interval often lasted for a period of months, rather than years. CCDC products will be generated annually, and a new formulation being evaluated labels Disturbance in the annual land cover map regardless of the date the change occurred within the year, overcoming the problem posed by selecting a speciﬁc anniversary date to survey for disturbance and removing much of the disagreement between CCDC and Trends Disturbance classes. Cumulating change across multiple annual maps output by CCDC then will produce results more comparable with those mapped by LC Trends interpreters across their multiyear mapping intervals. A second source of incompatibility between the Disturbance classes resulted from CCDC’s sensitivity to change, including changes in land surface condition where the land cover type did not actually change. For example, sequences such as Grass/Shrub to Disturbed to Grass/Shrub were observed in Arizona and California, where the Disturbed interval was apparently caused by multiyear wet or dry periods that created a measurable shift in vegetation response with no removal of vegetation cover or change in vegetation type. Such changes in vegetation condition were not recorded by LC Trends. There were two factors that led to CCDC identifying these changes in vegetation condition as “disturbance”. First, changes in condition caused legitimate breaks in the time series response trajectories, but there was no means to distinguish breaks caused by shifts in condition with breaks caused by changes in cover type. Recent reﬁnements to the CCDC algorithm are incorporating steps to ﬁlter changes in land cover condition from changes in type so that the latter can be better isolated for mapping changes in thematic cover. Second, unlike other class types, the Disturbance class was not trained for classiﬁcation with Random Forest; it was instead deﬁned by breaks in the time series models, as described in Section 2.1.2 (see also Figure 1). Training data now are being developed so that Random Forest can be used to classify disturbance directly. We found that the confusion between the Grass/Shrub class and the Agriculture class appeared to be due mostly to land use characteristics embedded in the LC Trends class deﬁnitions. The Trends Agriculture class deﬁnition includes “. . . cultivated and uncultivated croplands, haylands, [and] pasture . . . ,“ which in many cases led to capture by image interpreters of hayland and pasture that were spectrally indistinguishable from more natural grassland, particularly in Kansas (28/33). This land use distinction could be made because analysts employed a variety of contextual clues. CCDC generally mapped these areas of lightly managed hayland and pasture to the Grass/Shrub class. Redeﬁnition of the Agriculture class to exclude natural grassland that is lightly grazed or occasionally hayed might be suggested by this ﬁnding. Another ﬁnding related to how successional stages were handled for vegetation stands. The most obvious and widespread case we observed was the difference in how CCDC and LC Trends handled the recovery of forest following clearcut harvest. LC Trends interpreters distinguished early stages of this progression as Grass/Shrub before trees became dominant, then labeled the pixels as Forest once trees regained dominance. In comparison, CCDC ﬁt models to the full length of a time series Remote Sens. 2016, 8, 811 31 of 33 between the periods of abrupt change, then fed the coefﬁcients from these models to the classiﬁer. The coefﬁcients therefore represented a long-term forest trajectory, rather than the individual stand stages along the trajectory, and the resulting thematic label ended up as Forest. This ﬁnding prompted modifying the algorithm to run a separate classiﬁcation each year to enable the classiﬁer to focus on the evolving stand structure through time. The problem is further being addressed by developing training data indicative of early successional stages of forest recovery. We observed that the grass and shrub cover in these post-disturbance stands have different spectral characteristics from areas of perpetual (or long-term) grass/shrub and therefore require separate, representative training data. Results from our comparisons corroborated the expected difﬁculty in classifying woody wetlands or wetlands obscured by tree canopies. Our evaluation of Wetland class agreement was less certain in areas of Forested wetlands. Visual interpretation of forested wetlands was hindered where direct observation of ﬂooding conditions or speciﬁc wetland vegetation was obscured by tree cover. We compared areas of class confusion with data from the National Wetlands Inventory to augment our evaluation of Wetland class agreement. Although this informed our perspective, we note that NWI data also were used as an ancillary reference source by LC Trends analysts and are a component of the Wetland Potential Index (WPI) layer that was used as an ancillary input to the CCDC Random Forest classiﬁcation process. The WPI is a categorical ranking-index map generated based on convergence of evidence from information in the National Land Cover Database 2006 map [31], NWI data [32], and Soil Survey Geographic (SSURGO) hydric soils maps [33]. This complicates any conclusions we might draw from this comparison. CCDC annual land cover products should be evaluated with an independent dataset developed speciﬁcally to determine the accuracy of the thematic outputs, rather than only quantifying the level of agreement with another product. An independent evaluation is planned for the next stage of development towards operational continuous monitoring of land cover. 5. Conclusions We found 86% agreement between thematic land cover maps generated from two very different approaches applied with Landsat data, one based on manual interpretation of individual time periods spaced at 6- to 8-year intervals (LC Trends) and one based on automated interpretation of mathematical models constructed with dense time series of all available clear observations (CCDC). This agreement did not necessarily reﬂect the accuracy of the CCDC annual land cover maps, but rather the agreement between results from the two approaches encompassing the footprint of the 186 sample blocks used in this study. We observed consistency in results across time and across study areas of similar landscape types, and found relatively high levels of agreement for land cover classes that were well represented in the training data. Examination of the land cover associated with areas of disagreement suggested that, despite LC Trends classes being somewhat generalized and often incorporating contextual components of land use into class deﬁnitions, the annual land cover maps generated by CCDC generally differed from the LC Trends classiﬁcation in ways that were not problematic. For example, where LC Trends generalized highly fragmented and geometrically complex land cover features, CCDC adhered to the spatial detail represented in the spectral characteristics—often successfully. Whether CCDC was more accurate than the LC Trends data was not made clear by this analysis. Comparison with independent reference data will begin to address this question and is planned for the next stage of evaluation of CCDC annual land cover products. These efforts will help move the USGS towards operational implementation of a continuous monitoring capability. Supplementary Materials: The following are available online at www.mdpi.com/2072-4292/8/10/811/s1, Table S1: Trends/CCDC Agreement circa 2000, path 13 row 29, Table S2: Trends/CCDC Agreement circa 1992 path 13 row 29, Table S3: Trends/CCDC Agreement circa 2000, path 16 row 40, Table S4. Trends/CCDC Agreement circa 1992, path 16 row 40, Table S5. Trends/CCDC Agreement circa 2000, path 23 row 37, Table S6. Trends/CCDC Agreement circa 1992, path 23 row 37, Table S7. Trends/CCDC Agreement circa 2000, path 23 row 37, Table S8. Trends/CCDC Agreement circa 1992, path 23 row 37, Table S9. Trends/CCDC Agreement circa 2000, path 27 row 27, Table S10. Trends/CCDC Agreement circa 1992, path 27 row 27, Table S11. Trends/CCDC Remote Sens. 2016, 8, 811 32 of 33 Agreement circa 1986, path 27 row 27, Table S12. Trends/CCDC Agreement circa 2000, path 28 row 33, Table S13. Trends/CCDC Agreement circa 1992, path 28 row 33, Table S14. Trends/CCDC Agreement circa 2000, path 31 row 27, Table S15. Trends/CCDC Agreement circa 1992, path 31 row 27, Table S16. Trends/CCDC Agreement circa 2000, path 33 row 29, Table S17. Trends/CCDC Agreement circa 1992, path 33 row 29, Table S18. Trends/CCDC Agreement circa 2000, path 34 row 33, Table S19. Trends/CCDC Agreement circa 1992, path 34 row 33, Table S20. Trends/CCDC Agreement circa 2000, path 35 row 32, Table S21. Trends/CCDC Agreement circa 1992, path 35 row 32, Table S22. Trends/CCDC Agreement circa 2000, path 36 row 38, Table S23. Trends/CCDC Agreement circa 1992, path 36 row 38, Table S24. Trends/CCDC Agreement circa 2000, path 39 row 26, Table S25. Trends/CCDC Agreement circa 1992, path 39 row 26, Table S26. Trends/CCDC Agreement circa 2000, path 43 row 34, Table S27. Trends/CCDC Agreement circa 1992, path 43 row 34, Table S28. Trends/CCDC Agreement circa 2000, path 46 row 27, Table S29. Trends/CCDC Agreement circa 1992, path 46 row 27, Table S30. Trends/CCDC Agreement circa 1986, path 46 row 27, Table S31. Trends/CCDC Agreement summary for all comparison blocks in all path/rows, circa 2000, Table S32. Trends/CCDC Agreement summary for all comparison blocks in all path/rows, circa 1992. Acknowledgments: This work was supported with funding from the USGS Land Remote Sensing Program and the USGS LandCarbon Programs, partially under USGS contracts G15PC00012 (B.P. and D.D.) and G13PC00028 (Z.Z.). Any use of trade, ﬁrm, or product names is for descriptive purposes only and does not imply endorsement by the U.S. Government. Author Contributions: Bruce Pengra analyzed the data and wrote the majority of the manuscript. Alisa L. Gallant conceived the comparison and contributed to the manuscript. Zhe Zhu ran the CCDC algorithm to provide the output land cover maps and contributed to the manuscript. Devendra Dahal prepared training data and assisted with the analysis. Conﬂicts of Interest: The authors declare no conﬂicts of interest. References 1. Homer, C.; Dewitz, J.; Fry, J.; Coan, M.; Hossain, N.; Larson, C.; Herold, N.; McKerrow, A.; VanDriel, J.N.; Wickham, J. Completion of the 2001 National Land Cover Database for the counterminous United States. Photogramm. Eng. Remote Sens. 2007, 73, 337. 2. Bounoua, L.; DeFries, R.; Collatz, G.J.; Sellers, P.; Khan, H. Effects of land cover conversion on surface climate. Clim. Chang. 2002, 52, 29–64. [CrossRef] 3. Running, S.W. Ecosystem disturbance, carbon, and climate. Science 2008, 321, 652–653. [CrossRef] [PubMed] 4. Foley, J.A.; DeFries, R.; Asner, G.P.; Barford, C.; Bonan, G.; Carpenter, S.R.; Chapin, F.S.; Coe, M.T.; Daily, G.C.; Gibbs, H.K. Global consequences of land use. Science 2005, 309, 570–574. [CrossRef] [PubMed] 5. Turner, W.; Rondinini, C.; Pettorelli, N.; Mora, B.; Leidner, A.K.; Szantoi, Z.; Buchanan, G.; Dech, S.; Dwyer, J.; Herold, M. Free and open-access satellite data are key to biodiversity conservation. Biol. Conserv. 2015, 182, 173–176. [CrossRef] 6. Zhu, Z.; Woodcock, C.E. Continuous change detection and classiﬁcation of land cover using all available Landsat data. Remote Sens. Environ. 2014, 144, 152–171. [CrossRef] 7. Loveland, T.; Sohl, T.; Stehman, S.; Gallant, A.; Sayler, K.; Napton, D. A strategy for estimating the rates of recent United States land-cover changes. Photogramm. Eng. Remote Sens. 2002, 68, 1091–1099. 8. Sleeter, B.M.; Wilson, T.S.; Acevedo, W. Status and Trends of Land Change in the Western United States—1973 to 2000; U.S. Geological Survey Professional Paper 1794A; US Geological Survey: Menlo Park, CA, USA, 2012; p. 324. 9. Taylor, J.L.; Acevedo, W.; Auch, R.F.; Drummond, M.A. Status and Trends of land Change in the Great Plains of the United States—1973 to 2000; U.S. Geological Survey Professional Paper 1794B; U.S. Geological Survey: Reston, VA, USA, 2015; p. 190. 10. Auch, R.F.; Karstensen, K.A. Status and Trends of Land Change in the Midwest-South Central United States—1973 to 2000; U.S. Geological Survey Professional Paper 1794C; U.S. Geological Survey: Reston, VA, USA, 2015; p. 200. 11. Soulard, C.E.; Acevedo, W.; Auch, R.F.; Sohl, T.L.; Drummond, M.A.; Sleeter, B.M.; Sorenson, D.G.; Kambly, S.; Wilson, T.S.; Taylor, J.L. Land Cover Trends Dataset, 1973–2000; U.S. Geological Survey: Reston, VA, USA, 2014. 12. Anderson, J.R. A Land Use and Land Cover Classiﬁcation System for Use with Remote Sensor Data; US Government Printing Ofﬁce: Washington, DC, USA, 1976. 13. U.S. Geological Survey. Land Cover Trends Project Classiﬁcation System. Available online: http://landcovertrends.usgs.gov/main/classiﬁcation.html (accessed on 8 March 2016). Remote Sens. 2016, 8, 811 33 of 33 14. U.S. Environmental Protection Agency. Level III Ecoregions of the Continental United States (Revision of Omernik, 1987); Environmental Protection Agency—National Health and Environmental Effects Research Laboratory: Corvallis, OR, USA, 1999. 15. Omernik, J.M. Map supplement: Ecoregions of the conterminous United States. Ann. Assoc. Am. Geogr. 1987, 77, 118–125. [CrossRef] 16. Auch, R.F.; Drummond, M.A.; Sayler, K.L.; Gallant, A.L.; Acevedo, W. An approach to assess land-cover trends in the conterminous United States (1973–2000). In Remote Sensing of Land Use and Land Cover, Principles and Applications; Giri, C., Ed.; CRC Press: Boca Raton, FL, USA, 2012; pp. 351–368. 17. U.S. Geological Survey. National Land Cover Dataset 1992 (NLCD1992). Available online: http://www.mrlc. gov/nlcd1992.php (accessed on 8 March 2016). 18. Cochran, W.G. Sampling Techniques; Wiley and Sons: New York, NY, USA, 1977; Volume 98, pp. 259–261. 19. U.S. Geological Survey. Landsat 4–7 Climate Data Record (CDR) Surface Reﬂectance—User ’s Guide. Available online: http://landsat.usgs.gov/documents/cdr_sr_product_guide.pdf (accessed on 8 March 2016). 20. U.S. Geological Survey. Provisional Landsat 8 Surface Reﬂectance Product—Product Guide. Available online: http://landsat.usgs.gov/documents/provisional_l8sr_product_guide.pdf (accessed on 8 March 2016). 21. U.S. Geological Survey. Earth Resources Observation and Science (EROS) Center Science Processing Architecture (ESPA) on Demand Interface—User Guide. Available online: http://landsat.usgs.gov/ documents/espa_odi_userguide.pdf (accessed on 8 March 2016). 22. U.S. Geological Survey. Landsat Processing Details. Available online: http://landsat.usgs.gov/Landsat_ Processing_Details.php (accessed on 8 March 2016). 23. Zhu, Z.; Woodcock, C.E. Object-based cloud and cloud shadow detection in Landsat imagery. Remote Sens. Environ. 2012, 118, 83–94. [CrossRef] 24. Zhu, Z.; Wang, S.; Woodcock, C.E. Improvement and expansion of the Fmask algorithm: Cloud, cloud shadow, and snow detection for Landsats 4–7, 8, and Sentinel 2 images. Remote Sens. Environ. 2015, 159, 269–277. [CrossRef] 25. Zhu, Z.; Woodcock, C.E. Automated cloud, cloud shadow, and snow detection in multitemporal landsat data: An algorithm designed speciﬁcally for monitoring land cover change. Remote Sens. Environ. 2014, 152, 217–234. [CrossRef] 26. Zhu, Z.; Woodcock, C.E.; Holden, C.; Yang, Z. Generating synthetic Landsat images based on all available Landsat data: Predicting Landsat surface reﬂectance at any given time. Remote Sens. Environ. 2015, 162, 67–83. [CrossRef] 27. Davis, J.C.; Sampson, R.J. Statistics and Data Analysis in Geology; Wiley: New York, NY, USA, 1986; Volume 646. 28. Rayner, J.N. Introduction to Spectral Analysis; Pion: London, UK, 1971. 29. Breiman, L. Random forests. Mach. Learn. 2001, 45, 5–32. [CrossRef] 30. Zhu, Z. Optimizing selection of training and auxillary data for operational land cover classiﬁcation for the LCMAP initiative. ISPRS J. Photogramm. Remote Sens. 2016. submitted. 31. U.S. Geological Survey. National Land Cover Database 2006 (NLCD 2006). Available online: http://www.mrlc. gov/nlcd2006.php (accessed on 11 March 2016). 32. U.S. Fish and Wildlife Service. National Wetlands Inventory. Available online: http://www.fws.gov/ wetlands/ (accessed on 8 March 2016). 33. Natural Resources Conservation Service. Soil Survey Geographic Database. Available online: http://www.nrcs. usda.gov/wps/portal/nrcs/site/soils/home/ (accessed on 11 March 2016). 34. Wilen, B.O.; Bates, M. The U.S. Fish and Wildlife Service’s National Wetlands Inventory project. In Classiﬁcation and Inventory of the World’s Wetlands; Springer Netherlands: Dordrect, The Netherlands, 1995; pp. 153–169. © 2016 by the authors; licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC-BY) license (http://creativecommons.org/licenses/by/4.0/). http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Remote Sensing Multidisciplinary Digital Publishing Institute http://www.deepdyve.com/lp/multidisciplinary-digital-publishing-institute/evaluation-of-the-initial-thematic-output-from-a-continuous-change-bzMSkUazBp

Loading next page...

References (34)

(2000)
Status and Trends of Land Change in the Midwest-South Central United States—1973 to
Ramesh Jha, Andrew Leaver-Fay, Shuangye Yin, Yibing Wu, Glenn Butterfoss, T. Szyperski, Nikolay Dokholyan, B. Kuhlman (2010)
Computational design of a PAK1 binding protein.
Journal of molecular biology, 400 2
B. Wilen, M. Bates (1995)
The US Fish and Wildlife Service's National Wetlands Inventory Project
Vegetatio, 118
S. Running (2008)
Ecosystem Disturbance, Carbon, and Climate
Science, 321
(2015)
Status and Trends of land Change in the Great Plains of the United States—1973 to 2000; U.S. Geological Survey Professional Paper 1794B; U.S
(2012)
U . S . Geological Survey . National Land Cover Dataset 1992 ( NLCD 1992 )
Zhe Zhu, C. Woodcock (2014)
Automated cloud, cloud shadow, and snow detection in multitemporal Landsat data: An algorithm designed specifically for monitoring land cover change
Remote Sensing of Environment, 152
B. Conde, A. Villarino, M. Cabaleiro, D. González-Aguilera (2015)
Remote Sens
L. Bounoua, R. DeFries, G. Collatz, P. Sellers, H. Khan (2002)
Effects of Land Cover Conversion on Surface Climate
Climatic Change, 52
B. Longest (1971)
Sampling techniques.
Hospitals, 45 1
T. Loveland, T. Sohl, S. Stehman, A. Gallant, K. Sayler, D. Napton (2002)
A Strategy for Estimating the Rates of Recent United States Land-Cover Changes
Photogrammetric Engineering and Remote Sensing, 68
J. Omernik (2004)
Map Supplement Ecoregions of the Conterminous United States
(2016)
U.S. Geological Survey. Landsat Processing Details. Available
(2016)
Land Cover Trends Project Classification System
Zhe Zhu, Shixiong Wang, C. Woodcock (2015)
Improvement and expansion of the Fmask algorithm: cloud, cloud shadow, and snow detection for Landsats 4–7, 8, and Sentinel 2 images
Remote Sensing of Environment, 159
M. Miller (2010)
Agency
Change: The Magazine of Higher Learning, 42
J. Foley, R. DeFries, G. Asner, C. Barford, G. Bonan, S. Carpenter, F. Chapin, M. Coe, M. Coe, G. Daily, H. Gibbs, J. Helkowski, T. Holloway, E. Howard, C. Kucharik, C. Monfreda, J. Patz, I. Prentice, N. Ramankutty, P. Snyder (2005)
Global Consequences of Land Use
Science, 309
W. Turner, C. Rondinini, N. Pettorelli, B. Mora, A. Leidner, Z. Szantoi, G. Buchanan, S. Dech, J. Dwyer, M. Herold, L. Koh, P. Leimgruber, Hannes Taubenboeck, M. Wegmann, M. Wikelski, C. Woodcock (2015)
Free and open-access satellite data are key to biodiversity conservation
Biological Conservation, 182
(2016)
Natural Resources Conservation Service. Soil Survey Geographic Database Available online: http://www.nrcs. usda.gov/wps/portal/nrcs/site/soils/home
Zhe Zhu, C. Woodcock (2012)
Object-based cloud and cloud shadow detection in Landsat imagery
Remote Sensing of Environment, 118
R. Auch, Mark Drummond, K. Sayler, A. Gallant, W. Acevedo (2012)
An approach to assess land-cover trends in the conterminous United States (1973-2000)
James Anderson, Ernest Hardy, John Roach, Richard Witmer (1976)
A land use and land cover classification system for use with remote sensor data
Zhe Zhu, C. Woodcock (2012)
Continuous change detection and classification of land cover using all available Landsat data
Remote Sensing of Environment, 144
L. Breiman (2001)
Random Forests
Machine Learning, 45
B. Hofmann-Wellenhof, H. Moritz (1986)
Introduction to spectral analysis
, 7
U . S . Geological Survey . National Land Cover Database 2006 ( NLCD 2006 ) National Wetlands Inventory Soil Survey Geographic Database
C. Homer, J. Dewitz, J. Fry, M. Coan, Nazmul Hossain, Charles Larson, A. McKerrow, J. VanDriel, J. Wickham (2007)
Completion of the 2001 National Land Cover Database for the conterminous United States
Photogrammetric Engineering and Remote Sensing, 73
J. Omernik (1987)
Ecoregions of the Conterminous United States
Annals of The Association of American Geographers, 77
(1987)
Level III Ecoregions of the Continental United States (Revision of Omernik
C. Finlayson, A. Valk (1995)
Classification and Inventory of the World’s Wetlands
Zhe Zhu, A. Gallant, C. Woodcock, Bruce Pengra, P. Olofsson, T. Loveland, Suming Jin, D. Dahal, Limin Yang, R. Auch (2016)
Optimizing selection of training and auxiliary data for operational land cover classification for the LCMAP initiative
Isprs Journal of Photogrammetry and Remote Sensing, 122
Zhe Zhu, C. Woodcock, Christopher Holden, Zhiqiang Yang (2015)
Generating synthetic Landsat images based on all available Landsat data: Predicting Landsat surface reflectance at any given time
Remote Sensing of Environment, 162
R. Reyment, J. Davis (1988)
Statistics and Data Analysis in Geology.
Biometrics, 44
(1973)
Land Cover Trends Dataset

Publisher: Multidisciplinary Digital Publishing Institute
Copyright: © 1996-2019 MDPI (Basel, Switzerland) unless otherwise stated
ISSN: 2072-4292
DOI: 10.3390/rs8100811
Publisher site: See Article on Publisher Site

Abstract

remote sensing Article Evaluation of the Initial Thematic Output from a Continuous Change-Detection Algorithm for Use in Automated Operational Land-Change Mapping by the U.S. Geological Survey 1 , 2 3 , † 1 Bruce Pengra *, Alisa L. Gallant , Zhe Zhu and Devendra Dahal SGT Inc., Contractor to the U.S. Geological Survey, Earth Resources Observation and Science Center, 47914 252nd St., Sioux Falls, SD 57198, USA; [email protected] U.S. Geological Survey, Earth Resources Observation and Science Center, 47914 252nd St., Sioux Falls, SD 57198, USA; [email protected] Inuteq., Contractor to the U.S. Geological Survey, Earth Resources Observation and Science Center, 47914 252nd St., Sioux Falls, SD 57198, USA; [email protected] * Correspondence: [email protected]; Tel.: +1-605-594-6865 † Current address: Department of Geosciences, MS 1053, Science Building 125, Texas Tech University, Lubbock, TX 79409, USA; [email protected]. Academic Editors: Parth Sarathi Roy and Prasad S. Thenkabail Received: 3 May 2016; Accepted: 19 September 2016; Published: 1 October 2016 Abstract: The U.S. Geological Survey (USGS) has begun the development of operational, 30-m resolution annual thematic land cover data to meet the needs of a variety of land cover data users. The Continuous Change Detection and Classiﬁcation (CCDC) algorithm is being evaluated as the likely methodology following early trials. Data for training and testing of CCDC thematic maps have been provided by the USGS Land Cover Trends (LC Trends) project, which offers sample-based, manually classiﬁed thematic land cover data at 2755 probabilistically located sample blocks across the conterminous United States. These samples represent a high quality, well distributed source of data to train the Random Forest classiﬁer invoked by CCDC. We evaluated the suitability of LC Trends data to train the classiﬁer by assessing the agreement of annual land cover maps output from CCDC with output from the LC Trends project within 14 Landsat path/row locations across the conterminous United States. We used a small subset of circa 2000 data from the LC Trends project to train the classiﬁer, reserving the remaining Trends data from 2000, and incorporating LC Trends data from 1992, to evaluate measures of agreement across time, space, and thematic classes, and to characterize disagreement. Overall agreement ranged from 75% to 98% across the path/rows, and results were largely consistent across time. Land cover types that were well represented in the training data tended to have higher rates of agreement between LC Trends and CCDC outputs. Characteristics of disagreement are being used to improve the use of LC Trends data as a continued source of training information for operational production of annual land cover maps. Keywords: Continuous Change Detection and Classiﬁcation; USGS Land Cover Trends; training data; Landsat; high-resolution imagery; land cover mapping 1. Introduction Mapping land cover and monitoring land cover change are important for a variety of societal and scientiﬁc purposes, including land management, natural resource management, ecological studies, sustainable development, climate modeling, urban planning, habitat monitoring, and many others [1–5]. The U.S Geological Survey (USGS) is moving forward with a Land Change Monitoring, Remote Sens. 2016, 8, 811; doi:10.3390/rs8100811 www.mdpi.com/journal/remotesensing Remote Sens. 2016, 8, 811 2 of 33 Assessment, and Projection (LCMAP) initiative to develop an expanded operational capacity for land cover mapping and monitoring to support these needs. One goal of LCMAP is to provide high temporal and moderate spatial resolution land cover and land change products, including annual thematic land cover at 30 m resolution (30 30 m pixels). The Continuous Change Detection and Classiﬁcation (CCDC) algorithm [6] was developed to support continuous monitoring with Landsat data to take advantage of the multi-decadal Landsat archive housed by the USGS and is expected to play a central role in LCMAP mapping and monitoring activities. CCDC will be used to generate annual thematic land cover maps, with a class legend generally based on Anderson Level 1 categories of land cover previously adopted by the USGS Land Cover Trends project [7]. The USGS Land Cover LC Trends project plays an integral role in the development of the current capability for continuous monitoring by providing a reliable, consistent land cover product and related change assessments [8–10]. LC Trends data were generated through manual interpretation and were developed for the nominal years of 1973, 1980, 1986, 1992, and 2000 [7]. These data offer a basis for both training and initial testing of CCDC land cover classiﬁcation. Our objective was to conduct a comparison between output from the CCDC algorithm and LC Trends data [11] to determine whether CCDC can produce comparable maps. We selected 14 Landsat path/row locations within the conterminous United States to capture a wide range of land cover types and mapping challenges. CCDC currently operates on Landsat data collected since the operation of the Thematic Mapper sensor [6], launched in 1982, so we compared CCDC map output with LC Trends data for 1992 and 2000. Our goal was to determine if results were consistent across time, space, and thematic classes and gain insight into the use of LC Trends data for eventual operational training of annual land cover maps with CCDC. The results we present do not provide a statistical description of error in the CCDC Land Cover maps; they provide levels of agreement with maps generated from the LC Trends project and characterize features associated with common categories of disagreement between LC Trends and CCDC maps. The aim of our assessment is to inform reﬁnement of the CCDC algorithm’s approach for deriving annual land cover maps and provide internal information on data inputs and work ﬂows. 2. Materials and Methods 2.1. Data 2.1.1. Land Cover Trends The LC Trends project used analyst interpretation of Landsat data for sample blocks of 10 km 10 km (for most parts of the United States) or 20 km 20 km (for a few areas) to characterize land cover and land cover change [7]. Land cover map dates were nominally 1973, 1980, 1986, 1992 and 2000, with actual dates of imagery varying for some samples because of clouds, poor data quality, or availability. The classiﬁcation system used 11 classes representing a mix of land cover and land use types that “paralleled” the Anderson Level I classiﬁcation system (Table 1) [12,13]. A national set of 2755 blocks was selected using a probabilistic sampling design with stratiﬁcation based on 1999-era Level III ecoregions deﬁned by the U.S. Environmental Protection Agency [8,14,15]. The LC Trends project mapped land use/land cover for 1992 as a baseline from which successive dates of land cover were mapped both forward and backward in time [16]. This initial baseline was created by starting with the National Land Cover Dataset [17] for 1992, collapsing the more detailed classes to the LC Trends class legend, then manually editing to improve local accuracy [16]. Changes from the 1992 baseline were identiﬁed and delineated manually using Landsat data, aided by aerial photographs and other ancillary data, to produce the land cover for successive dates. A minimum mapping unit of 60 60 m was used. Analysts conducted group reviews of each other ’s work for every sample [7,16]. Remote Sens. 2016, 8, 811 3 of 33 Table 1. Land use/land cover classes and descriptions used by the U.S. Geological Survey, Land Cover Trends project. Land Cover Class Description Open water Areas persistently covered with water, such as streams, canals, lakes, reservoirs, bays, and oceans. Areas of intensive use where much of the land is covered with structures or anthropogenic impervious surfaces (residential, commercial, industrial, roads, etc.) or less-intensive use where Developed (urban or the land cover matrix includes both vegetation and structures (low-density residential, otherwise built-up) recreational facilities, cemeteries, utility corridors, etc.), including any land functionally related to urban or built-up environments (parks, golf courses, etc.). Land in either a vegetated or an unvegetated state used for the production of food and ﬁber, Agriculture (cropland including cultivated and uncultivated croplands, haylands, pasture, orchards, vineyards, and and pasture) conﬁned livestock operations. Note that forest plantations are considered forests regardless of their use for wood products. Non-developed land where the tree cover density is >10%. Note that cleared forest land Forest and woodland (i.e., clearcuts) is mapped according to current cover (e.g., mechanically disturbed or grassland/shrubland). Non-developed land where cover by grasses, forbs, and/or shrubs predominates and tree-cover Grassland/shrubland density is <10%. Land where water saturation is the determining factor in soil characteristics, vegetation types, Wetland and animal communities. Wetlands can contain both water and vegetated cover. Areas with extractive mining activities that have a signiﬁcant surface expression, including Mines and quarries mining buildings, quarry pits, overburden, leach, evaporative features, tailings, or other related components. Land composed of soils, sand, or rocks where <10% of the area is vegetated. Does not include Barren land in transition recently cleared by disturbance. Land in an altered, often unvegetated transitional state caused by disturbance from mechanical Mechanically disturbed means, as by forest clearcutting, earthmoving, scraping, chaining, reservoir drawdown, and other similar human-induced changes. Land in an altered, often unvegetated transitional state caused by disturbance from Non-mechanically disturbed non-mechanical means, as by ﬁre, wind, ﬂood, animals, and other similar phenomena. Land where the accumulation of snow and ice does not completely melt during summer Snow and ice (e.g., alpine glaciers and snowﬁelds). Whereas LC Trends maps provided an established USGS land cover product, we were not aware of any quantitative assessment of its accuracy. We conducted an accuracy assessment for LC Trends blocks in 5 of our 14 path/rows that overlapped high numbers of blocks (112 of 186 blocks) and represented a range of landscape settings. Included in the assessment were path/rows 23/37, 27/27, 28/33, 43/34 and 46/27. Based on an assumed accuracy rate of 95% with a target standard error of 0.025, a 300 point sample size met criteria for estimating overall accuracy [18]. Sample locations were randomly selected from within all LC Trends pixels in the ﬁve path/rows. Analysts manually interpreted high resolution imagery available in Google Earth™, assigning a primary land cover label based on LC Trends class deﬁnitions. Secondary labels were assigned for pixels where mixed cover types made class assignment ambiguous. These reference data showed LC Trends to have an overall accuracy of 91% based on the primary label, and 99% accuracy when both the primary and secondary labels were considered. This conﬁrmed expectations that LC Trends data were of high accuracy and appropriate for use as a source of training data for CCDC. 2.1.2. CCDC Annual Land Cover We acquired time series Landsat data processed to surface reﬂectance [19,20] from the USGS Earth Resources Observation and Science (EROS) Center ’s Science Processing Architecture (ESPA) data system [21]. We included all archived data from Landsat 4, Landsat 5, Landsat 7, and Landsat 8 scenes with processing to the L1T standard [22] that had more than 20% clear observations (no cloud, cloud shadow, or snow). Clouds, cloud shadows, and snow were screened initially by ESPA with the Fmask algorithm (speciﬁcally, the CFmask implementation) [23,24], then further screened via a multitemporal cloud, cloud shadow, and snow detection algorithm called Tmask [25]. Remote Sens. 2016, 8, 811 4 of 33 Remote Sens. 2016, 8, 811 4 of 33 The CCDC algorithm uses all available Landsat data to estimate time series models and applies the models to predict future observations [6,26]. If the values of new observations are not within The CCDC algorithm uses all available Landsat data to estimate time series models and applies the predicted range for six consecutive observations, a break in the time series is ﬂagged and a new the models to predict future observations [6,26]. If the values of new observations are not within the time series model will be estimated when sufﬁcient observations are available. The time series predicted range for six consecutive observations, a break in the time series is flagged and a new models are composed of harmonic models [27,28] that capture annual cycles, seasonality, and a slope time series model will be estimated when sufficient observations are available. The time series models are composed of harmonic models [27,28] that capture annual cycles, seasonality, and a component. The breaks found in the time series provide change information, such as caused by land slope component. The breaks found in the time series provide change information, such as caused cover conversion. The coefﬁcients that deﬁne the time series cycles and slope, along with the root by land cover conversion. The coefficients that define the time series cycles and slope, along with mean square errors (RMSE), are used as inputs to a land cover classiﬁer (Figure 1). CCDC uses the the root mean square errors (RMSE), are used as inputs to a land cover classifier (Figure 1). CCDC Random Forest classiﬁer [29] to derive decision tree models that are used to generate land cover maps. uses the Random Forest classifier [29] to derive decision tree models that are used to generate land The time-series approach used by CCDC means that model trajectories can be “consulted” at any given cover maps. The time-series approach used by CCDC means that model trajectories can be time within the time series period to generate a map of land cover. “consulted” at any given time within the time series period to generate a map of land cover. Figure 1. Example of time series models estimated for all Landsat bands for Forest and Developed Figure 1. Example of time series models estimated for all Landsat bands for Forest and Developed (residential) land cover classes. During the transition between classes, where Continuous Change (residential) land cover classes. During the transition between classes, where Continuous Change Detection Detection and Classification (CCDC) did not fit a model, land cover was labeled “Disturbed”. and Classification (CCDC) did not fit a model, land cover was labeled “Disturbed”. BT = brightness BT = brightness temperature; ETM = Enhanced Thematic Mapper Plus; NIR = near-infrared; temperature; ETM = Enhanced Thematic Mapper Plus; NIR = near-infrared; OLS = ordinary least OLS = ordinary least squares; SR = surface reflectance; SWIR1 = shortwave infrared 1; squares; SR = surface reﬂectance; SWIR1 = shortwave infrared 1; SWIR2 = shortwave infrared 2; SWIR2 = shortwave infrared 2; TIRS = Thermal Infrared Sensor; TM = Thematic Mapper. TIRS = Thermal Infrared Sensor; TM = Thematic Mapper. As the CCDC algorithm is capable of providing land cover maps at any given date, we selected a fixed day of the year (1 July) to provide annual CCDC land cover maps for our assessment. Note As the CCDC algorithm is capable of providing land cover maps at any given date, we selected that following a break in the time series (six consecutive observations not within the predicted a ﬁxed day of the year (1 July) to provide annual CCDC land cover maps for our assessment. Note range), observations may fluctuate such that CCDC is unable to initiate a new time series model. that following a break in the time series (six consecutive observations not within the predicted range), During this time period we label the pixel as “Disturbed” (Figure 1). Thus, the annual land cover at observations may ﬂuctuate such that CCDC is unable to initiate a new time series model. During the 14 path/row locations was labeled as Disturbed whenever a pixel was unable to initiate a time this time period we label the pixel as “Disturbed” (Figure 1). Thus, the annual land cover at the series model over the 1 July anniversary date of the map. We used LC Trends data from 2000 as the pool to extract training data for the classifier. We 14 path/row locations was labeled as Disturbed whenever a pixel was unable to initiate a time series extracted training data from LC Trends blocks based on criteria developed from an analysis of best model over the 1 July anniversary date of the map. practices [30]. That analysis found that a total of 20,000 pixels distributed across classes in We used LC Trends data from 2000 as the pool to extract training data for the classiﬁer. proportion to the LC Trends class distribution was optimal, with a minimum of 600 pixels and a We extracted training data from LC Trends blocks based on criteria developed from an analysis maximum of 8000 pixels required for each class (note, if the total number of pixels for a given class of best practices [30]. That analysis found that a total of 20,000 pixels distributed across classes was less than 600, we extracted all available pixels). Selection of training pixels within each class in proportion to the LC Trends class distribution was optimal, with a minimum of 600 pixels and was random from all available pixels in that class. We also incorporated eight ancillary datasets for a maximum of 8000 pixels required for each class (note, if the total number of pixels for a given training and classification: digital elevation data and derivatives (aspect, slope, and position index), class was less than 600, we extracted all available pixels). Selection of training pixels within each a Wetland Potential Index (an index map generated for the 2006 U.S. National Land Cover Database class [3 was 1], N random ational fr Wet om lands all available Inventory [ pixels 32], and in that Soil S class. urvey Geo We also graincorporated phic [SSURGO] h eight ydric ancillary soils map datasets s [33]), and probability of cloud, snow, and water occurrence. The latter probabilities were derived for training and classiﬁcation: digital elevation data and derivatives (aspect, slope, and position from Fmask statistics and represented the percent of cloud (or snow or water) observations from all index), a Wetland Potential Index (an index map generated for the 2006 U.S. National Land Cover available historical observations in the Landsat archive. Database [31], National Wetlands Inventory [32], and Soil Survey Geographic [SSURGO] hydric soils maps [33]), and probability of cloud, snow, and water occurrence. The latter probabilities were derived Remote Sens. 2016, 8, 811 5 of 33 from Fmask statistics and represented the percent of cloud (or snow or water) observations from all available Remote Sens historical . 2016, 8observations , 811 in the Landsat archive. 5 of 33 The CCDC workﬂow creates multiple products, but the thematic land cover product was the only The CCDC workflow creates multiple products, but the thematic land cover product was the type we used for the current analysis. only type we used for the current analysis. 2.2. Methods of Land Cover Product Comparison 2.2. Methods of Land Cover Product Comparison The area of comparison was deﬁned by the footprint of LC Trends blocks in each of the The area of comparison was defined by the footprint of LC Trends blocks in each of the 14 14 path/rows (Figure 2). We re-projected the data from the LC Trends blocks from Albers Equal path/rows (Figure 2). We re-projected the data from the LC Trends blocks from Albers Equal Area, Area, NAD83, to the Universal Transverse Mercator system, WGS84, to correspond with the Landsat NAD83, to the Universal Transverse Mercator system, WGS84, to correspond with the Landsat time time series data. It was likely that the re-projection resulted in some degradation in spatial ﬁdelity; series data. It was likely that the re-projection resulted in some degradation in spatial fidelity; however however, visual exam , visual examination ination showed this to be showed this to be of of minor minor concern, given the concern, given the 60 60 × 60 m minimum 60 m minimum mapping unit applied by the LC Trends project and the level of spatial generalization inherent in mapping unit applied by the LC Trends project and the level of spatial generalization inherent in the the (manually delineated) LC Trends data. (manually delineated) LC Trends data. Figure 2. Land Cover (LC) Trends sample blocks (dot symbols) within the 14 path/row test locations. Figure 2. Land Cover (LC) Trends sample blocks (dot symbols) within the 14 path/row test locations. We created a map of per-pixel agreement for the area covered by LC Trends blocks within each W ofe the 14 created test a pa map th/rows, of perma -pixel tching the yea agreement r of for CCD the C l area and cover coveredoutput to the yea by LC Trends blocks r of source da within each ta of used for the LC Trends samples. Output layers were used to associate categories of disagreement the 14 test path/rows, matching the year of CCDC land cover output to the year of source data used for with specific locations and to characterize the conditions (land cover characteristics and data the LC Trends samples. Output layers were used to associate categories of disagreement with speciﬁc characteristics) typical of the main categories of confusion. We constructed a set of confusion locations and to characterize the conditions (land cover characteristics and data characteristics) typical matrices for all LC Trends blocks in each of the 14 path/rows and for each date of comparison of the main categories of confusion. We constructed a set of confusion matrices for all LC Trends blocks (Tables S1–S30). These were aggregated across path/rows into error matrices representing all pixels in each of the 14 path/rows and for each date of comparison (Tables S1–S30). These were aggregated in all path/rows for each of the nominal dates of comparison (Tables S31 and S32). These summary across path/rows into error matrices representing all pixels in all path/rows for each of the nominal error matrices, primarily the one for the nominal 2000 data comparison (Table 2), were the basis for dates of comparison (Tables S31 and S32). These summary error matrices, primarily the one for the identifying categories of disagreement covering the largest aerial extent as a fraction of the entire nominal 2000 data comparison (Table 2), were the basis for identifying categories of disagreement area of study and/or covering the largest area as a fraction of the respective LC Trends and CCDC covering classes to which the pi the largest aerial xels extent belonged. as a fraction of the entire area of study and/or covering the largest For each category of disagreement, we ranked the degree of concentration within the path/row area as a fraction of the respective LC Trends and CCDC classes to which the pixels belonged. locations and the relative rate of occurrence per the area covered by LC Trends blocks. We For each category of disagreement, we ranked the degree of concentration within the path/row developed vector layers delineating the areas of disagreement for the path/rows with the largest locations and the relative rate of occurrence per the area covered by LC Trends blocks. We developed concentrations or highest rates per area of comparison. We displayed the vector layers in Google vector layers delineating the areas of disagreement for the path/rows with the largest concentrations Earth™ and overlaid them on Landsat time series images and the thematic classifications of CCDC or highest rates per area of comparison. We displayed the vector layers in Google Earth™ and and LC Trends. We evaluated the land cover associated with each category of disagreement to overlaid them on Landsat time series images and the thematic classiﬁcations of CCDC and LC Trends. identify patterns of occurrence that might be used to improve the CCDC annual land cover accuracy. We evaluated the land cover associated with each category of disagreement to identify patterns of occurrence that might be used to improve the CCDC annual land cover accuracy. Remote Sens. 2016, 8, 811 6 of 33 Table 2. Confusion matrix for all pixels from all path/rows for the 2000 period. Circa 2000 Land Cover Trends Water Developed Disturbed Mining Barren Forest Grass/Shrub Agriculture Wetlands Ice & Snow Total Agreement Water 462,323 4832 1301 615 6992 11,638 8910 11,907 11,395 0 519,913 89% Developed 4652 705,417 22,002 4237 2245 113,665 35,972 128,682 5009 0 1,021,881 69% Disturbed 3137 9223 17,088 1298 969 30,110 162,546 107,305 7734 52 339,462 5% Mining 747 14,105 2829 26,515 1423 5251 4611 13,700 595 0 69,776 38% Barren 4274 833 51 20 120,469 11,994 24,645 2495 2176 9701 176,658 68% Circa 2000 CCDC Forest 19,043 118,687 169,391 2278 14,835 5,781,165 273,434 161,377 133,544 227 6,673,981 87% Grass/Shrub 7402 28,907 45,205 883 19,596 205,283 3,958,903 292,673 19,373 2451 4,580,676 86% Agriculture 15,242 98,464 9168 1801 437 116,467 161,455 5,891,790 55,527 0 6,350,351 93% Wetlands 13,611 2102 11,446 155 1522 112,575 25,212 28,568 554,469 0 749,660 74% Ice & Snow 5 0 0 0 5863 106 136 0 0 60,573 66,683 91% 530,436 982,570 278,481 37,802 174,351 6,388,254 4,655,824 6,638,497 789,822 73,004 overall 85.5% 87% 72% 6% 70% 69% 90% 85% 89% 70% 83% agreement Remote Sens. 2016, 8, 811 7 of 38 Remote Sens. 2016, 8, 811 7 of 33 3. Results 3. Results 3.1. Class Distribution 3.1. Class Distribution The distribution of classes was very similar for LC Trends and CCDC output when aggregated across the 14 path/rows (Figure 3). The largest differences in absolute area were in the Agriculture The distribution of classes was very similar for LC Trends and CCDC output when aggregated and Forest Classes. The LC Trends data resulted in just over 1.4% more of the map labeled as across the 14 path/rows (Figure 3). The largest differences in absolute area were in the Agriculture and Agriculture and just under 1.4% less of the map labeled as Forest than did CCDC. The area mapped Forest Classes. The LC Trends data resulted in just over 1.4% more of the map labeled as Agriculture per class by CCDC and LC Trends differed substantially more for most classes when compared and just under 1.4% less of the map labeled as Forest than did CCDC. The area mapped per class by within individual path/rows (Figure 4). This difference was most obvious for the Disturbed class in CCDC and LC Trends differed substantially more for most classes when compared within individual Arizona (36/38) and California (43/34) and for smaller classes such as Mining and Barren. However, path/rows (Figure 4). This difference was most obvious for the Disturbed class in Arizona (36/38) and some large differences in area mapped for more common classes occurred as well. For example, California (43/34) and for smaller classes such as Mining and Barren. However, some large differences CCDC mapped 6.7% more Forest in Washington (46/27) and 6.4% less Forest in Minnesota (27/27) in area mapped for more common classes occurred as well. For example, CCDC mapped 6.7% more than did LC Trends. LC Trends mapped 7.9% more Agriculture than CCDC in California (43/34) Forest in Washington (46/27) and 6.4% less Forest in Minnesota (27/27) than did LC Trends. LC Trends and 5.8% more in Kansas (28/33). mapped 7.9% more Agriculture than CCDC in California (43/34) and 5.8% more in Kansas (28/33). Figure 3. Class distribution for CCDC and LC Trends output over the total area of comparison. Figure 3. Class distribution for CCDC and LC Trends output over the total area of comparison. 3.2. Per-Pixel Agreement 3.2. Per-Pixel Agreement 3.2.1. Summary of Per-Pixel Agreement 3.2.1. Summary of Per-Pixel Agreement The summary confusion matrix for the circa 2000 data comparison showed 85.5% overall The summary confusion matrix for the circa 2000 data comparison showed 85.5% overall agreement between CCDC and LC Trends land cover (Table 2). Agreement between the largest agreement between CCDC and LC Trends land cover (Table 2). Agreement between the largest classes (Forest, Agriculture, and Grass/Shrub) ranged from a low of 85% producer’s agreement for classes (Forest, Agriculture, and Grass/Shrub) ranged from a low of 85% producer ’s agreement for the Grass/Shrub class to a high of 93% user’s agreement for the Agriculture class. The smaller and the Grass/Shrub class to a high of 93% user ’s agreement for the Agriculture class. The smaller and generally more fragmented Developed and Wetland classes showed 69% user’s and 72% producers generally more fragmented Developed and Wetland classes showed 69% user ’s and 72% producers agreement for the Developed class and 74% user’s and 70% producer’s agreement for the Wetland agreement for the Developed class and 74% user ’s and 70% producer ’s agreement for the Wetland class. The Barren class, which accounted for approximately 1% of the mapped area in both CCDC class. The Barren class, which accounted for approximately 1% of the mapped area in both CCDC and and LC Trends maps, had 68% user’s and 69% producer’s agreement. Mining, by far the smallest LC Trends maps, had 68% user ’s and 69% producer ’s agreement. Mining, by far the smallest class class in either classification (0.3% of area with CCDC and 0.2% of area with LC Trends), had low in either classiﬁcation (0.3% of area with CCDC and 0.2% of area with LC Trends), had low user ’s user’s agreement at 38%, but 70% producer’s agreement. Class agreement for Disturbed area was agreement at 38%, but 70% producer ’s agreement. Class agreement for Disturbed area was extremely extremely low (6.1% producer’s agreement, 5.0% user’s agreement). low (6.1% producer ’s agreement, 5.0% user ’s agreement). Remote Sens. 2016, 8, 811 8 of 33 Remote Sens. 2016, 8, 811 8 of 33 Figure 4. Comparison of class area mapped in each path/row by CCDC and LC Trends. Graph y-axes Figure 4. Comparison of class area mapped in each path/row by CCDC and LC Trends. Graph y- are scaled per class size to best display detail of comparison. axes are scaled per class size to best display detail of comparison. 3.2.2. Per-Pixel Agreement by Path/Row Overall agreement at individual path/row test locations across all years of comparison ranged from 75% to 98% (Table 3). For the circa 2000 period only two of the 14 path/rows had overall Remote Sens. 2016, 8, 811 9 of 33 agreement below 80%, 27/27 in Northern Minnesota (79.4%) and 16/40 in Florida (75.1%). Agreement was very similar for the 1992 and 2000 periods at most path/row locations. The two path/rows with the largest difference in agreement between the 1992 and 2000 periods were 36/38 in Arizona (97.6% overall agreement in 1992 dropped to 82.0% in 2000) and 43/34 in California (78.9% agreement in 1992 rose to 84.8% in 2000). Most of the difference was accounted for by disagreement in the Disturbed class; in both cases CCDC classiﬁed very large areas of Disturbed land in one of the two periods under comparison, but LC Trends did not. Error matrices for the four path/rows with the highest numbers of LC Trends blocks (Table 4) account for 53% of the total area across all 14 path/rows and provide representative examples of the variation of class distribution, overall accuracy, and class accuracy across all test sites. 3.2.3. Per-Pixel Agreement by Land Cover Class Forest Class Agreement The CCDC and LC Trends Forest classes had 90.5% producer ’s and 86.6% user ’s agreement overall. CCDC mapped 1.4% more forest area than did LC Trends. CCDC mapped less forest area than LC Trends in only two locations, Kansas and Arizona (Table 5), the two path/rows with the lowest Forest class producer ’s agreement, 60.5% and 54.7%, respectively. Forest class agreement varied widely across the test path/rows (Table 5). The two locations with the highest Forest class agreement for both 1992 and 2000 were New Hampshire/Vermont (13/29) and South Dakota (33/29). Tree cover occurred as largely unbroken expanses of mixed forest species (New Hampshire/Vermont) or managed, predominantly conifer, national forest (South Dakota). The eight LC Trends blocks in New Hampshire/Vermont had 97.4% producer ’s and 96.2% user ’s agreement between CCDC Forest and LC Trends Forest classes in 2000, and 97.7% producer ’s and 96.4% user ’s agreement in 1992. With just a single forested LC Trends block, South Dakota results showed 98.8% producer ’s and 93.3% user ’s agreement in 2000 and 98.8% producer ’s and 93.0% user ’s agreement in 1992. LC Trends blocks in the Washington path/row covered 18.7% of the total study area and 35% of the LC Trends Forest class. The LC Trends and CCDC Forest classes in this path/row exhibited 91.9% producer ’s and 86.2% user ’s agreement. The majority of LC Trends Forest pixels with which CCDC disagreed were classiﬁed as Developed by CCDC, accounting for 55.5% of the Forest class producer ’s disagreement. In most cases, high resolution images showed these areas to be associated with low-intensity development, where tree cover mixed with some houses and roads had been generalized to the Forest class by LC Trends interpreters. Of the roughly 14% of CCDC Forest pixels in Washington that disagreed with LC Trends pixels, 30.6% had been classiﬁed as Grass/Shrub and 27.9% had been classiﬁed as Developed by LC Trends. Most cases of the former occurred in forest harvest footprints, where early stages of forest regeneration were classiﬁed as Grass/Shrub by LC Trends. Cases of the latter were where land cover generally occurred as fragmented clusters of tree cover within a larger context of low-intensity development. The fragmentation of land cover created a high proportion of mixed pixels and edge pixels, where minor misregistration was likely a contributing factor to disagreement. Nevertheless, the majority of disagreement pixels were actually covered by trees, but often were generalized to the Developed class by LC Trends interpreters and mapped as Forest by CCDC. The Minnesota path/row (27/27) was dominated by tree cover, with much of it in woody wetland. The LC Trends and CCDC Forest classes here had 88.0% producer ’s agreement and 82.7% user ’s agreement. CCDC disagreement with LC Trends Forest pixels predominantly (82.6% of the time) occurred where CCDC classiﬁed the pixels as Wetland. LC Trends disagreed with CCDC Forest pixels approximately 17% of the time, often (15.6% of the time) labeling those pixels as Grass/Shrub or Wetland (40.2%). Remote Sens. 2016, 8, 811 10 of 33 Table 3. Overall agreement per path/row for 1992 and 2000 time periods. AR = Arkansas; AZ = Arizona; CA = California; CO = Colorado; FL = Florida; IL = Illinois; IN = Indiana; KS = Kansas; MN = Minnesota; MS = Mississippi; MO = Montana; ND = North Dakota; NH = New Hampshire; SD = South Dakota; VT = Vermont; WA = Washington. Location 13/29 NH/VT 16/40 FL 22/33 IL/IN 23/37 AR/MS 27/27 MN 28/33 KS 31/27 ND 33/29 SD 34/33 CO 35/32 CO 36/38 AZ 39/26 MT 43/34 CA 46/27 WA Overall agreement 2000 93.5% 75.1% 87.8% 86.4% 79.4% 86.2% 89.0% 93.9% 89.9% 89.8% 82.1% 91.7% 84.8% 80.4% Overall agreement 1992 93.8% 76.9% 87.7% 87.4% 79.2% 85.8% 87.7% 89.9% 90.0% 89.3% 97.6% 89.5% 78.9% 80.3% Table 4. Confusion matrices for the four path/row locations with the most total area of LC Trends data available for comparison. Table values are numbers of pixels. Washington 46/27 Trends/CCDC Agreement Circa 2000 Land Cover Trends Water Developed Disturbed Mining Barren Forest Grass/Shrub Agriculture Wetlands Ice & Snow Total Agreement Water 199,996 1947 3 127 6810 2565 7 1074 1838 0 214,367 93% Developed 2789 406,045 15,638 3503 2147 99,507 12,121 50,405 2971 0 595,126 68% Disturbed 107 2758 5782 486 43 4047 522 1032 146 52 14,975 39% Mining 335 8606 2189 7155 1293 1995 311 707 121 0 22,712 32% Barren 775 504 0 17 51,056 6582 4616 12 105 9701 73,368 70% Circa 2000 CCDC 6 6 Forest 7654 91,646 71,903 1154 12,690 2 10 100,308 19,146 23,476 227 2 10 86% Grass/Shrub 26 2217 38,632 112 9728 49,568 95,688 355 213 2451 198,990 48% Agriculture 1895 32,531 1355 155 414 8105 1394 200,775 3969 0 250,593 80% Wetlands 3577 1068 177 21 1364 6695 741 3777 25,504 0 42,924 59% Ice & Snow 5 0 0 0 5863 106 136 0 0 60,573 66,683 91% Total 217,159 547,322 135,679 12,730 91,408 2 10 215,844 277,283 58,343 73,004 Overall 80.4% Agreement Agreement 92% 74% 4% 56% 56% 92% 44% 72% 44% 83% California 43/34 Trends/CCDC Agreement Circa 2000 Land Cover Trends Water Developed Disturbed Mining Barren Forest Grass/Shrub Agriculture Wetlands Total Agreement Water 13,606 1405 1247 430 0 180 1537 2519 302 21,226 64% Developed 459 170,339 675 286 0 340 9954 44,284 121 226,458 75% Disturbed 156 4158 907 522 0 736 36,406 86,812 799 130,496 1% Mining 64 40 29 4817 0 569 1039 1337 53 7948 61% Circa 2000 CCDC Barren 33 0 22 0 2643 1203 3536 0 0 7437 36% Forest 186 362 13,228 52 60 275,941 27,007 218 12 317,066 87% Grass/Shrub 620 15,837 1124 470 72 31,394 518,106 53,435 1555 622,613 83% Agriculture 2099 31,677 1207 998 0 1775 28,236 1,379,735 1365 1,447,092 95% Wetlands 696 213 3 37 0 237 5766 2503 5590 15,045 37% Total 17,919 224,031 18,442 7612 2775 312,375 631,587 1,570,843 9797 Overall 84.8% Agreement 76% 76% 5% 63% 95% 88% 82% 88% 57% Agreement Remote Sens. 2016, 8, 811 11 of 33 Table 4. Cont. Kansas 28/33 Trends/CCDC Agreement Circa 2000 Land Cover Trends Water Developed Disturbed Mining Barren Forest Grass/Shrub Agriculture Wetlands Total Agreement Water 22,263 392 0 9 65 537 3190 1519 592 28,567 78% Developed 131 16,170 0 8 5 375 1552 5216 34 23,491 69% Disturbed 947 55 0 103 488 53 379 3839 337 6201 0% Mining 59 1024 0 1712 0 98 1902 6127 80 11,002 16% Circa 2000 CCDC Barren 2372 22 0 0 1564 52 157 1266 620 6053 26% Forest 289 758 4 6 91,109 16,012 19,539 1085 128,802 71% Grass/Shrub 3756 2756 32 161 21 28,292 1,088,300 159,808 718 1,283,844 85% Agriculture 1828 5994 15 93 3 25,175 82,423 1,215,580 1446 1,332,557 91% Wetlands 755 47 0 0 20 4920 329 2178 5853 14,102 42% Overall Total 32,400 27,218 51 2092 2166 150,611 1,194,244 1,415,072 10,765 86.2% Agreement 69% 59% 0% 82% 72% 60% 91% 86% 54% Agreement Minnesota 27/27 Trends/CCDC Agreement Circa 2000 Land Cover Trends Water Developed Disturbed Mining Barren Forest Grass/Shrub Agriculture Wetlands Total Agreement Water 77,559 104 15 30 0 1831 34 6 1649 81,228 95% Developed 162 4021 553 294 0 2485 416 1368 106 9405 43% Disturbed 59 391 1247 14 0 219 123 74 503 2630 47% Mining 74 93 101 7310 0 517 254 126 11 8486 86% Barren 7 4 0 2 65 6 1 1 3 89 73% Circa 2000 CCDC Forest 4105 1564 47,315 838 0 695,183 22,649 10,375 58,469 8 10 83% Grass/Shrub 3 3 2979 0 0 1957 6278 591 603 12,414 51% Agriculture 58 276 1252 147 0 9491 10,553 63,340 5229 90,346 70% Wetlands 3529 97 9217 97 0 78,168 8427 5115 283,024 73% 4 10 Total 85,556 6553 62,679 8732 65 789,857 48,735 80,996 349,597 Overall 79.4% Agreement 91% 61% 2% 84% 100% 88% 13% 78% 81% Agreement Remote Sens. 2016, 8, 811 12 of 33 Table 5. (a) Forest pixels mapped by LC Trends distributed across CCDC classes for each path/row location and (b) Forest pixels mapped by CCDC distributed across LC Trends classes for each path/row location. AR = Arkansas; AZ = Arizona; CA = California; CO = Colorado; FL = Florida; IL = Illinois; IN = Indiana; KS = Kansas; MN = Minnesota; MS = Mississippi; MO = Montana; ND = North Dakota; NH = New Hampshire; SD = South Dakota; VT = Vermont; WA = Washington. FOREST (a) Trends Forest Pixels/CCDC Classes NH/VT FL IL/IN AR/MS MN KS ND SD CO CO AZ MT CA WA path/row 13/29 16/40 22/33 23/37 27/27 28/33 31/27 33/29 34/33 35/32 36/38 39/26 43/34 46/27 Total Dist. Water 1508 26 3700 860 1831 537 28 0 232 169 2 0 180 2565 11,638 0.2% Developed 4118 2410 2216 741 2485 375 62 0 1058 86 267 0 340 99,507 113,665 1.8% Disturbed 106 632 74 2205 219 53 1 28 18 5 21,986 0 736 4047 30,110 0.5% Mining 1371 3 265 404 517 98 0 0 28 0 1 0 569 1995 5251 0.1% Barren 2154 0 258 774 6 52 0 0 838 37 90 0 1203 6582 11,994 0.2% Forest 940,857 34,988 268,587 352,455 695,183 91,109 5405 100,139 572,227 362,365 38,500 31 275,941 2,043,378 5,781,165 90.5% Grass/Shrub 4988 567 1207 201 1957 28,292 259 1024 36,342 39,980 9501 3 31,394 49,568 205,283 3.2% Agriculture 4843 1166 41,492 22,570 9491 25,175 1035 94 608 87 22 4 1,775 8,105 116,467 1.8% Wetland 5884 4506 6942 3833 78,168 4920 41 62 366 921 0 0 237 6,695 112,575 1.8% Ice & Snow 0 0 0 0 0 0 0 0 0 0 0 0 0 106 106 0.0% Total 965,829 44,298 324,741 384,043 789,857 150,611 6831 101,347 611,717 403,650 70,369 38 312,375 2,222,548 6,388,254 “Producer’s” 97.4% 79.0% 82.7% 91.8% 88.0% 60.5% 79.1% 98.8% 93.5% 89.8% 54.7% 81.6% 88.3% 91.9% 90.5% All p/r (b) CCDC Forest Pixels/Trends Classes Path/Row Water Dev Disturbed Mining Barren Forest Grass/Shrub Ag Wetland Ice & Snow Total “User’s” NH/VT 13/29 1619 8338 7185 1 570 940,857 4797 7269 7370 0 978,006 96.2% FL 16/40 290 1686 7996 13 0 34,988 6310 1218 7005 0 59,506 58.8% IL/IN 22/33 2868 9465 76 208 0 268,587 720 63,014 8875 0 353,813 75.9% AR/MS 23/37 1631 1077 20,914 3 5 352,455 482 35,609 24,344 0 436,520 80.7% MN 27/27 4105 1564 47,315 838 0 695,183 22,649 10,375 58,469 0 840,498 82.7% KS 28/33 289 758 4 6 91,109 16,012 19,539 1085 0 128,802 70.7% ND 31/27 154 67 0 0 0 5405 806 4541 674 0 11,647 46.4% SD 33/29 0 0 694 0 0 100,139 6530 1 1 0 107,365 93.3% CO 34/33 211 3720 8 3 1432 572,227 33,929 388 1395 0 613,313 93.3% CO 35/32 36 0 68 0 78 362,365 41,215 53 838 0 404,653 89.5% AZ 36/38 0 4 0 0 0 38,500 12,669 0 0 0 51,173 75.2% MT 39/26 0 0 0 0 0 31 6 0 0 37 83.8% CA 43/34 186 362 13,228 52 60 275,941 27,007 218 12 0 317,066 87.0% WA 46/27 7654 91,646 71,903 1154 12,690 2,043,378 100,308 19,146 23,476 227 2,371,582 86.2% Total 19,043 118,687 169,391 2278 14,835 5,781,165 273,434 161,377 133,544 227 6,673,981 86.6% Dist. 0.3% 1.8% 2.5% 0.0% 0.2% 86.6% 4.1% 2.4% 2.0% 0.0% All p/r Remote Sens. 2016, 8, 811 13 of 33 Remote Sens. 2016, 8, 811 13 of 38 Forest class agreement was less strong in Kansas (28/33; 60.5% producer ’s and 70.7% user ’s agreement for circa 2000) and Illinois/Indiana (22/33; 83.0% producer ’s and 75.9% user ’s agreement Forest class agreement was less strong in Kansas (28/33; 60.5% producer’s and 70.7% user’s for circa 2000). In both locations forest occurrence was more fragmented, resulting in a much higher agreement for circa 2000) and Illinois/Indiana (22/33; 83.0% producer’s and 75.9% user’s agreement proportion of edge pixels relative to interior area in forest stands (Figure 5). Almost all of the Forest for circa 2000). In both locations forest occurrence was more fragmented, resulting in a much higher class disagreement in these two path/rows occurred along boundaries between Forest and Agriculture proportion of edge pixels relative to interior area in forest stands (Figure 5). Almost all of the Forest or Forclass disagre est and Grass/Shr ement in ub. th Some ese two of the pat disagr h/rows oc eement curred along in these cases boundar was friom es between mixed pixels, Forest and containing Agriculture or Forest and Grass/Shrub. Some of the disagreement in these cases was from mixed some fraction of Agriculture or Grass/Shrub land cover. This was especially common in the Kansas pixels, containing some fraction of Agriculture or Grass/Shrub land cover. This was especially path/row, where forest occurred as long linear features coinciding with the moister, ﬁre-protected common in the Kansas path/row, where forest occurred as long linear features coinciding with the topography along stream courses. The large fraction of edge pixels would have increased the impact moister, fire-protected topography along stream courses. The large fraction of edge pixels would from image misregistration—and we observed some apparent minor image misregistration. Frequently, have increased the impact from image misregistration—and we observed some apparent minor the 60 m minimum mapping unit and greater generalization in the LC Trends data resulted in pixels image misregistration. Frequently, the 60 m minimum mapping unit and greater generalization in classiﬁed as Agriculture or Grass/Shrub where available high-resolution imagery revealed the actual the LC Trends data resulted in pixels classified as Agriculture or Grass/Shrub where available high- land cover to be tree cover. In some cases, disagreement appeared to result from misclassiﬁcation in resolution imagery revealed the actual land cover to be tree cover. In some cases, disagreement the LC Trends blocks. appeared to result from misclassification in the LC Trends blocks. Figure 5. Example of the fragmented tree cover typical of LC Trends blocks in the Illinois/Indiana Figure 5. Example of the fragmented tree cover typical of LC Trends blocks in the Illinois/Indiana (22/33) study location (upper left corner: 39.40966°N. lat., −87.04695°W. long.). Overlain vector layers (22/33) study location (upper left corner: 39.40966 N. lat., 87.04695 W. long.). Overlain vector layers identify locations of areas of Forest/Agriculture confusion, generally occurring at patch edges. identify locations of areas of Forest/Agriculture confusion, generally occurring at patch edges. Generally, even path/rows with only limited forest area provided meaningful information regarding the performance of the CCDC classification process in different landscapes. Forest Generally, even path/rows with only limited forest area provided meaningful information disagreement in Arizona (36/38), for example, occurred primarily as confusion between regarding the performance of the CCDC classiﬁcation process in different landscapes. Forest Grass/Shrub and Forest classes in the drier oak and juniper forested areas in the southernmost LC disagreement in Arizona (36/38), for example, occurred primarily as confusion between Grass/Shrub Trends block. Tree density and forest structure there varied, often along gradients of topography and Forest classes in the drier oak and juniper forested areas in the southernmost LC Trends block. and elevation and along drainages. The spatial transitions between Forest and Grass/Shrub classes Tree density and forest structure there varied, often along gradients of topography and elevation and are often gradual, and it was in these areas that disagreement with LC Trends classes tended to be along drainages. The spatial transitions between Forest and Grass/Shrub classes are often gradual, concentrated. and it was in these areas that disagreement with LC Trends classes tended to be concentrated. Forest class agreement was high at the sites in Colorado (34/33 and 35/32) and California For (43/est 34), wit class h 93 agr .5% p eement roducwas er’s and high 93at .3% user’ the sites s agre inem Colorado ent for 34(34/33 /33, 89.8and % pro 35/32) ducer’s and and 8 California 9.6% user’s agreement for 35/32, and 88.3% producer’s and 87.0% user’s agreement for 43/34. In both (43/34), with 93.5% producer ’s and 93.3% user ’s agreement for 34/33, 89.8% producer ’s and 89.6% Colorado path/rows over 90% of the disagreement was with the CCDC Grass/Shrub class, where user ’s agreement for 35/32, and 88.3% producer ’s and 87.0% user ’s agreement for 43/34. In both Colorado path/rows over 90% of the disagreement was with the CCDC Grass/Shrub class, where disagreement tended to be in areas where gradual transitions from denser to more diffuse tree cover Remote Sens. 2016, 8, 811 14 of 33 created ambiguous spatial boundaries between classes. This often occurred in locations of high relief, often where the difference between tree height and shrub height was slight. In California (43/34) 86.2% of the LC Trends Forest confusion was with CCDC Grass/Shrub class, and in these cases the circumstances were generally similar to those observed for the Colorado sites; the coarser minimum mapping unit and tendency toward generalization in the LC Trends data contributed to the disagreement with the CCDC results. Likewise, when LC Trends pixels disagreed with CCDC Forest pixels it was because LC Trends interpreters had mapped those pixels as Grass/Shrub 65.7% of the time; in these cases, the LC Trends minimum mapping unit and areal generalization accounted for more of the confusion than did the spatially transitional nature of forest cover in 43/34. Agriculture Class Agreement The CCDC and LC Trends Agriculture classes had good overall agreement (88.8% producer ’s and 92.8% user ’s agreement), with the main categories of disagreement being: (1) LC Trends Agriculture confused with CCDC Grass/Shrub; (2) CCDC Agriculture confused with LC Trends Grass/Shrub; and (3) LC Trends Agriculture confused with CCDC Forest. Five of the 14 path/row locations contained 86.4% of the area LC Trends interpreters had classiﬁed as Agriculture. Results for these ﬁve locations ranged from a low of 85.9% producer ’s and 91.2% user ’s agreement in the Kansas path/row (28/33) to a high of 93.2% producer ’s and 94.1% user ’s agreement in North Dakota (31/27) (Table 6a,b). Across all 14 path/rows, 23.7% of the area classiﬁed as agriculture by LC Trends was located in California (43/34), where we observed 87.8% producer ’s and 95.3% user ’s agreement. The single largest category of confusion was where LC Trends interpreters mapped pixels as Agriculture and CCDC mapped them as Grass/Shrub (Table 6a), accounting for 1.4% of the entire study area. The inverse case, where LC Trends interpreters mapped pixels as Grass/Shrub and CCDC mapped them as Agriculture, made up 0.8% of the study area (Table 6b). Both directions of confusion were concentrated in a few path/rows and were heavily concentrated in the Kansas location (28/33), which represented 54.6% of the confusion between LC Trends Agriculture and CCDC Grass/Shrub and 51.1% of the confusion between LC Trends Grass/Shrub and CCDC Agriculture. For the former case, examination of high resolution imagery indicated the actual land cover was Grass/Shrub approximately 80% of the time and ranged in use from rangeland to lightly managed pasture/hayland, with many patches being difﬁcult or impossible to distinguish from rangeland (Figure 6). Other areas where LC Trends labeled pixels as Agriculture and CCDC labeled them as Grass/Shrub occurred in California (23.7%), North Dakota (11.6%), and Montana (8.2%). The overwhelming majority of the confusion across all locations occurred in areas of Grass/Shrub, based on high resolution imagery in TM Google Earth , with some cases showing evidence of haying. A lesser factor contributing to cases of low producer ’s agreement in the Agriculture category was where CCDC labeled pixels as Forest that had been classiﬁed as Agriculture by LC Trends (see previous details in second paragraph under “Forest Class Agreement”). This confusion was heavily concentrated in a few path/rows, including Illinois/Indiana (22/33), Arkansas/Mississippi (23/37), and Kansas (28/33). Confusion between the LC Trends Agriculture class and CCDC Developed class, as well as the inverse, made up 1.1% of the total area of the study, with the latter representing the smaller fraction. Most of this disagreement occurred in the suburban fringes, where developed and agricultural lands were fragmented and intermingled with low-intensity development. These two categories of confusion were heavily concentrated in Washington (46/27: 39.2% of LC Trends Agriculture confused with CCDC Developed and 33.0% of the inverse case) and California (43/34: 34.4% of LC Trends Agriculture confused with CCDC Developed and 32.2% of the inverse case). LC Trends map generalization in these settings often labeled ﬁelds on the order of 250–500 m 250–500 m, which are used for hay and pasture, as Developed. Alternatively, LC Trends generalized buildings and roads in agricultural settings to be labeled as Agriculture, whereas CCDC separated those buildings and roads into the Developed class. Remote Sens. 2016, 8, 811 15 of 33 Table 6. (a) LC Trends circa 2000 Agriculture pixels distributed across CCDC classes for each path/row location and (b) CCDC circa 2000 Agriculture pixels distributed across LC Trends classes for each path/row location. AR = Arkansas; AZ = Arizona; CA = California; CO = Colorado; FL = Florida; IL = Illinois; IN = Indiana; KS = Kansas; MN = Minnesota; MS = Mississippi; MO = Montana; ND = North Dakota; NH = New Hampshire; SD = South Dakota; VT = Vermont; WA = Washington. AGRICULTURE (a) Trends Agriculture Pixels/CCDC Classes NH/VT FL IL/IN AR/MS MN KS ND SD CO CO AZ MT CA WA path/row 13/29 16/40 22/33 23/37 27/27 28/33 31/27 33/29 34/33 35/32 36/38 39/26 43/34 46/27 Total Dist. Water 2 111 3709 2055 6 1519 860 21 26 4 1 0 2519 1074 11,907 0.2% Developed 1342 2898 13,357 4391 1368 5216 3704 0 1096 232 208 181 44,284 50,405 128,682 1.9% Disturbed 8 164 2043 4742 74 3839 1583 13 557 187 1297 4954 86,812 1032 107,305 1.6% Mining 283 12 3831 464 126 6127 754 0 51 0 8 0 1337 707 13,700 0.2% Barren 16 0 197 946 1 1266 0 2 1 54 0 0 12 2495 0.0% Forest 7269 1218 63,014 35,609 10,375 19,539 4541 1 388 53 0 6 218 19,146 161,377 2.4% Grass/Shrub 173 401 4925 1869 591 159,808 34,003 4732 3073 4153 1077 24,078 53,435 355 292,673 4.4% Agriculture 17,042 16,428 1,173,954 657,549 63,340 1,215,580 719,378 8906 47,227 10,349 2,891 378,636 1,379,735 200,775 5,891,790 88.8% Wetland 59 238 4442 2841 5115 2178 6576 0 11 201 0 627 2503 3777 28,568 0.4% Ice & Snow 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0.0% Total 26,194 21,470 1,269,472 710,466 80,996 1,415,072 771,399 13,675 52,429 15,180 5536 408,482 1,570,843 277,283 6,638,497 “Producer’s” 65.1% 76.5% 92.5% 92.6% 78.2% 85.9% 93.3% 65.1% 90.1% 68.2% 52.2% 92.7% 87.8% 72.4% 88.8% All p/r (b) CCDC Agriculture Pixels/Trends Classes Path/Row Water Dev Disturbed Mining Barren Forest Grass/Shrub Ag Wetland Ice & Snow Total “User’s” NH/VT 13/29 35 1316 172 0 0 4843 65 17,042 113 0 23,586 72.3% FL 16/40 122 1936 622 6 0 1166 1181 16,428 550 0 22,011 74.6% IL/IN 22/33 3405 18,420 173 376 0 41,492 1380 1,173,954 8405 0 1,247,605 94.1% AR/MS 23/37 4617 5411 4372 7 7 22,570 1446 657,549 17,762 0 713,741 92.1% MN 27/27 58 276 1252 147 0 9491 10,553 63,340 5229 0 90,346 70.1% KS 28/33 1828 5994 15 93 3 25,175 82,423 1,215,580 1446 0 1,332,557 91.2% ND 31/27 1162 499 0 16 0 1035 19,354 719,378 15,383 0 756,827 95.1% SD 33/29 0 0 0 0 1 94 2061 8906 0 0 11,062 80.5% CO 34/33 14 320 0 3 5 608 1180 47,227 678 0 50,035 94.4% CO 35/32 5 1 0 0 1 87 1823 10,349 289 0 12,555 82.4% AZ 36/38 1 70 0 0 6 22 405 2891 0 0 3395 85.2% MT 39/26 1 13 0 0 0 4 9954 378,636 338 0 388,946 97.3% CA 43/34 2099 31,677 1207 998 0 1775 28,236 1,379,735 1365 0 1,447,092 95.3% WA 46/27 1895 32,531 1355 155 414 8105 1394 200,775 3969 0 250,593 80.1% Total 15,242 98,464 9168 1801 437 116,467 161,455 5,891,790 55,527 0 6,350,351 92.8% Dist. 0.2% 1.6% 0.1% 0.0% 0.0% 1.8% 2.5% 92.8% 0.9% 0.0% All p/r Remote Sens. 2016, 8, 811 16 of 33 Remote Sens. 2016, 8, 811 16 of 38 Figure 6. The red polygons delineate areas of pasture mapped as Agriculture by LC Trends and Figure 6. The red polygons delineate areas of pasture mapped as Agriculture by LC Trends and Grass/Shrub by CCDC. Grass/Shrub by CCDC. Grass/Shrub Class Agreement Grass/Shrub Class Agreement The Grass/Shrub class covered the third largest extent across the study area after Forest and The Grass/Shrub class covered the third largest extent across the study area after Forest and Agriculture, accounting for 22.7% of the LC Trends classification and 22.3% of the CCDC classification. Agriculture, accounting for 22.7% of the LC Trends classiﬁcation and 22.3% of the CCDC classiﬁcation. The Grass/Shrub class had slightly lower agreement than either Forest or Agriculture at 85.0% The Grass/Shrub class had slightly lower agreement than either Forest or Agriculture at 85.0% producer’s and 86.4% user’s agreement. For eight of the 10 path/row test areas where more than 1% producer ’s and 86.4% user ’s agreement. For eight of the 10 path/row test areas where more than 1% of of the area was classified as Grass/Shrub by LC Trends, we observed 82.0% to 94.5% producer’s the area was classiﬁed as Grass/Shrub by LC Trends, we observed 82.0% to 94.5% producer ’s agreement agreement and 74.3% to 98.4% user’s agreement (Table 7a,b). The remaining two path/rows and 74.3% to 98.4% user ’s agreement (Table 7a,b). The remaining two path/rows exhibited low exhibited low Grass/Shrub class agreement, including 12.9% producer’s and 50.6% user’s agreement Grass/Shrub class agreement, including 12.9% producer ’s and 50.6% user ’s agreement in Minnesota in Minnesota (27/27) and 44.3% user’s and 48.7% producer’s agreement in Washington (46/27). Both (27/27) and 44.3% user ’s and 48.7% producer ’s agreement in Washington (46/27). Both LC Trends and LC Trends and CCDC Grass/Shrub classes were most often confused with the counterpart CCDC classiGrass/Shr fication’s Agr ubiclasses culture cl wer ass e(r most efer to often the sconfused econd para with graph the un counterpart der “Agricultu classiﬁcation’s re Class Agree Agr men icultur t”). e class (refer Confusion be to the second tween the Gr paragraph ass/under Shrub an “Agricultur d Forest cla e s Class ses account Agreement”). ed for 2.3% of the entire area of comparison (1.3% LC Trends Grass/Shrub confused with CCDC Forest and 1.0% LC Trends Forest Confusion between the Grass/Shrub and Forest classes accounted for 2.3% of the entire area of confused with CCDC Grass/Shrub). In Washington (46/27) and Minnesota (27/27), 83.5% and 53.4%, comparison (1.3% LC Trends Grass/Shrub confused with CCDC Forest and 1.0% LC Trends Forest respectively, of LC Trends Grass/Shrub class disagreement was where CCDC had classified pixels confused with CCDC Grass/Shrub). In Washington (46/27) and Minnesota (27/27), 83.5% and 53.4%, as Forest. In both locations this disagreement was associated with regenerating forest following respectively, of LC Trends Grass/Shrub class disagreement was where CCDC had classiﬁed pixels as timber harvest. These patches were classified as Forest by CCDC typically within a year of harvest, Forest. In both locations this disagreement was associated with regenerating forest following timber but were considered Grass/Shrub by LC Trends, typically for seven or more years following harvest. harvest. These patches were classiﬁed as Forest by CCDC typically within a year of harvest, but were Grass/Shrub class agreement was higher in the two Colorado path/rows, including 88.7% considered Grass/Shrub by LC Trends, typically for seven or more years following harvest. producer’s and 86.8% user’s agreement in 34/33 and 91.2% producer’s and 86.8% user’s agreement Grass/Shrub class agreement was higher in the two Colorado path/rows, including 88.7% in 35/32. As before, the largest fraction of disagreement was confusion between the Forest and producer ’s and 86.8% user ’s agreement in 34/33 and 91.2% producer ’s and 86.8% user ’s agreement Remote Sens. 2016, 8, 811 17 of 33 in 35/32. As before, the largest fraction of disagreement was confusion between the Forest and Grass/Shrub classes. Areas where LC Trends interpreters classiﬁed pixels as Grass/Shrub and CCDC classiﬁed them as Forest accounted for 74.7% (34/33) and 82.9% (35/32) of the producer ’s disagreement. User ’s disagreement was also predominantly the result of confusion between Grass/Shrub and Forest; areas where CCDC had classiﬁed pixels as Grass/Shrub and LC Trends interpreters classiﬁed them as Forest accounted for 67.3% of the disagreement in 34/33 and 84.6% of the disagreement in 35/32 (see second-to-last paragraph under “Forest Class Agreement”). Developed Class Agreement The Developed class covered 4.8% of the LC Trends pixels and 5.0% of the CCDC pixels in the study area. The Developed classes generally occurred in more complex, fragmented land cover mosaics and had 71.8% producer ’s agreement and 69.0% user ’s agreement. The majority of confusion was with the Forest and Agriculture classes and was concentrated in a few path/row locations (Table 8). Confusion of Developed land with Forest was heavily concentrated in the Washington location (46/27), with 77.2% of cases of LC Trends Developed pixels classiﬁed as Forest by CCDC and 87.5% of cases of the CCDC Developed pixels classiﬁed as Forest by LC Trends. The generalization of land cover features for the LC Trends classiﬁcation often included pixels of pure tree canopy that occurred in the very complex land cover mosaic of low-intensity development around the Puget Sound area of 46/27. CCDC generally classiﬁed these pixels as Forest. The inverse disagreement, where CCDC Developed pixels were classiﬁed as Forest by LC Trends, likewise was often associated with the generalization of the LC Trends classiﬁcation, which had included areas of developed land within larger tracts of Forest. A similar fraction of the Developed/Forest confusion was not caused by the LC Trends generalization, but from mixed pixels in the very fragmented land cover around the Puget Sound. The fragmented land cover mosaic also increased opportunities for image misregistration to contribute to class confusion. The Agriculture class also accounted for a large fraction of the confusion between the CCDC and LC Trends Developed classes. CCDC classiﬁed 10.0% of LC Trends Developed pixels as Agriculture; conversely, LC Trends interpreters classiﬁed 12.6% of the CCDC Developed pixels as Agriculture. In both cases, the vast majority of that confusion was distributed across the Washington (46/27), California (43/34), and Illinois/Indiana (22/33) locations. Pixels where LC Trends interpreters had classiﬁed the land as Agriculture and CCDC had classiﬁed it as Developed were most often associated with land cover that, if deﬁned strictly by cover as opposed to land use or a mixed deﬁnition, was Developed, such as roads, clusters of farm buildings, low intensity residential development and a few commercial/industrial sites. In the California location there were also some cases of bare farmland being classiﬁed as Developed by CCDC. Wetland Class Agreement LC Trends and CCDC mapped 3.8% and 3.6% of the entire study extent as Wetland, respectively; with 70.2% producer ’s and 74.0% user ’s class agreement. CCDC classiﬁed 16.9% of LC Trends Wetland pixels as Forest, 7.0% as Agriculture, 2.5% as Grass/Shrub, and 1.4% as water (Table 9a). Of the pixels classiﬁed as Wetland by CCDC, LC Trends interpreters classiﬁed 15.0% as Forest, 3.8% as Agriculture, and 3.4% as Grass/Shrub (Table 9b). Where CCDC disagreed with the LC Trends Wetland class, it labeled those pixels Forest 57% of the time. Where LC Trends disagreed with the CCDC Wetland class it labeled those pixels Forest 58% of the time. Most Wetland confusion occurred in the path/rows with the most Wetland area (Florida, Illinois/Indiana, Arkansas/Mississippi, Minnesota, and Washington). Minnesota (27/27) had 44.3% of LC Trends and 51.7% of the CCDC Wetland pixels for the entire study extent. The vast majority of wetlands within this path/row were forested and, consequently, most of the disagreement was between Forest and Wetland classes. CCDC mapped 19% of LC Trends Wetland pixels in 27/27 to other classes, primarily forest (87.8%). LC Trends pixels disagreed with 27% of the CCDC Wetland pixels in 27/27, mapping 74.7% of them as Forest. Remote Sens. 2016, 8, 811 18 of 33 Table 7. (a) LC Trends circa 2000 Grass/Shrub pixels distributed across CCDC classes for each path/row location and (b) CCDC circa 2000 Grass/Shrub pixels distributed across LC Trends classes for each path/row location. AR = Arkansas; AZ = Arizona; CA = California; CO = Colorado; FL = Florida; IL = Illinois; IN = Indiana; KS = Kansas; MN = Minnesota; MS = Mississippi; MO = Montana; ND = North Dakota; NH = New Hampshire; SD = South Dakota; VT = Vermont; WA = Washington. GRASS/SHRUB (a) Trends Grass/Shrub Pixels/CCDC Classes NH/VT FL IL/IN AR/MS MN KS ND SD CO CO AZ MT CA WA path/row 13/29 16/40 22/33 23/37 27/27 28/33 31/27 33/29 34/33 35/32 36/38 39/26 43/34 46/27 Total Dist. Water 3 25 63 19 34 3190 2191 1572 51 205 11 2 1537 7 8910 0.2% Developed 75 716 293 14 416 1552 509 0 1148 604 8249 321 9954 12,121 35,972 0.8% Disturbed 15 974 6 31 123 379 92 2979 1534 446 117,790 1249 36,406 522 162,546 3.5% Mining 50 6 114 4 254 1902 283 0 628 18 2 0 1039 311 4611 0.1% Barren 98 0 0 0 1 157 0 7420 3849 2779 2189 0 3536 4616 24,645 0.5% Forest 4797 6310 720 482 22,649 16,012 806 6530 33,929 41,215 12,669 0 27,007 100,308 273,434 5.9% Grass/Shrub 4033 5173 2438 6480 6278 1,088,300 122,322 363,773 354,855 513,857 776,649 100,951 518,106 95,688 3,958,903 85.0% Agriculture 65 1181 1380 1446 10,553 82,423 19,354 2061 1180 1823 405 9954 28,236 1394 161,455 3.5% Wetland 117 495 42 87 8,427 329 2221 443 3076 2600 0 868 5766 741 25,212 0.5% Ice & Snow 0 0 0 0 0 0 0 0 0 0 0 0 0 136 136 0.0% Total 9253 14,880 5056 8563 48,735 1,194,244 147,778 384,778 400,250 563,547 917,964 113,345 631,587 215,844 4,655,824 “Producer’s” 43.6% 34.8% 48.2% 75.7% 12.9% 91.1% 82.8% 94.5% 88.7% 91.2% 84.6% 89.1% 82.0% 44.3% 85.0% All p/r (b) CCDC Grass/Shrub Pixels/Trends Classes path/row Water Dev Disturbed Mining Barren Forest Grass/Shrub Ag Wetland Ice & Snow Total “User’s” NH/VT 13/29 2 334 1178 6 4988 4033 173 46 0 10,760 37.5% FL 16/40 41 133 748 0 0 567 5173 401 677 0 7740 66.8% IL/IN 22/33 175 424 12 26 0 1207 2438 4925 362 0 9569 25.5% AR/MS 23/37 15 4 394 201 6480 1869 655 0 9618 67.4% MN 27/27 3 3 2979 0 0 1957 6278 591 603 0 12,414 50.6% KS 28/33 3756 2756 32 161 21 28,292 1,088,300 159,808 718 0 1,283,844 84.8% ND 31/27 2382 327 5 4 0 259 122,322 34,003 5395 0 164,697 74.3% SD 33/29 119 0 67 0 5838 1024 363,773 4732 2 0 375,555 96.9% CO 34/33 182 5248 1 81 3149 36,342 354,855 3073 5931 0 408,862 86.8% CO 35/32 10 15 33 0 712 39,980 513,857 4153 2355 0 561,115 91.6% AZ 36/38 47 1581 0 29 70 9501 776,649 1077 0 0 788,954 98.4% MT 39/26 24 28 0 0 0 3 100,951 24,078 861 0 125,945 80.2% CA 43/34 620 15,837 1124 470 72 31,394 518,106 53,435 1555 0 622,613 83.2% WA 46/27 26 2217 38,632 112 9728 49,568 95,688 355 213 0 196,539 48.7% Total 7402 28,907 45,205 883 19,596 205,283 3,958,903 292,673 19,373 0 4,578,225 86.5% Dist. 0.2% 0.6% 1.0% 0.0% 0.4% 4.5% 86.5% 6.4% 0.4% 0.0% All p/r Remote Sens. 2016, 8, 811 19 of 33 Table 8. (a) LC Trends Developed pixels distributed across CCDC classes for each path/row location and (b) CCDC Developed pixels distributed across LC Trends classes for each path/row. AR = Arkansas; AZ = Arizona; CA = California; CO = Colorado; FL = Florida; IL = Illinois; IN = Indiana; KS = Kansas; MN = Minnesota; MS = Mississippi; MO = Montana; ND = North Dakota; NH = New Hampshire; SD = South Dakota; VT = Vermont; WA = Washington. DEVELOPED (a) Trends Developed Pixels/CCDC Classes NH/VT FL IL/IN AR/MS MN KS ND SD CO CO AZ MT CA WA path/row 13/29 16/40 22/33 23/37 27/27 28/33 31/27 33/29 34/33 35/32 36/38 39/26 43/34 46/27 Total Dist. Water 15 227 605 71 104 392 37 0 23 0 6 0 1405 1947 4832 0.5% Developed 6795 32,393 44,060 5214 4021 16,170 3294 0 12,553 234 3471 828 170,339 406,045 705,417 71.8% Disturbed 23 283 205 97 391 55 10 0 24 0 1218 1 4158 2758 9223 0.9% Mining 1140 72 2553 117 93 1024 16 0 439 0 5 0 40 8606 14,105 1.4% Barren 68 0 10 20 4 22 0 0 1 0 204 0 0 504 833 0.1% Forest 8338 1686 9465 1077 1564 758 67 0 3720 0 4 0 362 91,646 118,687 12.1% Grass/Shrub 334 133 424 4 3 2756 327 0 5248 15 1581 28 15,837 2217 28,907 2.9% Agriculture 1316 1936 18,420 5411 276 5994 499 0 320 1 70 13 31,677 32,531 98,464 10.0% Wetland 81 466 78 18 97 47 31 0 2 1 0 0 213 1068 2102 0.2% Ice & Snow 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0.0% Total 18,110 37,196 75,820 12,029 6553 27,218 4281 0 22,330 251 6559 870 224,031 547,322 982,570 “Producer’s” 37.5% 87.1% 58.1% 43.3% 61.4% 59.4% 76.9% na 56.2% 93.2% 52.9% 95.2% 76.0% 74.2% 71.8% All p/r (b) CCDC Developed Pixels/Trends Classes Path/Row Water Dev Disturbed Mining Barren Forest Grass/Shrub Ag Wetland Ice & Snow Total “User’s” NH/VT 13/29 41 6795 176 3 7 4118 75 1342 160 0 12,717 53.4% FL 16/40 559 32,393 1151 25 0 2410 716 2898 893 0 41,045 78.9% IL/IN 22/33 229 44,060 76 55 0 2216 293 13,357 301 0 60,587 72.7% AR/MS 23/37 162 5214 3729 1 1 741 14 4391 111 0 14,364 36.3% MN 27/27 162 4021 553 294 0 2485 416 1368 106 0 9405 42.8% KS 28/33 131 16,170 0 8 5 375 1552 5216 34 0 23,491 68.8% ND 31/27 23 3294 4 55 0 62 509 3704 130 0 7781 42.3% SD 33/29 0 0 0 0 0 0 0 0 0 0 0 na CO 34/33 11 12,553 0 2 2 1058 1148 1096 12 0 15,882 79.0% CO 35/32 0 234 0 0 0 86 604 232 113 0 1269 18.4% AZ 36/38 26 3471 0 5 83 267 8249 208 0 0 12,309 28.2% MT 39/26 60 828 0 0 0 0 321 181 57 0 1447 57.2% CA 43/34 459 170,339 675 286 0 340 9954 44,284 121 0 226,458 75.2% WA 46/27 2789 406,045 15,638 3503 2147 99,507 12,121 50,405 2971 0 595,126 68.2% Total 4652 705,417 22,002 4237 2245 113,665 35,972 128,682 5009 0 1,021,881 69.0% Dist. 0.5% 69.0% 2.2% 0.4% 0.2% 11.1% 3.5% 12.6% 0.5% 0.0% All p/r Remote Sens. 2016, 8, 811 20 of 33 Table 9. (a) LC Trends Wetland pixels distributed across CCDC classes for each path/row location and (b) CCDC Wetland pixels distributed across LC Trends classes for each path/row. AR = Arkansas; AZ = Arizona; CA = California; CO = Colorado; FL = Florida; IL = Illinois; IN = Indiana; KS = Kansas; MN = Minnesota; MS = Mississippi; MO = Montana; ND = North Dakota; NH = New Hampshire; SD = South Dakota; VT = Vermont; WA = Washington. WETLAND (a) Trends Wetland Pixels across CCDC Classes NH/VT FL IL/IN AR/MS MN KS ND SD CO CO AZ MT CA WA path/row 13/29 16/40 22/33 23/37 27/27 28/33 31/27 33/29 34/33 35/32 36/38 39/26 43/34 46/27 Total Dist. Water 804 276 1521 1132 1649 592 2710 0 85 486 0 0 302 1838 11,395 1.4% Developed 160 893 301 111 106 34 130 0 12 113 0 57 121 2971 5009 0.6% Disturbed 17 3056 49 469 503 337 186 0 2,105 9 0 58 799 146 7734 1.0% Mining 209 1 7 56 11 80 4 0 52 1 0 0 53 121 595 0.1% Barren 156 0 165 792 3 620 0 0 0 335 0 0 0 105 2176 0.3% Forest 7370 7005 8875 24,344 58,469 1085 674 1 1395 838 0 0 12 23,476 133,544 16.9% Grass/Shrub 46 677 362 655 603 718 5395 2 5931 2355 0 861 1555 213 19,373 2.5% Agriculture 113 550 8405 17,762 5229 1446 15,383 0 678 289 0 338 1365 3969 55,527 7.0% Wetland 18,188 33,424 47,659 84,884 283,024 5853 17,425 160 25,948 5463 0 1347 5590 25,504 554,469 70.2% Ice & Snow 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0.0% Total 27,063 45,882 67,344 130,205 349,597 10,765 41,907 163 36,206 9889 0 2661 9797 58,343 789,822 “Producer’s” 67.2% 72.8% 70.8% 65.2% 81.0% 54.4% 41.6% 98.2% 71.7% 55.2% na 50.6% 57.1% 43.7% 70.2% All p/r (b) CCDC Wetland Pixels/Trends Classes Path/Row Water Dev Disturbed Mining Barren Forest Grass/Shrub Ag Wetland Ice & Snow Total “User’s” NH/VT 13/29 881 81 271 9 5884 117 59 18,188 0 25,490 71.4% FL 16/40 499 466 1,571 0 0 4506 495 238 33,424 0 41,199 81.1% IL/IN 22/33 775 78 0 0 1 6942 42 4442 47,659 0 59,939 79.5% AR/MS 23/37 876 18 205 3833 87 2841 84,884 0 92,744 91.5% MN 27/27 3529 97 9,217 97 0 78,168 8427 5115 283,024 0 387,674 73.0% KS 28/33 755 47 0 0 20 4920 329 2178 5853 0 14,102 41.5% ND 31/27 1943 31 0 0 0 41 2221 6576 17,425 0 28,237 61.7% SD 33/29 0 0 0 0 0 62 443 0 160 0 665 24.1% CO 34/33 27 2 2 0 0 366 3076 11 25,948 0 29,432 88.2% CO 35/32 47 1 0 0 128 921 2600 201 5463 0 9361 58.4% AZ 36/38 0 0 0 0 0 0 0 0 0 0 0 na MT 39/26 6 0 0 0 0 0 868 627 1347 0 2848 47.3% CA 43/34 696 213 3 37 0 237 5766 2503 5590 0 15,045 37.2% WA 46/27 3577 1068 177 21 1364 6695 741 3777 25,504 0 42,924 59.4% Total 13,611 2102 11,446 155 1522 112,575 25,212 28,568 554,469 0 749,660 74.0% Dist. 1.8% 0.3% 1.5% 0.0% 0.2% 15.0% 3.4% 3.8% 74.0% 0.0% All p/r Remote Sens. 2016, 8, 811 21 of 33 Remote Sens. 2016, 8, x FOR PEER REVIEW 21 of 38 Much of the confusion between LC Trends Wetland and CCDC Forest pixels occurred along river Much of the confusion between LC Trends Wetland and CCDC Forest pixels occurred along river channels and other elongated features, as well as boundaries between wetland and forest classes where channels and other elongated features, as well as boundaries between wetland and forest classes where mixed pixels and minor misregistration may have been factors. This was less often the case with CCDC mixed pixels and minor misregistration may have been factors. This was less often the case with CCDC Wetland confusion with LC Trends Forest pixels, where areas of disagreement were clearly associated Wetland confusion with LC Trends Forest pixels, where areas of disagreement were clearly associated with a different interpretation of patches of land cover. In many cases it was difﬁcult to distinguish with a different interpretation of patches of land cover. In many cases it was difficult to distinguish conclusively between forest and forested wetland using visual interpretation of Landsat and high conclusively between forest and forested wetland using visual interpretation of Landsat and high resolution imagery in Google Earth™. Consequently, it was difﬁcult to determine with certainty what resolution imagery in Google Earth™. Consequently, it was difficult to determine with certainty what the the true land cover class should have been. We consulted data from the National Wetlands Inventory true land cover class should have been. We consulted data from the National Wetlands Inventory (NWI) (NWI) [32,34] for additional information on the occurrence of wetlands in the landscape (Figure 7). [32,34] for additional information on the occurrence of wetlands in the landscape (Figure 7). For pixels For pixels labeled by LC Trends as Wetland and by CCDC as Forest, the NWI favored the CCDC Forest labeled by LC Trends as Wetland and by CCDC as Forest, the NWI favored the CCDC Forest classification classiﬁcation 58.4% of the time (i.e., NWI did not classify these pixels as wetland). For pixels classiﬁed 58.4% of the time (i.e., NWI did not classify these pixels as wetland). For pixels classified by LC Trends as by LC Trends as Forest and by CCDC as Wetland, NWI again favored the CCDC interpretation, 60.3% Forest and by CCDC as Wetland, NWI again favored the CCDC interpretation, 60.3% of the time. of the time. Figure 7. Comparison of CCDC and LC Trends Wetland classes with National Wetlands Inventory (NWI) Figure 7. Comparison of CCDC and LC Trends Wetland classes with National Wetlands Inventory data and high resolution imagery. (NWI) data and high resolution imagery. The other concentration of wetlands and Wetland class disagreement was in Arkansas/Mississippi The other concentration of wetlands and Wetland class disagreement was in Arkansas/Mississippi (23/37), accounting for 16.5% of the LC Trends Wetland area and 12.4% of the CCDC Wetland area. (23/37), accounting for 16.5% of the LC Trends Wetland area and 12.4% of the CCDC Wetland area. Producer’s agreement was only 65.2%, with CCDC labeling 18.7% of LC Trends Wetland as Forest and Producer ’s agreement was only 65.2%, with CCDC labeling 18.7% of LC Trends Wetland as Forest and 13.6% of LC Trends Wetland as Agriculture (Table 9). The Wetland class user’s agreement was 91.5%, 13.6% of LC Trends Wetland as Agriculture (Table 9). The Wetland class user ’s agreement was 91.5%, with LC Trends labeling 4.1% of CCDC Wetland pixels as Forest and 3.1% as Agriculture. NWI data with LC Trends labeling 4.1% of CCDC Wetland pixels as Forest and 3.1% as Agriculture. NWI data favored the CCDC interpretation 72.4% of the time in pixels identified by LC Trends as Wetland and by favored the CCDC interpretation 72.4% of the time in pixels identiﬁed by LC Trends as Wetland and Remote Sens. 2016, 8, 811 22 of 33 Remote Sens. 2016, 8, x FOR PEER REVIEW 22 of 33 by CCDC as Forest. NWI favored the CCDC interpretation 67.8% of the time in pixels labeled by LC Trends as Wetland and by CCDC as Agriculture. CCDC as Forest. NWI favored the CCDC interpretation 67.8% of the time in pixels labeled by LC Trends as Wetland and by CCDC as Agriculture. Water Class Agreement Water Class Agreement The Water class had 87.2% producer ’s and 88.9% user ’s agreement. This class accounted for only 2.6% of the LC Trends mapped area and 2.5% of the CCDC mapped area. Five path/rows had less The Water class had 87.2% producer’s and 88.9% user’s agreement. This class accounted for only than 1% of area mapped as water by either CCDC or LC Trends. In path/rows with greater than 1% 2.6% of the LC Trends mapped area and 2.5% of the CCDC mapped area. Five path/rows had less than 1% water, low agreement was concentrated in Illinois/Indiana (22/33), Arkansas (23/37), and Kansas of area mapped as water by either CCDC or LC Trends. In path/rows with greater than 1% water, low (28/33) (Table 10). agreement was concentrated in Illinois/Indiana (22/33), Arkansas (23/37), and Kansas (28/33) (Table 10). The Washington location (46/27) had 41% of the total water mapped by LC Trends or CCDC and The Washington location (46/27) had 41% of the total water mapped by LC Trends or CCDC and had had the largest concentration (25%) of all disagreement, despite a producer ’s agreement of 92.1% and the largest concentration (25%) of all disagreement, despite a producer’s agreement of 92.1% and a user’s a user ’s agreement of 93.3% (Table 10). CCDC disagreement with LC Trends Water was split primarily agreement of 93.3% (Table 10). CCDC disagreement with LC Trends Water was split primarily among among Forest (44.6%), Wetland (20.8%), and Agriculture (11.0%). All three types of disagreement were Forest (44.6%), Wetland (20.8%), and Agriculture (11.0%). All three types of disagreement were concentrated along stream courses (Figure 8) and, to a lesser extent, along small inland water bodies concentrated along stream courses (Figure 8) and, to a lesser extent, along small inland water bodies and and coastal margins of the Puget Sound. Image registration was likely a contributing factor, as well as coastal margins of the Puget Sound. Image registration was likely a contributing factor, as well as mixed mixed pixels, minimum mapping unit, changing water levels, and, perhaps, shifts in water courses in pixels, minimum mapping unit, changing water levels, and, perhaps, shifts in water courses in some some streams. streams. LC Trends interpreters classiﬁed only 6.7% of CCDC Water pixels as other land cover types in the LC Trends interpreters classified only 6.7% of CCDC Water pixels as other land cover types in the Washington path/row, most often as Barren. This confusion occurred almost exclusively at edges of Washington path/row, most often as Barren. This confusion occurred almost exclusively at edges of water water bodies, often within the intertidal zone of the Puget Sound itself, and along stream channels bodies, often within the intertidal zone of the Puget Sound itself, and along stream channels where water where water level variation and changing sandbars and shorelines were likely contributing factors. level variation and changing sandbars and shorelines were likely contributing factors. Figure 8. Green polygons correspond with pixels LC Trends interpreters had classified as Forest and CCDC Figure 8. Green polygons correspond with pixels LC Trends interpreters had classiﬁed as Forest and classified as Water; blue polygons correspond with pixels LC Trends interpreters classified as Water and CCDC classiﬁed as Water; blue polygons correspond with pixels LC Trends interpreters classiﬁed as CCDC classified as Forest. Water and CCDC classiﬁed as Forest. Remote Sens. 2016, 8, 811 23 of 33 Table 10. (a) LC Trends Water pixels distributed across CCDC classes for each path/row location and (b) CCDC Water pixels distributed across LC Trends classes for each path/row. AR = Arkansas; AZ = Arizona; CA = California; CO = Colorado; FL = Florida; IL = Illinois; IN = Indiana; KS = Kansas; MN = Minnesota; MS = Mississippi; MO = Montana; ND = North Dakota; NH = New Hampshire; SD = South Dakota; VT = Vermont; WA = Washington. WATER (a) Trends Water/CCDC Classes NH/VT FL IL/IN AR/MS MN KS ND SD CO CO AZ MT CA WA path/row 13/29 16/40 22/33 23/37 27/27 28/33 31/27 33/29 34/33 35/32 36/38 39/26 43/34 46/27 Total Dist. Water 47,892 41,807 16,232 18,176 77,559 22,263 18,915 1188 3374 1248 57 10 13,606 199,996 462,323 87.2% Developed 41 559 229 162 162 131 23 0 11 0 26 60 459 2789 4652 0.9% Disturbed 2 131 291 1,194 59 947 39 168 0 8 35 0 156 107 3,137 0.6% Mining 16 0 148 44 74 59 1 0 6 0 0 0 64 335 747 0.1% Barren 56 0 529 459 7 2372 0 2 0 22 19 0 33 775 4274 0.8% Forest 1619 290 2868 1631 4105 289 154 0 211 36 0 0 186 7654 19,043 3.6% Grass/Shrub 2 41 175 15 3 3756 2382 119 182 10 47 24 620 26 7402 1.4% Agriculture 35 122 3405 4617 58 1828 1162 0 14 5 1 1 2099 1895 15,242 2.9% Wetland 881 499 775 876 3529 755 1943 0 27 47 0 6 696 3577 13,611 2.6% Ice & Snow 0 0 0 0 0 0 0 0 0 0 0 0 0 5 5 0.0% Total 50,544 43,449 24,652 27,174 85,556 32,400 24,619 1477 3825 1376 185 101 17,919 217,159 530,436 “Producer’s” 94.8% 96.2% 65.8% 66.9% 90.7% 68.7% 76.8% 80.4% 88.2% 90.7% 30.8% 9.9% 75.9% 92.1% 87.2% All p/r (b) CCDC Water Pixels/Trends Classes Path/Row Water Dev Disturbed Mining Barren Forest Grass/Shrub Ag Wetland Ice & Snow Total “User’s” NH/VT 13/29 47,892 15 6 0 0 1508 3 2 804 0 50,230 95.3% FL 16/40 41,807 227 5 0 0 26 25 111 276 0 42,477 98.4% IL/IN 22/33 16,232 605 4 9 11 3700 63 3709 1521 0 25,854 62.8% AR/MS 23/37 18,176 71 21 10 0 860 19 2055 1132 0 22,344 81.3% MN 27/27 77,559 104 15 30 0 1831 34 6 1649 0 81,228 95.5% KS 28/33 22,263 392 0 9 65 537 3190 1519 592 0 28,567 77.9% ND 31/27 18,915 37 0 0 0 28 2191 860 2710 0 24,741 76.5% SD 33/29 1188 0 0 0 92 0 1572 21 0 0 2873 41.4% CO 34/33 3374 23 0 0 1 232 51 26 85 0 3792 89.0% CO 35/32 1248 0 0 0 13 169 205 4 486 0 2125 58.7% AZ 36/38 57 6 0 0 0 2 11 1 0 0 77 74.0% MT 39/26 10 0 0 0 0 2 0 0 0 12 83.3% CA 43/34 13,606 1405 1247 430 0 180 1537 2519 302 0 21,226 64.1% WA 46/27 199,996 1947 3 127 6810 2565 7 1074 1838 0 214,367 93.3% Total 462,323 4832 1301 615 6992 11,638 8910 11,907 11,395 0 519,913 88.9% Dist. 88.9% 0.9% 0.3% 0.1% 1.3% 2.2% 1.7% 2.3% 2.2% 0.0% All p/r Remote Sens. 2016, 8, 811 24 of 33 The Illinois/Indiana path/row (22/33) had less than 5% of the total area mapped in water by both LC Trends and CCDC and had only 65.8% producer ’s agreement and 62.8% user ’s agreement. The Kansas location (28/33) had slightly more water (5.5% of CCDC total water and 6.1% of LC Trends total water), with a producer ’s agreement of 68.7% and a user ’s agreement of 77.9%. In both locations the vast majority of disagreement occurred along stream courses and shorelines of small water bodies. In these cases, LC Trends generalization and larger minimum mapping unit appeared to account for much of the confusion, and image registration may have contributed as well. In a few cases, LC Trends interpreters had classiﬁed agricultural ﬁelds as Water. In Kansas (28/33), confusion between LC Trends Water and CCDC Barren pixels occurred along the channel of the Kansas River, where some areas mapped as exposed sediment by CCDC were included in the LC Trends water class. The closest date of high resolution data in Google Earth™ (16 February 2002) agreed roughly as often with CCDC as with LC Trends. Different dates of high resolution data conﬁrmed the variability of the exposed sediment with water levels, erosion, and deposition. Barren Class Agreement The Barren class covered less than 1% of the entire mapped area in either classiﬁcation and had only 69.1% producer ’s agreement and 68.2% user ’s agreement—among the lowest across classes. The disagreement was heavily concentrated in Washington (46/27), South Dakota (33/29), and Colorado (35/32) (Table 11). Confusion between Barren and Forest was heavily concentrated in the Washington path/row, which had 54.9% of all LC Trends Forest pixels that had been identiﬁed as Barren by CCDC and 85.5% of all LC Trends Barren pixels that had been identiﬁed as Forest by CCDC. Pixels that LC Trends interpreters classiﬁed as Barren, but CCDC called Forest, occurred mostly in extreme terrain at or near treeline and snowline in the LC Trends blocks falling in the northern Cascades. In about half these cases, the pixels represented a mix of cover types, with some combination of bare rock, bare soil, terrain shadow, ground vegetation, and trees. Roughly a third of these pixels were where LC Trends interpreters had misclassiﬁed or generalized tree cover as Barren. At slightly lower elevations, there were cases of pixels identiﬁed by LC Trends as Barren and by CCDC as Forest that occurred along streams where mixed pixels were likely and where LC Trends interpreters had classiﬁed the bare soil and rock of the streams quite liberally, in some instances overlapping obvious tree cover. The inverse confusion, pixels classiﬁed by LC Trends interpreters as Forest and by CCDC as Barren, was highly concentrated in four sample blocks located at high elevations. Most of this confusion occurred in lightly to moderately vegetated rocky areas with few, if any, trees. Confusion between LC Trends Barren and CCDC Grass/Shrub classes occurred almost exclusively in the high-elevation terrain in just ﬁve sample blocks. Most of these pixels were lightly vegetated, with varying mixes of rock and soil in the pixel in most cases. Some additional Barren class disagreement occurred in South Dakota (33/29) and Colorado (34/33). In South Dakota the confusion was almost entirely between the Barren and Grass/Shrub classes. Most pixels were lightly to moderately vegetated, with components of exposed soil and rock. The generalization of LC Trends data appeared to have been a factor in many cases of confusion. In Colorado the majority of the disagreement was between the LC Trends Barren and CCDC Grass/Shrub classes. In most cases where LC Trends interpreters classiﬁed pixels as Barren and CCDC classiﬁed them as Grass/Shrub, the land cover was lightly vegetated, sometimes with scattered trees and rocky understory. Many of the disagreement pixels were at the boundaries of CCDC class patches. Remote Sens. 2016, 8, 811 25 of 33 Table 11. (a) LC Trends Barren pixels distributed across CCDC classes for each path/row location and (b) CCDC Barren pixels distributed across LC Trends classes for each path/row. AR = Arkansas; AZ = Arizona; CA = California; CO = Colorado; FL = Florida; IL = Illinois; IN = Indiana; KS = Kansas; MN = Minnesota; MS = Mississippi; MO = Montana; ND = North Dakota; NH = New Hampshire; SD = South Dakota; VT = Vermont; WA = Washington. BARREN (a) Trends Barren Pixels/Trends Classes NH/VT FL IL/IN AR/MS MN KS ND SD CO CO AZ MT CA WA path/row 13/29 16/40 22/33 23/37 27/27 28/33 31/27 33/29 34/33 35/32 36/38 39/26 43/34 46/27 Total Dist. Water 0 0 11 0 0 65 0 92 1 13 0 0 0 6810 6992 4.0% Developed 7 0 0 1 0 5 0 0 2 0 83 0 0 2147 2245 1.3% Disturbed 0 0 1 1 0 488 0 39 1 8 388 0 0 43 969 0.6% Mining 59 0 1 2 0 0 0 0 68 0 0 0 0 1293 1423 0.8% Barren 1826 0 159 562 65 1564 0 46,169 12,185 3275 965 0 2643 51,056 120,469 69.1% Forest 570 0 0 5 0 0 0 1432 78 0 0 60 12,690 14,835 8.5% Grass/Shrub 6 0 0 0 0 21 0 5838 3149 712 70 0 72 9728 19,596 11.2% Agriculture 0 0 0 7 0 3 0 1 5 1 6 0 0 414 437 0.3% Wetland 9 0 1 0 0 20 0 0 0 128 0 0 0 1364 1522 0.9% Ice & Snow 0 0 0 0 0 0 0 0 0 0 0 0 0 5863 5863 3.4% Total 2477 0 173 578 65 2166 0 52,139 16,843 4215 1512 0 2,775 91,408 174,351 “Producer’s” 9.0% na 91.9% 97.2% 100.0% 72.2% na 88.5% 72.3% 77.7% 63.8% na 95.2% 55.9% 69.1% All p/r (b) CCDC Barren Pixels/Trends Classes Path/Row Water Dev Disturbed Mining Barren Forest Grass/Shrub Ag Wetland Ice & Snow Total “User’s” NH/VT 13/29 56 68 18 1826 2154 98 16 156 0 4392 41.6% FL 16/40 0 0 0 0 0 0 0 0 0 0 0 na IL/IN 22/33 529 10 0 0 159 258 0 197 165 0 1318 12.1% AR/MS 23/37 459 20 11 1 562 774 0 946 792 0 3565 15.8% MN 27/27 7 4 0 2 65 6 1 1 3 0 89 73.0% KS 28/33 2372 22 0 0 1564 52 157 1266 620 0 6053 25.8% ND 31/27 0 0 0 0 0 0 0 0 0 0 0 na SD 33/29 2 0 0 0 46,169 0 7420 2 0 0 53,593 86.1% CO 34/33 0 1 0 0 12,185 838 3849 0 0 0 16,873 72.2% CO 35/32 22 0 0 0 3275 37 2779 1 335 0 6449 50.8% AZ 36/38 19 204 0 0 965 90 2189 54 0 0 3521 27.4% MT 39/26 0 0 0 0 0 0 0 0 0 0 0 na CA 43/34 33 0 22 0 2643 1203 3536 0 0 0 7437 35.5% WA 46/27 775 504 0 17 51,056 6582 4616 12 105 9701 73,368 69.6% Total 4274 833 51 20 120,469 11,994 24,645 2495 2176 9701 176,658 68.2% Dist. 2.4% 0.5% 0.0% 0.0% 68.2% 6.8% 14.0% 1.4% 1.2% 5.5% All p/r Remote Sens. 2016, 8, 811 26 of 33 Mining and Ice-Snow Classes Agreement The Mining class accounted for only 0.2% of the LC Trends mapped area and 0.3% of the CCDC mapped area. Mining class producer ’s agreement across all path/rows was 70.1%. However, CCDC mapped 85% more mining pixels than did LC Trends, and user ’s agreement across all path/rows was only 38.0%. Confusion between developed and mining classes was the largest category of disagreement. Over 20% of the CCDC Mining pixels were classiﬁed as Developed by LC Trends, and 11.2 % of the LC Trends Mining pixels were classiﬁed as Developed by CCDC. Another 19.6% of the CCDC Mining pixels were mapped as Agriculture by LC Trends. The lowest rate of producer ’s agreement (56.2%) was in Washington (46/27) (Table 12a), which accounted for 83% of the confusion between LC Trends Mining pixels and CCDC Developed pixels. The Mining class user ’s agreement for 46/27 was only 31.5% (Table 12b), with over 60% of all confusion between CCDC Mining pixels and LC Trends Developed pixels occurring in that path/row. User ’s agreement for Mining was below 50% for all path/rows having more than 125 pixels of the CCDC Mining class, with the exceptions of California (43/34) (60.6%) and Minnesota (27/27) (86.1%). The Ice and Snow class accounted for only 0.4% of the LC Trends mapped area and 0.3% of the CCDC mapped area and only occurred in Washington (46/27). Producer ’s agreement was 83% and user ’s agreement was 90.8%, with confusion between the Ice and Snow and Barren classes accounting for most disagreement. Disturbed Class Agreement LC Trends and CCDC Disturbed classes agreed in only a small minority of cases (6.1% producer ’s agreement and 5.0% user ’s agreement). When all path/rows were summarized together, LC Trends interpreters mapped 21.9% more area as disturbed than did CCDC. Summarized by path/row the differences appear extreme (Figure 4 and Table 13a,b). In the Washington path/row, LC Trends interpreters mapped nine times more area as Disturbed, with LC Trends identifying 3.5% of area as Disturbed and CCDC identifying only 0.39% as Disturbed. In the California path/row, CCDC mapped seven times more area as Disturbed than did LC Trends. For Arizona, CCDC mapped 14.2% of the map as disturbed, but LC Trends did not identify any area as disturbed. Areas where LC Trends mapped Disturbed and CCDC disagreed were heavily concentrated, with 85% of the cases occurring in just three path/rows (Washington—46/27, Minnesota—27/27, and Arkansas—23/37). Almost all of this disagreement occurred in forest harvest footprints, which in many cases were several years old. Areas where CCDC mapped Disturbed and LC Trends disagreed were also highly concentrated, with over 85% occurring in just two path/rows (Arizona—36/38 and California—43/34). In Arizona 83% of this was where LC Trends had mapped Grass/Shrub and CCDC mapped Disturbed. In California most of the LC Map disagreement with CCDC Disturbed was classiﬁed as Agriculture (67%). Remote Sens. 2016, 8, 811 27 of 33 Table 12. (a) LC Trends Mining pixels distributed across CCDC classes for each path/row location and (b) CCDC Mining pixels distributed across LC Trends classes for each path/row. AR = Arkansas; AZ = Arizona; CA = California; CO = Colorado; FL = Florida; IL = Illinois; IN = Indiana; KS = Kansas; MN = Minnesota; MS = Mississippi; MO = Montana; ND = North Dakota; NH = New Hampshire; SD = South Dakota; VT = Vermont; WA = Washington. MINING (a) Trends Mining Pixels/CCDC Classes NH/VT FL IL/IN AR/MS MN KS ND SD CO CO AZ MT CA WA path/row 13/29 16/40 22/33 23/37 27/27 28/33 31/27 33/29 34/33 35/32 36/38 39/26 43/34 46/27 Total Dist. Water 0 0 9 10 30 9 0 0 0 0 0 0 430 127 615 1.6% Developed 3 25 55 1 294 8 55 0 2 0 5 0 286 3503 4237 11.2% Disturbed 0 3 135 1 14 103 29 0 5 0 0 0 522 486 1298 3.4% Mining 595 215 2822 379 7310 1712 364 0 981 57 108 0 4817 7155 26,515 70.1% Barren 0 0 0 1 2 0 0 0 0 0 0 0 0 17 20 0.1% Forest 1 13 208 3 838 6 0 0 3 0 0 0 52 1154 2278 6.0% Grass/Shrub 0 0 26 0 0 161 4 0 81 0 29 0 470 112 883 2.3% Agriculture 0 6 376 7 147 93 16 0 3 0 0 0 998 155 1801 4.8% Wetland 0 0 0 0 97 0 0 0 0 0 0 0 37 21 155 0.4% Ice & Snow 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0.0% Total 599 262 3631 402 8732 2092 468 0 1075 57 142 0 7612 12,730 37,802 “Producer’s” 99.3% 82.1% 77.7% 94.3% 83.7% 81.8% 77.8% NA 91.3% 100.0% 76.1% NA 63.3% 56.2% 70.1% All p/r (b) CCDC Mining Pixels/Trends Classes Path/Row Water Dev Disturbed Mining Barren Forest Grass/Shrub Ag Wetland Ice & Snow Total “User’s” NH/VT 13/29 16 1140 46 595 59 1371 50 283 209 0 3769 15.8% FL 16/40 0 72 129 215 0 3 6 12 1 0 438 49.1% IL/IN 22/33 148 2553 76 2822 1 265 114 3831 7 0 9817 28.7% AR/MS 23/37 44 117 259 379 2 404 4 464 56 0 1729 21.9% MN 27/27 74 93 101 7310 0 517 254 126 11 0 8486 86.1% KS 28/33 59 1024 0 1712 0 98 1902 6127 80 0 11,002 15.6% ND 31/27 1 16 0 364 0 0 283 754 4 0 1422 25.6% SD 33/29 0 0 0 0 0 0 0 0 0 0 0 na CO 34/33 6 439 0 981 68 28 628 51 52 0 2253 43.5% CO 35/32 0 0 0 57 0 0 18 0 1 0 76 75.0% AZ 36/38 0 5 0 108 0 1 2 8 0 0 124 87.1% MT 39/26 0 0 0 0 0 0 0 0 0 0 0 na CA 43/34 64 40 29 4817 0 569 1039 1337 53 0 7948 60.6% WA 46/27 335 8606 2189 7155 1293 1995 311 707 121 0 22,712 31.5% Total 747 14,105 2829 26,515 1423 5251 4611 13,700 595 0 69,776 38.0% Dist. 1.1% 20.2% 4.1% 38.0% 2.0% 7.5% 6.6% 19.6% 0.9% na All p/r Remote Sens. 2016, 8, 811 28 of 33 Table 13. (a) LC Trends Disturbed pixels distributed across CCDC classes for each path/row location and (b) CCDC Disturbed pixels distributed across LC Trends classes for each path/row. AR = Arkansas; AZ = Arizona; CA = California; CO = Colorado; FL = Florida; IL = Illinois; IN = Indiana; KS = Kansas; MN = Minnesota; MS = Mississippi; MO = Montana; ND = North Dakota; NH = New Hampshire; SD = South Dakota; VT = Vermont; WA = Washington. DISTURBED (a) Trends Disturbed Pixels/CCDC Classes NH/VT FL IL/IN AR/MS MN KS ND SD CO CO AZ MT CA WA path/row 13/29 16/40 22/33 23/37 27/27 28/33 31/27 33/29 34/33 35/32 36/38 39/26 43/34 46/27 Total Dist. Water 6 5 4 21 15 0 0 0 0 0 0 0 1247 3 1301 0.5% Developed 176 1151 76 3729 553 0 4 0 0 0 0 0 675 15,638 22,002 7.9% Disturbed 125 1828 70 7124 1247 0 1 4 0 0 0 0 907 5782 17,088 6.1% Mining 46 129 76 259 101 0 0 0 0 0 0 0 29 2189 2829 1.0% Barren 18 0 0 11 0 0 0 0 0 0 0 0 22 0 51 0.0% Forest 7185 7996 76 20,914 47,315 4 0 694 8 68 0 0 13,228 71,903 169,391 60.8% Grass/Shrub 1178 748 12 394 2979 32 5 67 1 33 0 0 1124 38,632 45,205 16.2% Agriculture 172 622 173 4372 1252 15 0 0 0 0 0 0 1207 1355 9168 3.3% Wetland 271 1,571 0 205 9217 0 0 0 2 0 0 0 3 177 11,446 4.1% Ice & Snow 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0.0% Total 9177 14,050 487 37,029 62,679 51 10 765 11 101 0 0 18,442 135,679 278,481 “Producer’s” 1.4% 13.0% 14.4% 19.2% 2.0% 0.0% 10.0% 0.5% 0.0% 0.0% na na 4.9% 4.3% 6.1% All p/r (b) CCDC Disturbed Pixels/Trends Classes path/row Water Dev Disturbed Mining Barren Forest Grass/Shrub Ag Wetland Ice & Snow Total “User’s” NH/VT 13/29 2 23 125 106 15 8 17 0 296 42.2% FL 16/40 131 283 1,828 3 0 632 974 164 3056 0 7,071 25.9% IL/IN 22/33 291 205 70 135 1 74 6 2043 49 0 2,874 2.4% AR/MS 23/37 1194 97 7124 1 1 2,205 31 4742 469 0 15,864 44.9% MN 27/27 59 391 1247 14 0 219 123 74 503 0 2,630 47.4% KS 28/33 947 55 0 103 488 53 379 3839 337 0 6,201 0.0% ND 31/27 39 10 1 29 0 1 92 1583 186 0 1,941 0.1% SD 33/29 168 0 4 0 39 28 2979 13 0 0 3,231 0.1% CO 34/33 0 24 0 5 1 18 1534 557 2105 0 4,244 0.0% CO 35/32 8 0 0 0 8 5 446 187 9 0 663 0.0% AZ 36/38 35 1218 0 0 388 21,986 117,790 1297 0 0 142,714 0.0% MT 39/26 0 1 0 0 0 0 1249 4954 58 0 6,262 0.0% CA 43/34 156 4158 907 522 0 736 36,406 86,812 799 0 130,496 0.7% WA 46/27 107 2758 5782 486 43 4047 522 1032 146 52 14,975 38.6% Total 3137 9223 17,088 1298 969 30,110 162,546 107,305 7734 52 339,462 5.0% Dist. 0.9% 2.7% 5.0% 0.4% 0.3% 8.9% 47.9% 31.6% 2.3% 0.0% All p/r Remote Sens. 2016, 8, 811 29 of 33 4. Discussion We undertook this analysis to assess the performance of a continuous change-detection algorithm for mapping thematic land cover across a variety of landscape settings and to evaluate the efﬁcacy of applying data from a national study of land cover, LC Trends, to train the classiﬁer in preparation for operational continuous monitoring of land change. We subjectively selected 14 path/row areas that offered different types of challenges for mapping land cover and made opportunistic use of an existing, high-quality land cover dataset to characterize results. We intended to beneﬁt both from the actual analysis as well as from gaining familiarity with the workﬂow that will be needed to support eventual operations. The number of sample blocks (and therefore area of comparison) available for each of the 14 test path/rows varied greatly because the samples were originally selected based on ecoregion strata for the LC Trends project, rather than on Landsat path/rows. For example, the Florida path/row (16/40) had only two sample blocks, accounting for only 1.1% of the total area in this study, but the Washington path/row (46/27) had 36 sample blocks and represented 18.7% of the study area. Our results do not provide a statistical description of either error in the CCDC land cover or of the relation between Land Cover LC Trends and CCDC annual land cover outside of the areas compared. The results will, however, feed back into understanding the suitability of LC Trends data as a source for training data and the types of confusion that may be introduced by the LC Trends classiﬁcation scheme, minimum mapping unit, level of mapping generalization, and contrasting interpretation approach. We found good consistency in map results across time periods for all but two (Arizona and California) study areas (Table 3). Rates of agreement between LC Trends and CCDC maps varied geographically (ranging from 75% to 94% in 2000 and 77% to 98% in 1992), but nine of 14 path/rows had rates exceeding 86% agreement (Table 3). At the class level, we observed that Forest, Agriculture, Grass/Shrub, and Water had the highest rates of agreement (all >87% for both producer ’s and user ’s agreement) between LC Trends and CCDC maps pooled across study areas, but typically showed greater rates of agreement in study areas where the classes occupied appreciable portions of the landscape within the LC Trends blocks. The most important ﬁnding was that CCDC’s automated, efﬁcient, and repeatable approach was able to reproduce results obtained through the LC Trends project’s lengthy manual image interpretation process 86% of the time without any post-classiﬁcation reﬁnement while using the existing LC Trends dataset to guide the selection of training data. We initially questioned the suitability of the LC Trends data for training CCDC’s classiﬁer, as LC Trends data followed a classiﬁcation scheme that included some land use characteristics in its class deﬁnitions, and the LC Trends methods relied on analyst interpretation and a 60 60 m minimum mapping unit (the combination of which resulted in spatial generalization of land cover patches). However, the good overall agreement of CCDC annual land cover with LC Trends maps across the 14 path/rows we studied suggests that the national set of LC Trends data can provide an adequate source of training information to enable CCDC to generate wall-to-wall thematic land cover for the conterminous United States for the 1985 to current Landsat record. We also note that in areas of disagreement, ancillary information indicated that CCDC often made the better choice of class labels. Conversely, where classes only made up a small portion of the LC Trends blocks in a given path/row we observed several issues. For training, a minimum of 600 pixels for any class was found to provide the best classiﬁcation result [30]. For several of the 14 path/rows the limited number of LC Trends blocks available did not provide class populations of 600 pixels for one or more classes (for example, see Table 10a for Montana and Arizona). Furthermore, where classes were represented in small fragmented patches, registration and spatial generalization were observed to be a potential problem due to selection of some pixels representing other-than-intended classes within the training data. These small patches also impacted the CCDC/LC Trends comparison process, sometimes measuring disagreement that was the result of misregistration rather than misclassiﬁcation. The effect of misregistration error on training data selection, which had resulted from reprojection of the LC Remote Sens. 2016, 8, 811 30 of 33 Trends blocks from Albers to UTM, will be eliminated in operational LCMAP classiﬁcation when the input Landsat data will be processed to the USGS Albers/NAD83 grid, the native projection of the LC Trends blocks. To address the underrepresentation of small classes in the training data, we have since expanded the collection of training data to LC Trends blocks beyond the area being classiﬁed. Preliminary results suggest that this helps in meeting the 600 pixel minimum and better represents landscape variability near the edges of the area being classiﬁed. Collection of training pixels from within LC Trends blocks available in a window including and surrounding the area being classiﬁed has so far produced superior classiﬁcation for the Puget Sound, our initial test ground. Another key ﬁnding was the incompatibility of the methods by which LC Trends and CCDC interpreted the Disturbed class. For example, LC Trends land cover was mapped at intervals of 6 to 8 years, and interpreters showed a strong tendency to label pixels as disturbed long after the actual event had occurred to make certain the disturbance was recorded. CCDC only labeled disturbance for the brief intervals in which a time series model could not be ﬁtted following an abrupt change in land cover, and only if this interval overlapped the deﬁned anniversary date of 1 July. This interval often lasted for a period of months, rather than years. CCDC products will be generated annually, and a new formulation being evaluated labels Disturbance in the annual land cover map regardless of the date the change occurred within the year, overcoming the problem posed by selecting a speciﬁc anniversary date to survey for disturbance and removing much of the disagreement between CCDC and Trends Disturbance classes. Cumulating change across multiple annual maps output by CCDC then will produce results more comparable with those mapped by LC Trends interpreters across their multiyear mapping intervals. A second source of incompatibility between the Disturbance classes resulted from CCDC’s sensitivity to change, including changes in land surface condition where the land cover type did not actually change. For example, sequences such as Grass/Shrub to Disturbed to Grass/Shrub were observed in Arizona and California, where the Disturbed interval was apparently caused by multiyear wet or dry periods that created a measurable shift in vegetation response with no removal of vegetation cover or change in vegetation type. Such changes in vegetation condition were not recorded by LC Trends. There were two factors that led to CCDC identifying these changes in vegetation condition as “disturbance”. First, changes in condition caused legitimate breaks in the time series response trajectories, but there was no means to distinguish breaks caused by shifts in condition with breaks caused by changes in cover type. Recent reﬁnements to the CCDC algorithm are incorporating steps to ﬁlter changes in land cover condition from changes in type so that the latter can be better isolated for mapping changes in thematic cover. Second, unlike other class types, the Disturbance class was not trained for classiﬁcation with Random Forest; it was instead deﬁned by breaks in the time series models, as described in Section 2.1.2 (see also Figure 1). Training data now are being developed so that Random Forest can be used to classify disturbance directly. We found that the confusion between the Grass/Shrub class and the Agriculture class appeared to be due mostly to land use characteristics embedded in the LC Trends class deﬁnitions. The Trends Agriculture class deﬁnition includes “. . . cultivated and uncultivated croplands, haylands, [and] pasture . . . ,“ which in many cases led to capture by image interpreters of hayland and pasture that were spectrally indistinguishable from more natural grassland, particularly in Kansas (28/33). This land use distinction could be made because analysts employed a variety of contextual clues. CCDC generally mapped these areas of lightly managed hayland and pasture to the Grass/Shrub class. Redeﬁnition of the Agriculture class to exclude natural grassland that is lightly grazed or occasionally hayed might be suggested by this ﬁnding. Another ﬁnding related to how successional stages were handled for vegetation stands. The most obvious and widespread case we observed was the difference in how CCDC and LC Trends handled the recovery of forest following clearcut harvest. LC Trends interpreters distinguished early stages of this progression as Grass/Shrub before trees became dominant, then labeled the pixels as Forest once trees regained dominance. In comparison, CCDC ﬁt models to the full length of a time series Remote Sens. 2016, 8, 811 31 of 33 between the periods of abrupt change, then fed the coefﬁcients from these models to the classiﬁer. The coefﬁcients therefore represented a long-term forest trajectory, rather than the individual stand stages along the trajectory, and the resulting thematic label ended up as Forest. This ﬁnding prompted modifying the algorithm to run a separate classiﬁcation each year to enable the classiﬁer to focus on the evolving stand structure through time. The problem is further being addressed by developing training data indicative of early successional stages of forest recovery. We observed that the grass and shrub cover in these post-disturbance stands have different spectral characteristics from areas of perpetual (or long-term) grass/shrub and therefore require separate, representative training data. Results from our comparisons corroborated the expected difﬁculty in classifying woody wetlands or wetlands obscured by tree canopies. Our evaluation of Wetland class agreement was less certain in areas of Forested wetlands. Visual interpretation of forested wetlands was hindered where direct observation of ﬂooding conditions or speciﬁc wetland vegetation was obscured by tree cover. We compared areas of class confusion with data from the National Wetlands Inventory to augment our evaluation of Wetland class agreement. Although this informed our perspective, we note that NWI data also were used as an ancillary reference source by LC Trends analysts and are a component of the Wetland Potential Index (WPI) layer that was used as an ancillary input to the CCDC Random Forest classiﬁcation process. The WPI is a categorical ranking-index map generated based on convergence of evidence from information in the National Land Cover Database 2006 map [31], NWI data [32], and Soil Survey Geographic (SSURGO) hydric soils maps [33]. This complicates any conclusions we might draw from this comparison. CCDC annual land cover products should be evaluated with an independent dataset developed speciﬁcally to determine the accuracy of the thematic outputs, rather than only quantifying the level of agreement with another product. An independent evaluation is planned for the next stage of development towards operational continuous monitoring of land cover. 5. Conclusions We found 86% agreement between thematic land cover maps generated from two very different approaches applied with Landsat data, one based on manual interpretation of individual time periods spaced at 6- to 8-year intervals (LC Trends) and one based on automated interpretation of mathematical models constructed with dense time series of all available clear observations (CCDC). This agreement did not necessarily reﬂect the accuracy of the CCDC annual land cover maps, but rather the agreement between results from the two approaches encompassing the footprint of the 186 sample blocks used in this study. We observed consistency in results across time and across study areas of similar landscape types, and found relatively high levels of agreement for land cover classes that were well represented in the training data. Examination of the land cover associated with areas of disagreement suggested that, despite LC Trends classes being somewhat generalized and often incorporating contextual components of land use into class deﬁnitions, the annual land cover maps generated by CCDC generally differed from the LC Trends classiﬁcation in ways that were not problematic. For example, where LC Trends generalized highly fragmented and geometrically complex land cover features, CCDC adhered to the spatial detail represented in the spectral characteristics—often successfully. Whether CCDC was more accurate than the LC Trends data was not made clear by this analysis. Comparison with independent reference data will begin to address this question and is planned for the next stage of evaluation of CCDC annual land cover products. These efforts will help move the USGS towards operational implementation of a continuous monitoring capability. Supplementary Materials: The following are available online at www.mdpi.com/2072-4292/8/10/811/s1, Table S1: Trends/CCDC Agreement circa 2000, path 13 row 29, Table S2: Trends/CCDC Agreement circa 1992 path 13 row 29, Table S3: Trends/CCDC Agreement circa 2000, path 16 row 40, Table S4. Trends/CCDC Agreement circa 1992, path 16 row 40, Table S5. Trends/CCDC Agreement circa 2000, path 23 row 37, Table S6. Trends/CCDC Agreement circa 1992, path 23 row 37, Table S7. Trends/CCDC Agreement circa 2000, path 23 row 37, Table S8. Trends/CCDC Agreement circa 1992, path 23 row 37, Table S9. Trends/CCDC Agreement circa 2000, path 27 row 27, Table S10. Trends/CCDC Agreement circa 1992, path 27 row 27, Table S11. Trends/CCDC Remote Sens. 2016, 8, 811 32 of 33 Agreement circa 1986, path 27 row 27, Table S12. Trends/CCDC Agreement circa 2000, path 28 row 33, Table S13. Trends/CCDC Agreement circa 1992, path 28 row 33, Table S14. Trends/CCDC Agreement circa 2000, path 31 row 27, Table S15. Trends/CCDC Agreement circa 1992, path 31 row 27, Table S16. Trends/CCDC Agreement circa 2000, path 33 row 29, Table S17. Trends/CCDC Agreement circa 1992, path 33 row 29, Table S18. Trends/CCDC Agreement circa 2000, path 34 row 33, Table S19. Trends/CCDC Agreement circa 1992, path 34 row 33, Table S20. Trends/CCDC Agreement circa 2000, path 35 row 32, Table S21. Trends/CCDC Agreement circa 1992, path 35 row 32, Table S22. Trends/CCDC Agreement circa 2000, path 36 row 38, Table S23. Trends/CCDC Agreement circa 1992, path 36 row 38, Table S24. Trends/CCDC Agreement circa 2000, path 39 row 26, Table S25. Trends/CCDC Agreement circa 1992, path 39 row 26, Table S26. Trends/CCDC Agreement circa 2000, path 43 row 34, Table S27. Trends/CCDC Agreement circa 1992, path 43 row 34, Table S28. Trends/CCDC Agreement circa 2000, path 46 row 27, Table S29. Trends/CCDC Agreement circa 1992, path 46 row 27, Table S30. Trends/CCDC Agreement circa 1986, path 46 row 27, Table S31. Trends/CCDC Agreement summary for all comparison blocks in all path/rows, circa 2000, Table S32. Trends/CCDC Agreement summary for all comparison blocks in all path/rows, circa 1992. Acknowledgments: This work was supported with funding from the USGS Land Remote Sensing Program and the USGS LandCarbon Programs, partially under USGS contracts G15PC00012 (B.P. and D.D.) and G13PC00028 (Z.Z.). Any use of trade, ﬁrm, or product names is for descriptive purposes only and does not imply endorsement by the U.S. Government. Author Contributions: Bruce Pengra analyzed the data and wrote the majority of the manuscript. Alisa L. Gallant conceived the comparison and contributed to the manuscript. Zhe Zhu ran the CCDC algorithm to provide the output land cover maps and contributed to the manuscript. Devendra Dahal prepared training data and assisted with the analysis. Conﬂicts of Interest: The authors declare no conﬂicts of interest. References 1. Homer, C.; Dewitz, J.; Fry, J.; Coan, M.; Hossain, N.; Larson, C.; Herold, N.; McKerrow, A.; VanDriel, J.N.; Wickham, J. Completion of the 2001 National Land Cover Database for the counterminous United States. Photogramm. Eng. Remote Sens. 2007, 73, 337. 2. Bounoua, L.; DeFries, R.; Collatz, G.J.; Sellers, P.; Khan, H. Effects of land cover conversion on surface climate. Clim. Chang. 2002, 52, 29–64. [CrossRef] 3. Running, S.W. Ecosystem disturbance, carbon, and climate. Science 2008, 321, 652–653. [CrossRef] [PubMed] 4. Foley, J.A.; DeFries, R.; Asner, G.P.; Barford, C.; Bonan, G.; Carpenter, S.R.; Chapin, F.S.; Coe, M.T.; Daily, G.C.; Gibbs, H.K. Global consequences of land use. Science 2005, 309, 570–574. [CrossRef] [PubMed] 5. Turner, W.; Rondinini, C.; Pettorelli, N.; Mora, B.; Leidner, A.K.; Szantoi, Z.; Buchanan, G.; Dech, S.; Dwyer, J.; Herold, M. Free and open-access satellite data are key to biodiversity conservation. Biol. Conserv. 2015, 182, 173–176. [CrossRef] 6. Zhu, Z.; Woodcock, C.E. Continuous change detection and classiﬁcation of land cover using all available Landsat data. Remote Sens. Environ. 2014, 144, 152–171. [CrossRef] 7. Loveland, T.; Sohl, T.; Stehman, S.; Gallant, A.; Sayler, K.; Napton, D. A strategy for estimating the rates of recent United States land-cover changes. Photogramm. Eng. Remote Sens. 2002, 68, 1091–1099. 8. Sleeter, B.M.; Wilson, T.S.; Acevedo, W. Status and Trends of Land Change in the Western United States—1973 to 2000; U.S. Geological Survey Professional Paper 1794A; US Geological Survey: Menlo Park, CA, USA, 2012; p. 324. 9. Taylor, J.L.; Acevedo, W.; Auch, R.F.; Drummond, M.A. Status and Trends of land Change in the Great Plains of the United States—1973 to 2000; U.S. Geological Survey Professional Paper 1794B; U.S. Geological Survey: Reston, VA, USA, 2015; p. 190. 10. Auch, R.F.; Karstensen, K.A. Status and Trends of Land Change in the Midwest-South Central United States—1973 to 2000; U.S. Geological Survey Professional Paper 1794C; U.S. Geological Survey: Reston, VA, USA, 2015; p. 200. 11. Soulard, C.E.; Acevedo, W.; Auch, R.F.; Sohl, T.L.; Drummond, M.A.; Sleeter, B.M.; Sorenson, D.G.; Kambly, S.; Wilson, T.S.; Taylor, J.L. Land Cover Trends Dataset, 1973–2000; U.S. Geological Survey: Reston, VA, USA, 2014. 12. Anderson, J.R. A Land Use and Land Cover Classiﬁcation System for Use with Remote Sensor Data; US Government Printing Ofﬁce: Washington, DC, USA, 1976. 13. U.S. Geological Survey. Land Cover Trends Project Classiﬁcation System. Available online: http://landcovertrends.usgs.gov/main/classiﬁcation.html (accessed on 8 March 2016). Remote Sens. 2016, 8, 811 33 of 33 14. U.S. Environmental Protection Agency. Level III Ecoregions of the Continental United States (Revision of Omernik, 1987); Environmental Protection Agency—National Health and Environmental Effects Research Laboratory: Corvallis, OR, USA, 1999. 15. Omernik, J.M. Map supplement: Ecoregions of the conterminous United States. Ann. Assoc. Am. Geogr. 1987, 77, 118–125. [CrossRef] 16. Auch, R.F.; Drummond, M.A.; Sayler, K.L.; Gallant, A.L.; Acevedo, W. An approach to assess land-cover trends in the conterminous United States (1973–2000). In Remote Sensing of Land Use and Land Cover, Principles and Applications; Giri, C., Ed.; CRC Press: Boca Raton, FL, USA, 2012; pp. 351–368. 17. U.S. Geological Survey. National Land Cover Dataset 1992 (NLCD1992). Available online: http://www.mrlc. gov/nlcd1992.php (accessed on 8 March 2016). 18. Cochran, W.G. Sampling Techniques; Wiley and Sons: New York, NY, USA, 1977; Volume 98, pp. 259–261. 19. U.S. Geological Survey. Landsat 4–7 Climate Data Record (CDR) Surface Reﬂectance—User ’s Guide. Available online: http://landsat.usgs.gov/documents/cdr_sr_product_guide.pdf (accessed on 8 March 2016). 20. U.S. Geological Survey. Provisional Landsat 8 Surface Reﬂectance Product—Product Guide. Available online: http://landsat.usgs.gov/documents/provisional_l8sr_product_guide.pdf (accessed on 8 March 2016). 21. U.S. Geological Survey. Earth Resources Observation and Science (EROS) Center Science Processing Architecture (ESPA) on Demand Interface—User Guide. Available online: http://landsat.usgs.gov/ documents/espa_odi_userguide.pdf (accessed on 8 March 2016). 22. U.S. Geological Survey. Landsat Processing Details. Available online: http://landsat.usgs.gov/Landsat_ Processing_Details.php (accessed on 8 March 2016). 23. Zhu, Z.; Woodcock, C.E. Object-based cloud and cloud shadow detection in Landsat imagery. Remote Sens. Environ. 2012, 118, 83–94. [CrossRef] 24. Zhu, Z.; Wang, S.; Woodcock, C.E. Improvement and expansion of the Fmask algorithm: Cloud, cloud shadow, and snow detection for Landsats 4–7, 8, and Sentinel 2 images. Remote Sens. Environ. 2015, 159, 269–277. [CrossRef] 25. Zhu, Z.; Woodcock, C.E. Automated cloud, cloud shadow, and snow detection in multitemporal landsat data: An algorithm designed speciﬁcally for monitoring land cover change. Remote Sens. Environ. 2014, 152, 217–234. [CrossRef] 26. Zhu, Z.; Woodcock, C.E.; Holden, C.; Yang, Z. Generating synthetic Landsat images based on all available Landsat data: Predicting Landsat surface reﬂectance at any given time. Remote Sens. Environ. 2015, 162, 67–83. [CrossRef] 27. Davis, J.C.; Sampson, R.J. Statistics and Data Analysis in Geology; Wiley: New York, NY, USA, 1986; Volume 646. 28. Rayner, J.N. Introduction to Spectral Analysis; Pion: London, UK, 1971. 29. Breiman, L. Random forests. Mach. Learn. 2001, 45, 5–32. [CrossRef] 30. Zhu, Z. Optimizing selection of training and auxillary data for operational land cover classiﬁcation for the LCMAP initiative. ISPRS J. Photogramm. Remote Sens. 2016. submitted. 31. U.S. Geological Survey. National Land Cover Database 2006 (NLCD 2006). Available online: http://www.mrlc. gov/nlcd2006.php (accessed on 11 March 2016). 32. U.S. Fish and Wildlife Service. National Wetlands Inventory. Available online: http://www.fws.gov/ wetlands/ (accessed on 8 March 2016). 33. Natural Resources Conservation Service. Soil Survey Geographic Database. Available online: http://www.nrcs. usda.gov/wps/portal/nrcs/site/soils/home/ (accessed on 11 March 2016). 34. Wilen, B.O.; Bates, M. The U.S. Fish and Wildlife Service’s National Wetlands Inventory project. In Classiﬁcation and Inventory of the World’s Wetlands; Springer Netherlands: Dordrect, The Netherlands, 1995; pp. 153–169. © 2016 by the authors; licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC-BY) license (http://creativecommons.org/licenses/by/4.0/).

Journal

Remote Sensing – Multidisciplinary Digital Publishing Institute

Published: Oct 1, 2016

Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 7-Day Trial for You or Your Team.

Learn More →

Evaluation of the Initial Thematic Output from a Continuous Change-Detection Algorithm for Use in Automated Operational Land-Change Mapping by the U.S. Geological Survey

Evaluation of the Initial Thematic Output from a Continuous Change-Detection Algorithm for Use in Automated Operational Land-Change Mapping by the U.S. Geological Survey

Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 7-Day Trial for You or Your Team.

Learn More →

Evaluation of the Initial Thematic Output from a Continuous Change-Detection Algorithm for Use in Automated Operational Land-Change Mapping by the U.S. Geological Survey

Evaluation of the Initial Thematic Output from a Continuous Change-Detection Algorithm for Use in Automated Operational Land-Change Mapping by the U.S. Geological Survey

References (34)

Abstract

Journal

Recommended Articles

There are no references for this article.

Our policy towards the use of cookies