Uiowa

Emma-Shanks

Transformation of Spatial data and uploading into Observable HQ:

How to upload TopoJSON:

1. Go to mapshapper.org

2. Press select. Go to previously extracted file of the county boundary data. Press CTRL and click on data ending in .prj, .shp, .shx, .dbf. You will want all of these files.

3. Press import. You will see a map of the data outlining the counties of Washington. 

4. Simplify the map if there is extra detail applied. This is found in the upper right-hand corner next to export. Make sure your map is not too simplified, losing its edges, but not supported extra unnecessary detail.

5. Press export in upper right-hand corner.

6. A series of options will appear. Press TopoJSON for this lab (explanation for this found in other part of cheat sheet).

7. Press export and save to your saved drive/folder for this lab. I suggest naming this something easy to remember and focused on this lab. For this lab I chose "Washington_Counties"

8. Add the file to Observable: In the upper right-hand corner, press the paperclip icon to see the used data. Press the plus button and add to the .json file you just made for Washington_Counties.json. 

9. Then, where the old information used to be, start typing Cou-. This should have the Washington_Counties.json pop up. Click on it. 

10. Run the line. Press the paperclip again to make sure your Washington_Counties data now says "Used".

What is the difference between TopoJSON and GeoJSON formats?

GeoJSON: allows for geographic data encoding through larger files sizes. Unlike Topo, it can include point, line, polygon, and multipolygon information. 

TopoJSON: this is an extension of the GeoJSON that includes topology boundaries as shared boundaries. This is useful for large documents since it reduces repeated data. 

Why are we using Topo over Geo?

Topo is being used in this lab for a couple of reasons. 

1. With boundaries lines and polygons, Geo is likely to cause slivers to happen since it will include both boundaries' data, even if it can be simplfied into one boundary. Topo does this and reduces potential redundancy in the data.

2. This reduction of information makes Topo files smaller, limiting wait times and simplifying data.


Cited Information:
https://github.com/topojson/topojson

How to read through the topoJSON library and make sure your information is correct:

1. First, next to County_features, press the arrow pointing to "Object".

2.  Now, in features:, press the arrow pointing to "Array".

3. Next, press the arrow point to the first "Object"

4. In geometry:, press the arrow pointing to "Object". 

5. Next, in coordinates:, press the arrow pointing to "Array".

6. Press the arrow after "0:" pointing to "Array".

7. This should pull up a list of coordinate points in longitude and latitude. Make sure these coordinate points match the area of data being tested in this project. Reading through these points will walk you through different parts of the TopoJSON function.

What is Normalization?

Normalization provides correlation between my datasets to ensure consistency between my sources and locations. This makes sure my data is consistent throughout my project, even if I pull from multiple sources. This is important for spatial comparison because it ensures the data I pull for spatial data is consistent even when comparing areas of different sizes. 

For example:
In this study, I am evaluating the male and female populations per county in Washington. If pull this data exclusively from total population of Washington, I will get information that may incorrectly represent different areas in the state. Through integration of the data, there will be a better spatial comparison since the information is standardized. 

Cited Info:
https://www.geeksforgeeks.org/dbms/introduction-of-database-normalization/

How do you upload and read CSV files in ObservableHQ?

*I have found two ways to transform data in CSV form to ObservableHQ*

Version 1:
To make a .csv file from your data, you need to follow these steps:

1. In mapshapper.org, refresh the page to delete the old map we made earlier. 

2. Press select. Select the .dbf file from the earlier extracted file from the County_Boundaries folder.

3. This will only show square columns. Press export in the top right corner. Select csv and press export.

4. In the downloads folder you now have .csv file. Click this and you will open an excel document with the different variables that were in your data!

To upload a CSV file, follow these steps:

1. Back in your observable notebook, press the paperclip in the upper right-hand corner. Add in the CSV file you just made. 

2. In the line this comment is attached to, delete the old information in the quotations ("iowa.counties.csv") and replace with the new CSV file we just made. You can do this by starting to type the name of this file, it should pop up (always press the pop-up to make sure the name is exact).


Version 2: 

1. Go to https://www.nhgis.org/ and fill out the testing requirements you want to achieve. 

2. Create an excel cheat of the gathered information and save it as comma-delimited (csv) to your lab file. 

3. Back in your observable notebook, press the paperclip in the upper right-hand corner. Add in the CSV file you just made. 

4. In the line this comment is attached to, delete the old information in the quotations ("iowa.counties.csv") and replace with the new CSV file we just made. You can do this by starting to type the name of this file, it should pop up (always press the pop-up to make sure the name is exact).

How do you create a new variable from existing variables (e.g., population density), and why is normalization important?

a. To create a new variable from existing variables, follow these steps:

1. Upload a CSV file into ObservableHQ. This information is found above. 

2. Once the CSV file is in, you can add your testing variables. It is better to directly copy the names from your Excel file to make sure there is consistency between the Excel file (CSV) and the Observable. For this project, the testing variables were Total Population, Total Male Population, and Total Female Population.  Separate these variables by commas.

3. Now you must list the data you want to create. First, put just the Total population. Now in two parentheses, put a (+TOTAL_MALE/+TOTAL_POP)*100. 
                - The (+) before each of these to make my line transform from a string to an integer. 
                - The (*100) makes my information a percentage

4. Now, follow a similar pattern for measuring the female population against the total population. (+TOTAL_FEMALE/+TOTAL_POP)*100). 

5. This should create two new variables. The first variable is the percentage of men per county area. The second variable is the percentage of women per county area. 


Why is normalization important? 
Normalization makes my information more efficient and applicable to my study area. Through following these steps, I was able to standardize my information. This will make my spatial patterns more recognizable when I am creating my map. Not normalizing the data could make my information misleading when I eventually create a map since it would relate my information to a larger geographic area, instead of my areas of study.

I added a title to my features by deleting the old information and adding new, data appropriate, titles.

I made d[1] to "MaleRat_county" to match the normalized male data per county information.

To change the title of your document:

1. Delete the previous title placed between quotations. Write a new title that fits your study criteria, making sure it stays between ''.

How to find data:

1. Check Profs. links in the data category found in Icon Modules. 

2. Choose one that will match your desired testing subject. I chose Counties in Washington.

3. Download this data file. You will get a zipped file. In file explorer, right click on your zipped file and press extract all.  Extract this information to your personal data folder for this lab.

4. Check out the extracted file. Make sure it includes .shp, .shx, .dbf, .prj.

Coordinate System:

Important: Check if data is in WGS84 coordinate system before starting.

How?

1. Go to File explorer and open extracted file from Lab 1 for your dataset. 

2. Right click on the .prj file and click "open with Notepad"

3. It will open the coordinate system. You will find the information you need after "GEOGS"

4. This dataset was already in the correct coordinate system. 

Why is this important?

Coordinate systems are important for future labs, TopoJSON files, latitude/longitude attachment, and Observable D3. WGS1984 provides the latitude and longitude for our locations, something the D3 will need. Not using the correct coordinate system will cause issues with the data, rendering, and projection. 

What do you do if you have the wrong Geographic Coordinate System:
*Conventional methods to fix this did not work for me, so this is how I fixed it for this project*

1. Open ArcGIS and enter the data file you want to work with. We want to first change the projected coordinate system to WGS84 so we can access long and lat coordinates. This is found in world coordinate systems, scroll until you find WGS1984.

2. Go to Analysis. Go to Tools. Search: "Feature Vertices to Points". Input your projected layer. Put point type to All. --> This creates a general one point for every polygon to outline it. This has given x and y coordinates for us to eventually view in Observable. 

3. Open your attribute type. Go to tools and type "Calculate Geometry Attributes". Press Point X coordinate and Point Y Coordinate. Double check you are still using the right coordinate system. 

4. Export the file as .geojson --> We will change this in mapshaper but this allows for all the data to be exported, not just my points.

5. Import to mapshaper. Import .prj, .shp, .shx, .dbf

6. More instructions below on how to use mapshaper.

I made d[2] to "FemaleRat_county" to match the normalized female data per county information.