Lonely Planet Dark Skies is a great book of places around the world to see the sky at night, but doesn’t have a map. Problem solved:
What is a Genetic Algorithm (GA)?
GAs simulate evolution by having genotypes (like DNA in humans) generate phenotypes (like human beings). The genotypes are usually random at the start. The phenotypes are put through some kind of testing function, and the best ones get recombined with the other best ones to make children. Some noise is added, since in the real world DNA isn’t copied exactly correctly and cosmic rays break things occasionally. The phenotypes that don’t do well are discarded. Then the whole process is repeated. For more, see wikipedia.
The Wolfram Language makes it remarkably easy to generate random images, combine them, add noise, and test them against neural networks as a fitness function.
We can thus evolve images to fool neural networks fairly easily.
For example, the default image identification network model is 99.9977% sure that this is an image of Yoda:
That image took about 30 min of compute time to generate, and could be sped up in a variety of ways. It would be interesting to experiment with this to:
- Select fitness for compression-ability to try and remove the randomness
- Restrict the colors
- Use geometry features instead of pixels (lines, rectangles and so on)
- Use the generated negative images to improve the models
I’m assuming all of this isn’t novel, but after some quick googling around I couldn’t find anything particularly similar.
You can find the notebook here and play with it yourself, and it’s also embedded below:
HERE XYZ shipped yesterday. It’s hard to talk about it without referring to CARTO so we can do that – it’s just like CARTO in some ways. But really, it’s a glimpse of what GIS will look like in the future as it gets democratized and cheaper. While fulfilling similar goals to GIS – creating maps – it’s opposite GIS in many ways. XYZ is quicker, free (or freeish, it looks like paid plans are coming) and far simpler. It’s built for making mashups and publishable maps today, but you can see a bunch of directions it can go in the future. What it misses today is analysis, we need something in between XYZ and QGIS which would be a real tool you could use to make decisions with.
One super cool thing (for me at least) is the use of OSM everywhere.
Here’s a map I made pulling in some MSFT buildings :
It’s relatively easy to pull in geojson and get it on a map, I grabbed the smallest MSFT buildings file and uploaded that (Washington DC). At first I thought there were Safari bugs as many little things and prompts are missing but switching to Chrome didn’t seem to make much difference. There’s missing squares of buildings that I’m guessing either failed or are still importing. There’s a 1 million object limit so it could have just stopped importing too.
What’s happening is we’re moving to maps 3.0.
- Maps 1.0 was paper and clunky tools, like reloading a page to pan or zoom a map. Stored in a silo on someones website.
- Maps 2.0 is/was Google Maps. “free” maps, easy access, easy to add to my site but the same experience for everyone. Generally required a developer to implement.
- Maps 3.0 isn’t here but we’re starting to get hints of it with CARTO and XYZ. Maps 3.0 removes the need for a developer, lets me customize the map and deploy it anywhere by just clicking around, and include my own data.
What we need to add here are tools that let me tell a story with the map. That means transitions, text, video and interactivity to let an end user explore the narrative. Beyond that it means tools that let me rip, mix and burn a new map. Can I take your map and add something or mash it with another one? Can I let end users do something other than just panning around and zooming, like measuring things, dropping notes, adding more context? That’s what’s really interesting as a set of possiblities.
XYZ bugs & features missing, remembering it’s day 2:
- Upload of .gz or .zip files looks unsupported, which is important with gigantic MSFT buildings files
- Upload is kind of slow. I was curious where XYZ is hosted but they have CloudFront in front of it. Maybe it’s in Europe or some other AWS AZ.
- Login is HERE-only, would be nice to have social logins to make it quicker.
- Double-click to zoom is missing(!)
- Inertial map dragging seems to work in Chrome but not Safari.
- shift-drag to zoom to an area is missing – a major power tool.
- The default map zoom & location is to show western Europe. It would be nice if it used ip or browser location to show something more relevant.
- There doesn’t seem to be indication on layer processing, though you do get a upload process bar.
- The relationships between layers and data is slightly confusing in that you have to upload and then select the layer and then add it. XYZ has gone with the model of having maps, layers and datasets which feels a little inherited from GIS. Instead I really just want to load data on a map and not have these abstractions and dialogs. Just drag a geojson on to the map, boom, data shows up. Not click-click-click-click…
- Exporting a map gives you some iframe code but it has no css, you have to add height and width
- Location search is buggy in Safari and in Chrome can show 5 different “Denver” results
Today I livestreamed the process of validating geodatamarket.net. This focus was chosen based on a quick twitter poll with 5 votes. It’s not a lot of data to go on but yet still felt like pretty clear feedback.
I set up google and facebook ads for $5/day, set up email capture on the site and also wrote a reddit post.
Why are we doing this? To test the idea as quickly and cheaply as possible. If it doesn’t resonate with anyone, then drop the idea and shut the site down and move on to the next idea.
So far I’ve spent $11 on a domain, and in the end probably about $20 on advertising, for a total of $31. Time-wise I spent 2.5 hours yesterday building the site and 1 hour today doing some marketing. In a day or two, I’ll look at the ad statistics and review the reddit responses, if any. This will mean a total of about 4.5 hours of work.
It’s very arguable that I didn’t need to build the website or buy the domain and thus cut the investment down to $20 and 2 hours. So very roughly that would cut the cost in half. Next time!
Here’s the live stream:
Imagine you want to buy GIS data. Where would you go? There are lots of free or free-ish catalogs, maybe it would be nice if there was a place to share data that had been made ready for various uses. I ran a quick twitter survey on what to live stream and a whopping two people voted, the first of which was to work on geo data market ideas. Therefore, I live streamed on twitch the creation of a geo data marketplace which now lives at geodatamarket.net.
It took about two hours and is backed by WordPress, woo commerce and a marketplace plugin.
There’s a single product available so far, it’s natural earth raster geoTIFFs cut out by country (250 files) with transparent backgrounds. Anybody could make this data with some work or, just go download it. People like you can log in and create their own downloadable map data products and sell them, or just set them to free.
The livestream videos are broken because my microphone broke, but here they are anyway:
Here’s a walkthrough of glider, fixing lots of bugs along the way. Livestreamed on twitch last night.
Continuing adventures in 3d printing, I had a double pane window fail and got a $1,100 quote to replace it. In the end the failure was the “shoe” locking mechanism inside the counter-weight system. Although there is no counter-weight like a windows from the 1700’s that Nash might have installed, but four coil springs. The rest of the window was fine. I designed and printed replacement parts which you can find here on thingiverse.
Nash, by the way, designed all those beautiful houses around Regent’s Park. Notice the sash windows. These have counter-weights in them, or at least did a few hundred years ago:
The original shoe in my (crappier, more modern) windows used a fairly small amount of plastic at the most stressed part of the design. Failure of decade-old plastic mechanical parts appears to be a common thing, and it looked as if designed to fail exactly when the warranty expired.
I wonder if the designers were designing this to fail on purpose or really thought the thing would last. It’s possible they thought the plastic wouldn’t become so brittle, yet it’s clear they were minimizing the amount of material used to make it. This reminds me of a carrier furnace I used to own where they made the drip pan out of a rusting metal. Some bad design is so egregiously bad it strays in to being plain immoral, like the Bronco roof which couldn’t support its own weight and would kill you if it rolled.
The wonderful thing about 3d printing is, nobody has to design these things ever again and you can go print them out of titanium if you want to.
I have some strange fascination with old Encarta and other CD-ROM titles from 20+ years ago, and how they relate to today. How did encyclopedias and atlas products work then, and how was it different to today? I made a video so you can see how it worked.
Encarta Atlas’ main limitation was storage. So there’s only secondary roads in places, and the aerial imagery is 1 or 10 miles per pixel instead of the high resolution we’re used to today.
Still, most of the UIUX hasn’t changed between then and now. It’s fair to call Encarta Atlas the Google Maps of its time – while Encarta didn’t support shift-drag to zoom to an area… neither does Google today. That’s the only real thing missing on either platform.
Problems become much easier to solve if you can relax their constraints.
For example, if you want to buy something and can relax when you buy it then you can find it much cheaper. Candy is vastly cheaper the day after halloween than the day before.
Along with time, the other major constraint is space. If you can relax where you do something then many opportunities open up. For example if you have to buy water at an airport it’s going to be more expensive than elsewhere. If it doesn’t matter where you live then there are far more job opportunities.
If you watch a movie on a Tuesday it’s not only cheaper but there are far less people, so it’s a higher quality experience as well as saving dollars.
So, try to avoid the “right here, right now” and relax all the constraints you can.
Grab the files:
wget https://usbuildingdata.blob.core.windows.net/usbuildings-v1-1/Alabama.zip wget https://usbuildingdata.blob.core.windows.net/usbuildings-v1-1/Alaska.zip wget https://usbuildingdata.blob.core.windows.net/usbuildings-v1-1/Arizona.zip wget https://usbuildingdata.blob.core.windows.net/usbuildings-v1-1/Arkansas.zip wget https://usbuildingdata.blob.core.windows.net/usbuildings-v1-1/California.zip wget https://usbuildingdata.blob.core.windows.net/usbuildings-v1-1/Colorado.zip wget https://usbuildingdata.blob.core.windows.net/usbuildings-v1-1/Connecticut.zip wget https://usbuildingdata.blob.core.windows.net/usbuildings-v1-1/Delaware.zip wget https://usbuildingdata.blob.core.windows.net/usbuildings-v1-1/DistrictofColumbia.zip wget https://usbuildingdata.blob.core.windows.net/usbuildings-v1-1/Florida.zip wget https://usbuildingdata.blob.core.windows.net/usbuildings-v1-1/Georgia.zip wget https://usbuildingdata.blob.core.windows.net/usbuildings-v1-1/Hawaii.zip wget https://usbuildingdata.blob.core.windows.net/usbuildings-v1-1/Idaho.zip wget https://usbuildingdata.blob.core.windows.net/usbuildings-v1-1/Illinois.zip wget https://usbuildingdata.blob.core.windows.net/usbuildings-v1-1/Indiana.zip wget https://usbuildingdata.blob.core.windows.net/usbuildings-v1-1/Iowa.zip wget https://usbuildingdata.blob.core.windows.net/usbuildings-v1-1/Kansas.zip wget https://usbuildingdata.blob.core.windows.net/usbuildings-v1-1/Kentucky.zip wget https://usbuildingdata.blob.core.windows.net/usbuildings-v1-1/Louisiana.zip wget https://usbuildingdata.blob.core.windows.net/usbuildings-v1-1/Maine.zip wget https://usbuildingdata.blob.core.windows.net/usbuildings-v1-1/Maryland.zip wget https://usbuildingdata.blob.core.windows.net/usbuildings-v1-1/Massachusetts.zip wget https://usbuildingdata.blob.core.windows.net/usbuildings-v1-1/Michigan.zip wget https://usbuildingdata.blob.core.windows.net/usbuildings-v1-1/Minnesota.zip wget https://usbuildingdata.blob.core.windows.net/usbuildings-v1-1/Mississippi.zip wget https://usbuildingdata.blob.core.windows.net/usbuildings-v1-1/Missouri.zip wget https://usbuildingdata.blob.core.windows.net/usbuildings-v1-1/Montana.zip wget https://usbuildingdata.blob.core.windows.net/usbuildings-v1-1/Nebraska.zip wget https://usbuildingdata.blob.core.windows.net/usbuildings-v1-1/Nevada.zip wget https://usbuildingdata.blob.core.windows.net/usbuildings-v1-1/NewHampshire.zip wget https://usbuildingdata.blob.core.windows.net/usbuildings-v1-1/NewJersey.zip wget https://usbuildingdata.blob.core.windows.net/usbuildings-v1-1/NewMexico.zip wget https://usbuildingdata.blob.core.windows.net/usbuildings-v1-1/NewYork.zip wget https://usbuildingdata.blob.core.windows.net/usbuildings-v1-1/NorthCarolina.zip wget https://usbuildingdata.blob.core.windows.net/usbuildings-v1-1/NorthDakota.zip wget https://usbuildingdata.blob.core.windows.net/usbuildings-v1-1/Ohio.zip wget https://usbuildingdata.blob.core.windows.net/usbuildings-v1-1/Oklahoma.zip wget https://usbuildingdata.blob.core.windows.net/usbuildings-v1-1/Oregon.zip wget https://usbuildingdata.blob.core.windows.net/usbuildings-v1-1/Pennsylvania.zip wget https://usbuildingdata.blob.core.windows.net/usbuildings-v1-1/RhodeIsland.zip wget https://usbuildingdata.blob.core.windows.net/usbuildings-v1-1/SouthCarolina.zip wget https://usbuildingdata.blob.core.windows.net/usbuildings-v1-1/SouthDakota.zip wget https://usbuildingdata.blob.core.windows.net/usbuildings-v1-1/Tennessee.zip wget https://usbuildingdata.blob.core.windows.net/usbuildings-v1-1/Texas.zip wget https://usbuildingdata.blob.core.windows.net/usbuildings-v1-1/Utah.zip wget https://usbuildingdata.blob.core.windows.net/usbuildings-v1-1/Vermont.zip wget https://usbuildingdata.blob.core.windows.net/usbuildings-v1-1/Virginia.zip wget https://usbuildingdata.blob.core.windows.net/usbuildings-v1-1/Washington.zip wget https://usbuildingdata.blob.core.windows.net/usbuildings-v1-1/WestVirginia.zip wget https://usbuildingdata.blob.core.windows.net/usbuildings-v1-1/Wisconsin.zip wget https://usbuildingdata.blob.core.windows.net/usbuildings-v1-1/Wyoming.zip
California and Texas are too big for GDAL so here’s a semi-broken script to split them up:
#!/usr/bin/ruby f = File.open("Texas.geojson", "r") f1 = File.open("TX-1.geojson", "w") f2 = File.open("TX-2.geojson", "w") count = 0 toggle = true f.each_line do |line| if count < 4 f1.write(line) f2.write(line) else if line == " ]" #FIXME this doesn't match, probably needs line.chomp f1.write("]}") f2.write("]}") exit end if toggle f1.write(line) else f2.write(line) end toggle = !toggle end count = count + 1 end
Then assuming GDAL is installed and you have a PostGIS db:
for f in *.geojson; do ogr2ogr -f "PostgreSQL" PG:"dbname=steve user=steve" $f -nln usa_buildings -append; done