Map request: Notified maps and locations from Gazettes of India

[planemad]

Why is this map important to add on amche.in?

The Gazettes of India are the authoritative source of information publicly notified by the Government of India and contain various location information covering:

• Limits of notified protected areas (National Parks, WLS, Eco-Sensitive Zones)
• Land parcels for land acquisitions
• Important locations for public infrastructure

Describe the data source

• Format: Map image OR coordinate text OR survey number+village
• Attribution: Gazette name and series
• URL: https://archive.org/details/gazetteofindia

Technical Documentation

There are thousands of such maps and locations within the gazettes. They need to be extracted and georeferenced in an appropriate format before they can be added. This will have to be done in several steps:

• Document examples of mappable locations in https://archive.org/details/gazetteofindia
• Categorize the maps and the published formats so they can be auto extracted into a separate archive
• Pipeline to manually georeference gazette maps using an appropriate tools like https://mapwarper.net or https://allmaps.org
• Create a map collection json for viewing on amche.in

cc @sushant354 @ramSeraph

[planemad]

Sample Of Eco Sensitive Zone maps

Extent and boundaries of Eco-sensitive Zone of Shravati Valley Wildlife Sanctuary

Ministry:

• Ministry of Environment, Forests and Climate Change

Keywords:

• BOUNDARY DESCRIPTION 4721
• latitudes and longitudes 961
• boundary details 683
• List of geo-coordinates 492
• GOOGLE MAP 296
• Google Earth 201
• MAP OF ECO-SENSITIVE ZONE 1084
• LATITUDE AND LONGITUDE OF PROMINENT LOCATIONS 423
• GEO- COORDINATES OF PROMINENT LOCATIONS 420

https://archive.org/details/in.gazette.central.e.2019-08-05.210456/page/n27/mode/2up?q=buffer

[ramSeraph]

IIIF url for the above example is https://iiif.archive.org/iiif/in.gazette.central.e.2019-08-05.210456/manifest.json and the allmaps links to help georeference the file is https://editor.allmaps.org/mask?url=https%3A%2F%2Fiiif.archive.org%2Fiiif%2Fin.gazette.central.e.2019-08-05.210456%2Fmanifest.json&image=https%3A%2F%2Fiiif.archive.org%2Fimage%2Fiiif%2F3%2Fin.gazette.central.e.2019-08-05.210456%252F210456_jp2.zip%252F210456_jp2%252F210456_0037.jp2

[planemad]

This is fantastic and much simpler than expected. We essentially need a catalogue of such maps and the georeferencing status from allmaps to build a queue that an operator can work through.

An ideal solution would be the ability to filter gazettes having a map and its georeferencing status directly on archive.org. This will help an operator to just work through a particular filtered search result like WLS maps.

From https://www.leventhalmap.org/projects/digital-projects/georeferencing/

[planemad]

IA Metadata

Looking at the IA manifest.json, the map is currently tagged with motivation:painting. Whatever library is doing this needs to learn about detecting maps from paintaings.

• Docs: https://iiif.io/api/extension/georef/#32-georeference-annotation-motivation

Thumbnail

https://iiif.archive.org/image/iiif/3/in.gazette.central.e.2019-08-05.210456%2F210456_jp2.zip%2F210456_jp2%2F210456_0037.jp2/1015,1460,799,824/pct:25/0/default.jpg

Image API: https://www.learniiif.org/image-api/playground

Allmaps Metadata

• https://annotations.allmaps.org/maps/8484fdf31a7689c7

Tile URL

The tile url id can be parsed from _allmaps.image.id -> 4bb4d3fbc63f6393 -> ``https://allmaps.xyz/images/4bb4d3fbc63f6393/{z}/{x}/{y}@2x.png`

"_allmaps": {
      "id": "https://annotations.allmaps.org/maps/8484fdf31a7689c7",
      "version": "https://annotations.allmaps.org/maps/8484fdf31a7689c7@1d8519f7824731a0",
      "image": {
        "id": "https://annotations.allmaps.org/images/4bb4d3fbc63f6393",
        "version": "https://annotations.allmaps.org/images/4bb4d3fbc63f6393@a57e4bd156826e47",

View on amche.in

Allmaps Annotations API

This returns all the georeferenced maps in a given coordinate

https://annotations.allmaps.org/maps?limit=40&intersects={lat},{lng}&maxarea=500000000000

[ramSeraph]

I don't think the allmaps annotation urls are meant to be permanent.. I think we are supposed to save the annotations away from the allmaps temporary annotation storage. I remember this being mentioned in their docs( needs confirmation though ).

The georeferencing happens in the allmaps editor. An annotation is generated and we need to save it in some place( with CORS enabled ) where it can accessed by the allmaps viewer. I think supplying the annotations as a query parameter to the viewer is supported. Might be supported for the tiler too( feels unlikely though ). Anyway, we might not really need the tiler as there is a maplibre-plugin that can be used to show a georeferenced layer using just the annotation url or annotation itself. Alternatively. we can run the allmaps tiler code ourself where we supply our own annotation urls( in case we are stressing out the allmaps tiler server or if the allmaps tiler server doesn't support our annotation urls ).

BTW, I suspect the annotation is itself a manifest with a georeferencing annotation as defined in this extension.. So, I wonder if we can ask IA(Internet Archive) folks to let us edit the existing manifests for adding georeferencing annotations to them. It could be one more file to upload to the IA item and the IA folks could just add the annotation to the manifest they are already generating. This is a long shot though, given we have no idea how to petition the IA folks to consider this. Even then we will run into credential issues mentioned below.

On the sourcing and tracking side.. the keyword search and the further filtering by the "painting" tag to locate the pages might be good to start with. But I think tracking completion on IA will be hard, we will need our own landing page from which we can tap into IA's search API and our own tracking of georeferencing status. For tracking completion on IA, we will have a hard time given that only an item owner can edit an item, even though in this case luckily the items were mostly created by people involved in this project.. I don't think we will be people georeferencing them and accessing our credentials away from IA just sounds unappetizing to me atleast.

So basically we will need our own server for tracking and maybe hosting the annotation files if required( I would suggest going with cloudflare workers and R2 storage and maybe their sqlite offering for database ).

There are still a few more things to understand/explore.. how do we georeference multiple maps in the same pdf.. do they automatically become a mosaic and how do we do intentional mosaics across different pdfs?

[planemad]

don't think the allmaps annotation urls are meant to be permanent.

This would be good to verify. There is a CC-0 dump available of the georeferencing data twice a day from https://allmaps.org/#open-data

So, I wonder if we can ask IA(Internet Archive) folks to let us edit the existing manifests for adding georeferencing annotations to them. It could be one more file to upload to the IA item and the IA folks could just add the annotation to the manifest they are already generating. This is a long shot though, given we have no idea how to petition the IA folks to consider this. Even then we will run into credential issues mentioned below.

👍 we should get this suggestion in front of the right folks to take forward

how do we georeference multiple maps in the same pdf.. do they automatically become a mosaic and how do we do intentional mosaics across different pdfs?

My understanding is that mosaics can be created from images within an item collection. Mosaicing across pdfs does not seem supported directly, which is fine.

[planemad]

Sample Goa Command Area Addition/Deletions Map

Ministry:

• Department of Water Resource, Goa
• https://archive.org/details/in.goa.egaz.2526-48.SI/2526-48-SI-OG/page/2440/mode/2up

Table of parcels identified with survey numbers with maps

Additions

Omissions

Georeferencing

• IA identifier: in.goa.egaz.2526-48.SI
• manifest.json
• Allmaps georeferencer with Regional Plan background

[ramSeraph]

This looks interesting BTW.. https://monarcha.ai/ product seems to match our problem statement.

[planemad]

General workflow for digitizing Gazette maps

Step 1: Georeference

1.Identify Gazette with maps

• Use text search to identify gazette of interest on internet archive.org
  • eg. MAP OF ECO-SENSITIVE ZONE 1084

https://archive.org/details/in.gazette.central.e.2017-08-10.178009/page/n27/mode/2up

2.Get IIIF manifest.json

• manifest.json url: https://iiif.archive.org/iiif/{document_identifier}/manifest.json
• Substitute {document_identifier} from IA link metadata section or url details/in.gazette.central.e.2017-08-10.178009
  • eg.in.gazette.central.e.2017-08-10.178009 -> https://iiif.archive.org/iiif/in.gazette.central.e.2017-08-10.178009/manifest.json

3.Open IIIF manifest with Allmaps Editor

• Allmaps editor URL: https://editor.allmaps.org/images?url={url/to/manifest.json} eg.
  https://editor.allmaps.org/images?url=https://iiif.archive.org/iiif/in.gazette.central.e.2017-08-10.178009/manifest.json&bg-preset=osm
• Choose the desired map page that includes point coordinates or grid reference to georeference accurately
• Add background TMS to url &bg-url=https://indianopenmaps.fly.dev/soi/osm/{z}/{x}/{y}.webp (Survey of India 1:50K toposheet)

4.Georeference with Allmaps Editor

• Target 3 GCPs with maximum spacing from each other and best visual match to satellite imagery/OpenStreetMap
• Use coordinate converter between degrees/minutes and decimal degrees format if needed for manual entry https://latlongdata.com/lat-long-converter/
• Change basemap layer for better reference
  • Survey Of India Open Series Map: https://indianopenmaps.fly.dev/soi/osm/{z}/{x}/{y}.webp

• Verify results and adjust GCP as required

• Locate XYZ tile URL from export: https://allmaps.xyz/images/80713f728801d925/{z}/{x}/{y}@2x.png

Step 2: Digitize on OSM

• Open georeferenced map in JOSM http://127.0.0.1:8111/imagery?title=Allmaps&type=tms&url=https://allmaps.xyz/images/80713f728801d925/{z}/{x}/{y}@2x.png or manually with Imagery>Imagery Preferences>TMS
• Imagery can by manually added in osm.org iD web editor with the custom background url https://allmaps.xyz/images/80713f728801d925/{z}/{x}/{y}@2x.png

• Digitize/update the required boundary feature with appropriate tags
  • Eco Sensitive Zone: eg. https://www.openstreetmap.org/relation/3963830

Comparison of the new boundary (orange outline)) and old one (green outline)

Publish composite maps with amche.in

Georeferenced gazette maps, digitized OSM boundary and Survey Of India toposheets can be published on amche.in using a single link by including layer= url params for the various map layers:

Eg. https://amche.in/dev/?atlas=india&layers=mapbox-satellite,osm:relation/3963830,osm:relation/21057070,{type:allmaps,opacity:0.3,id:80713f728801d925},{id:soi-osm,opacity:0.6}

URL Params

• osm:relation/3963830 for https://www.openstreetmap.org/relation/3963830/
• osm:relation/21057070 for https://www.openstreetmap.org/relation/21057070/
• {type:allmaps,opacity:0.3,id:80713f728801d925} for digitized Gazette maps with 0.3 opacity from https://viewer.allmaps.org/?url=https://annotations.allmaps.org/images/80713f728801d925
• {id:soi-osm,opacity:0.6} for Survey Of India toposheet with 0.6 opacity

Join the project

Editable worksheet for collaboration: https://docs.google.com/spreadsheets/d/1W0R_LX1oHSdhOMs_vcIsoACHTCy1cQatCQ8UHolpoE4/edit?gid=0#gid=0

• Step 1. In a new row add State and Protected Area name from the reference list
  • https://en.wikipedia.org/wiki/List_of_national_parks_of_India
  • https://en.wikipedia.org/wiki/List_of_wildlife_sanctuaries_of_India
• Step 2. Search ESZ notification Gazette on Internet Archive and paste URL to the Gazette in col H
• Step 3: Open the generated Allmaps URL in col J and open the PDF map page to georeference
• Step 4: Paste the georeferencer URL to the map page in col K. Follow the georeferencing workflow to add matching GCPs to the Gazette amp and reference Survey Of India map (use the search to browse to the correct map location if map is not visible)
• Step 5: After georeferencing is complete, verify the output on the Results tab. Copy the Export>XYZ Map Tiles URL and paste in col L
• Step 6: Digitize the georeferenced Gazette map in OSM. Use the url in col M to directly open the imagery in JOSM.