How to publish OFDS data

0.3.0 release

Welcome to the Open Fibre Data Standard 0.3.0 release.

We want to hear your feedback on the standard and its documentation. For general feedback, questions and suggestions, you can comment on an existing discussion or start a new one. For bug reports or feedback on specific elements of the data model and documentation, you can comment on the issues in the issue tracker or you can create a new issue.

To comment on or create discussions and issues, you need to sign up for a free GitHub account. If you prefer to provide feedback privately, you can email info@opentelecomdata.net.

This page provides an overview of the process for publishing Open Fibre Data Standard (OFDS) data and how-to guides for specific topics.

Overview

The process for publishing OFDS data can be divided into three phases:

Plan

The plan phase covers identifying your priority use cases, deciding what data to publish and identifying your data sources.

Identify your priority use cases

There are many use cases for OFDS data, each with their own data needs. You ought to decide which use cases to prioritise so that you can make sure that your data includes the necessary fields and that it is available via suitable publication formats and access methods.

Decide what data to publish

Bearing in mind your priority use cases, you ought to review the OFDS schema and decide which fields you want to publish.

OFDS is designed for the public disclosure of open data. However, you can also use it to structure data that you want to share only with specific partners and data that you want to keep within your own organisation. As such, this step can involve deciding which fields to make public, which to share with partners and which to keep private.

Most fields in the OFDS schema are optional. However, the more fields you publish, the more useful your data will be.

If you are concerned about disclosing sensitive location data, see how to obfuscate location data.

Identify your data sources

Once you have decided what data to publish, you ought to identify your data sources. These will be the systems, databases and documents that contain the data that you will convert to OFDS format for publication.

Prepare

The prepare phase covers mapping your data to OFDS, collecting missing data and choosing your publication formats and access methods.

Map your data to OFDS

Once you have identified your data sources, you ought to map your data to the OFDS schema, that is, identify which data elements within your data sources match which OFDS fields and codes. If there are data elements that you want to publish but for which you cannot identify a suitable mapping, you can add additional fields to your OFDS data.

Your mapping acts as a blueprint for preparing your data. It will help you to identify the steps involved in converting your data to OFDS format.

Collect missing data

Your mapping might identify fields that you want to publish but that are missing from your data sources. If so, you’ll need to collect the missing data.

Choose your publication formats and access methods

Bearing in mind your priority use cases, you ought to decide which publication formats and access methods you will use to publish your OFDS data.

For more information, see how to format data for publication and how to provide access to data.

Publish

The publish phase covers preparing your data, checking your data and publishing your data.

Prepare your data

Once you have completed your mapping and decided on your publication formats and access methods, the next step is to convert your data to OFDS format.

The suggested approach is to develop a reproducible data pipeline so that you can easily update your OFDS publication when the data in your data sources is updated. However, you can prepare your data using whichever tools you are most comfortable with.

For guidance on common steps in converting your data to OFDS format, see the following guides:

Check your data

Once you have prepared your data, the next step is to use the OFDS Convert, Validate, Explore tool (CoVE) to check that your data is correctly structured and formatted according to the OFDS schema.

Publish your data

Once any issues with the structure and format of your data have been resolved, the next step is to publish your data using your chosen access methods.

For your data to be open, you need to publish it using an open license. For more information, see how to license your data.

How-to guides

This section contains how-to guides for specific topics. To learn about the process for publishing OFDS data, see the overview.

How to obfuscate location data

If you’re concerned about disclosing the exact location of fibre infrastructure, you can truncate the coordinates of node locations and span routes in your public or shared data to obfuscate their exact locations, whilst retaining the precise coordinates for use within your own organisation. Before truncating coordinates, you ought to consider what level of accuracy is required to satisfy your priority use cases. You can use the following table as a guide to the relationship between coordinate precision and accuracy:

Coordinate precision

Accuracy

0.001°

± 111 m

0.0001°

± 11.1 m

0.00001°

± 1.11 m

0.000001°

± 0.111 m

How to add additional fields

The OFDS schema does not restrict the use of additional fields, except where noted in the schema reference. If there is a data element that you wish to publish for which you cannot identify a suitable mapping in OFDS, you can add an additional field to your data.

Before adding an additional field, you ought to search the standard issue tracker to see if a similar concept has already been discussed. If there are no existing discussions, you ought to open a new issue and describe the concept that you want to publish and your proposed modelling.

If you add an additional field, you ought to describe its structure, format and meaning in your data user guide. For more information, see how to write a data user guide.

How to format data for publication

OFDS data can be published in three publication formats:

  • The JSON format reflects the structure of the schema, is useful to developers who want to use the data to build web apps, and offers a ‘base’ format that other publication formats can be converted to and from.

  • The GeoJSON format is useful to GIS analysts who want to import the data directly into GIS tools without any pre-processing.

  • The CSV format is useful to data analysts who want to import data directly into databases and other tabular analysis tools, and to users who want to explore the data in spreadsheet tools.

To meet the widest range of use cases, you ought to publish data in all three formats. The suggested approach is to export your data in the JSON format and to use the following tools to transform it to the GeoJSON and CSV formats:

The command line tool Lib CoVE OFDS provides an interface for transforming OFDS data from JSON to GeoJSON format.

To convert a network package to GeoJSON format, install Lib CoVE OFDS and run the following command:

libcoveofds jsontogeojson network-package.json nodes.geojson spans.geojson

Flatten Tool provides a command-line interface for transforming OFDS data from JSON to CSV format.

To convert data to CSV format:

flatten-tool flatten --truncation-length=9 --root-list-path=networks --main-sheet-name=networks --schema=network-schema.json network-package.json -f csv

How to publish large networks

This section describes how to:

  • Use pagination to publish an individual network that is too large to return in a single API response

  • Use streaming to publish an individual network that is too large to load into memory.

For information on how to use pagination and streaming to publish multiple networks, see the publication formats reference.

This guidance is applicable to the JSON publication format, for information on pagination and streaming for the GeoJSON format see the GeoJSON publication format reference.

Pagination

The preferred approach is to publish embedded nodes and spans in .nodes and .spans, respectively. If your network is too large to return in a single API response, you ought to use .links to reference separate endpoints for nodes and spans. Each endpoint ought to return a top-level JSON object with a nodes or a spans array, respectively, and a links object with URLs for the next and previous pages of results:

The following example shows a network with embedded nodes and spans:

{
    "id": "a096d627-72e1-4f9b-b129-951b1737bff4",
    "name": "Ghana Fibre Network",
    "nodes": [
        {
            "id": "1",
            "name": "Accra",
            "phase": {
                "id": "1",
                "name": "NextGen Phase 1"
            },
            "status": "operational",
            "location": {
                "type": "Point",
                "coordinates": [
                    -0.174,
                    5.625
                ]
            },
            "address": {
                "streetAddress": "Boundary Rd",
                "locality": "Shiashie",
                "region": "Accra",
                "postalCode": "12345",
                "country": "GH"
            },
            "type": [
                "pop"
            ],
            "accessPoint": true,
            "internationalConnections": [
                {
                    "region": "Lom\u00e9",
                    "country": "TG"
                }
            ],
            "power": true,
            "technologies": [
                "ip"
            ],
            "physicalInfrastructureProvider": {
                "id": "GH-RGD-CS111111111",
                "name": "FibreCo"
            },
            "networkProviders": [
                {
                    "id": "GH-RGD-CS222222222",
                    "name": "FastWeb"
                }
            ]
        },
        {
            "id": "2",
            "name": "Kumasi",
            "phase": {
                "id": "1",
                "name": "NextGen Phase 1"
            },
            "status": "operational",
            "location": {
                "type": "Point",
                "coordinates": [
                    -1.628,
                    6.711
                ]
            },
            "address": {
                "streetAddress": "Kejetia Road",
                "locality": "Bantama",
                "region": "Kumasi",
                "postalCode": "00000",
                "country": "GH"
            },
            "type": [
                "pop"
            ],
            "accessPoint": false,
            "power": true,
            "technologies": [
                "ip"
            ],
            "physicalInfrastructureProvider": {
                "id": "GH-RGD-CS111111111",
                "name": "FibreCo"
            },
            "networkProviders": [
                {
                    "id": "GH-RGD-CS222222222",
                    "name": "FastWeb"
                }
            ]
        }
    ],
    "spans": [
        {
            "id": "1",
            "name": "Accra to Kumasi",
            "phase": {
                "id": "1",
                "name": "NextGen Phase 1"
            },
            "status": "operational",
            "readyForServiceDate": "2018-01-01",
            "start": "1",
            "end": "2",
            "directed": false,
            "route": {
                "type": "LineString",
                "coordinates": [
                    [
                        -0.173,
                        5.626
                    ],
                    [
                        -0.178,
                        5.807
                    ],
                    [
                        -0.112,
                        5.971
                    ],
                    [
                        -0.211,
                        5.963
                    ],
                    [
                        -0.321,
                        6.17
                    ],
                    [
                        -0.488,
                        6.29
                    ],
                    [
                        -0.56,
                        6.421
                    ],
                    [
                        -0.752,
                        6.533
                    ],
                    [
                        -0.867,
                        6.607
                    ],
                    [
                        -1.101,
                        6.585
                    ],
                    [
                        -1.304,
                        6.623
                    ],
                    [
                        -1.461,
                        6.727
                    ],
                    [
                        -1.628,
                        6.713
                    ]
                ]
            },
            "physicalInfrastructureProvider": {
                "id": "GH-RGD-CS111111111",
                "name": "FibreCo"
            },
            "networkProviders": [
                {
                    "id": "GH-RGD-CS222222222",
                    "name": "FastWeb"
                }
            ],
            "supplier": {
                "id": "GH-RGD-CS333333333",
                "name": "ProjectCo"
            },
            "transmissionMedium": [
                "fibre"
            ],
            "deployment": [
                "belowGround"
            ],
            "deploymentDetails": {
                "description": "Buried in ducts"
            },
            "darkFibre": true,
            "fibreType": "G.652",
            "fibreTypeDetails": {
                "fibreSubtype": "G.652.D",
                "description": "CABLED STANDARD ENHANCED SINGLEMODE FIBRE ESMF, LOW WATER PEAK SINGLE MODE FIBRE G.652.D, OS2"
            },
            "fibreCount": 24,
            "fibreLength": 276000,
            "technologies": [
                "sdh"
            ],
            "capacity": 4.976,
            "capacityDetails": {
                "description": "2xSTM-16"
            },
            "countries": [
                "GH"
            ]
        }
    ],
    "phases": [
        {
            "id": "1",
            "name": "NextGen Phase 1",
            "description": "The first phase of the Ghana Fibre Network NextGen roll-out.",
            "funders": [
                {
                    "id": "1",
                    "name": "Ghana Infrastructure Investment Fund"
                }
            ]
        }
    ],
    "organisations": [
        {
            "id": "GH-RGD-CS111111111",
            "name": "FibreCo",
            "identifier": {
                "id": "CS111111111",
                "scheme": "GH-RGD",
                "legalName": "FibreCo Ghana Limited",
                "uri": "https://example.com/companies/CS111111111"
            },
            "country": "GH",
            "roles": [
                "physicalInfrastructureProvider"
            ],
            "roleDetails": "FibreCo owns and maintains the physical infrastructure.",
            "website": "https://www.example.com",
            "logo": "https://www.example.com/logo.png"
        },
        {
            "id": "GH-RGD-CS222222222",
            "name": "FastWeb",
            "identifier": {
                "id": "CS222222222",
                "scheme": "GH-RGD",
                "legalName": "FastWeb Ghana Limited",
                "uri": "https://example.com/companies/CS222222222"
            },
            "country": "GH",
            "roles": [
                "networkProvider"
            ],
            "roleDetails": "FastWeb provides wholesale services to service providers and dark fibre access to other network providers.",
            "website": "https://www.example.com",
            "logo": "https://www.example.com/logo.png"
        },
        {
            "id": "GH-RGD-CS333333333",
            "name": "ProjectCo",
            "identifier": {
                "id": "CS333333333",
                "scheme": "GH-RGD",
                "legalName": "ProjectCo Ghana Limited",
                "uri": "https://example.com/companies/CS333333333"
            },
            "country": "GH",
            "roles": [
                "supplier"
            ],
            "roleDetails": "ProjectCo was responsible for the construction of phase 1 of the NextGen network.",
            "website": "https://www.example.com",
            "logo": "https://www.example.com/logo.png"
        },
        {
            "id": "1",
            "name": "Ghana Infrastructure Investment Fund",
            "country": "GH",
            "roles": [
                "funder"
            ],
            "roleDetails": "The Ghana Infrastructure Investment Fund provided equity financing for the construction of phase 1 of the NextGen network.",
            "website": "https://www.example.com",
            "logo": "https://www.example.com/logo.png"
        }
    ],
    "contracts": [
        {
            "id": "1",
            "title": "NextGen Phase 1 Construction Contract",
            "description": "Contract for the construction of phase 1 of the NextGen network.",
            "type": "private",
            "value": {
                "amount": 1537459500.0,
                "currency": "GHS"
            },
            "dateSigned": "2016-06-30",
            "documents": [
                {
                    "title": "NextGen Phase 1 Construction Contract",
                    "description": "Signed contract.",
                    "url": "http://example.com/contracts/nextgen-phase1-construction.pdf",
                    "format": "application/pdf"
                }
            ],
            "relatedPhases": [
                {
                    "id": "1",
                    "name": "NextGen Phase 1"
                }
            ]
        }
    ],
    "website": "http://example.com/network",
    "publisher": {
        "name": "FibreCo",
        "identifier": {
            "id": "CS111111111",
            "scheme": "GH-RGD",
            "legalName": "FibreCo Ghana Limited",
            "uri": "https://example.com/companies/CS111111111"
        }
    },
    "publicationDate": "2022-01-01",
    "collectionDate": "2021-09-30",
    "crs": {
        "name": "urn:ogc:def:crs:OGC::CRS84",
        "uri": "http://www.opengis.net/def/crs/OGC/1.3/CRS84"
    },
    "accuracy": 250,
    "accuracyDetails": "Measured according to the US National Standard for Spatial Data Accuracy.",
    "language": "en",
    "links": [
        {
            "rel": "describedby",
            "href": "https://raw.githubusercontent.com/Open-Telecoms-Data/open-fibre-data-standard/0__3__0/schema/network-schema.json"
        }
    ]
}

The following example shows a network with references to separate endpoints for nodes and spans:

{
    "id": "a096d627-72e1-4f9b-b129-951b1737bff4",
    "name": "Ghana Fibre Network",
    "links": [
        {
            "rel": "describedby",
            "href": "https://raw.githubusercontent.com/Open-Telecoms-Data/open-fibre-data-standard/0__3__0/schema/network-schema.json"
        },
        {
            "href": "http://example.com/networks/a096d627-72e1-4f9b-b129-951b1737bff4/nodes",
            "rel": "tag:opentelecomdata.net,2022:nodesAPI"
        },
        {
            "href": "http://example.com/networks/a096d627-72e1-4f9b-b129-951b1737bff4/spans",
            "rel": "tag:opentelecomdata.net,2022:spansAPI"
        }
    ]
}

The following example shows the response returned by the nodes endpoint with URLs for the next and previous pages of results.

{
    "nodes": [
        {},
        {}
    ],
    "links": {
        "next": "http://example.com/network/1/nodes?cursor=100",
        "prev": "http://example.com/network/1/nodes?cursor=96"
    }
}

The following example shows the response returned by the spans endpoint with URLs for the next and previous pages of results.

{
    "spans": [
        {},
        {}
    ],
    "links": {
        "next": "http://example.com/network/1/spans?cursor=100",
        "prev": "http://example.com/network/1/spans?cursor=96"
    }
}
Streaming

The preferred approach is to publish embedded nodes and spans. If your network is too large to load into memory, you ought to use .links to reference separate files for nodes and spans. Each file ought to be formatted as a JSON Lines file in which each line is a valid Node or Span, respectively.

The following example shows a network with embedded nodes and spans:

{
    "id": "a096d627-72e1-4f9b-b129-951b1737bff4",
    "name": "Ghana Fibre Network",
    "nodes": [
        {
            "id": "1",
            "name": "Accra",
            "phase": {
                "id": "1",
                "name": "NextGen Phase 1"
            },
            "status": "operational",
            "location": {
                "type": "Point",
                "coordinates": [
                    -0.174,
                    5.625
                ]
            },
            "address": {
                "streetAddress": "Boundary Rd",
                "locality": "Shiashie",
                "region": "Accra",
                "postalCode": "12345",
                "country": "GH"
            },
            "type": [
                "pop"
            ],
            "accessPoint": true,
            "internationalConnections": [
                {
                    "region": "Lom\u00e9",
                    "country": "TG"
                }
            ],
            "power": true,
            "technologies": [
                "ip"
            ],
            "physicalInfrastructureProvider": {
                "id": "GH-RGD-CS111111111",
                "name": "FibreCo"
            },
            "networkProviders": [
                {
                    "id": "GH-RGD-CS222222222",
                    "name": "FastWeb"
                }
            ]
        },
        {
            "id": "2",
            "name": "Kumasi",
            "phase": {
                "id": "1",
                "name": "NextGen Phase 1"
            },
            "status": "operational",
            "location": {
                "type": "Point",
                "coordinates": [
                    -1.628,
                    6.711
                ]
            },
            "address": {
                "streetAddress": "Kejetia Road",
                "locality": "Bantama",
                "region": "Kumasi",
                "postalCode": "00000",
                "country": "GH"
            },
            "type": [
                "pop"
            ],
            "accessPoint": false,
            "power": true,
            "technologies": [
                "ip"
            ],
            "physicalInfrastructureProvider": {
                "id": "GH-RGD-CS111111111",
                "name": "FibreCo"
            },
            "networkProviders": [
                {
                    "id": "GH-RGD-CS222222222",
                    "name": "FastWeb"
                }
            ]
        }
    ],
    "spans": [
        {
            "id": "1",
            "name": "Accra to Kumasi",
            "phase": {
                "id": "1",
                "name": "NextGen Phase 1"
            },
            "status": "operational",
            "readyForServiceDate": "2018-01-01",
            "start": "1",
            "end": "2",
            "directed": false,
            "route": {
                "type": "LineString",
                "coordinates": [
                    [
                        -0.173,
                        5.626
                    ],
                    [
                        -0.178,
                        5.807
                    ],
                    [
                        -0.112,
                        5.971
                    ],
                    [
                        -0.211,
                        5.963
                    ],
                    [
                        -0.321,
                        6.17
                    ],
                    [
                        -0.488,
                        6.29
                    ],
                    [
                        -0.56,
                        6.421
                    ],
                    [
                        -0.752,
                        6.533
                    ],
                    [
                        -0.867,
                        6.607
                    ],
                    [
                        -1.101,
                        6.585
                    ],
                    [
                        -1.304,
                        6.623
                    ],
                    [
                        -1.461,
                        6.727
                    ],
                    [
                        -1.628,
                        6.713
                    ]
                ]
            },
            "physicalInfrastructureProvider": {
                "id": "GH-RGD-CS111111111",
                "name": "FibreCo"
            },
            "networkProviders": [
                {
                    "id": "GH-RGD-CS222222222",
                    "name": "FastWeb"
                }
            ],
            "supplier": {
                "id": "GH-RGD-CS333333333",
                "name": "ProjectCo"
            },
            "transmissionMedium": [
                "fibre"
            ],
            "deployment": [
                "belowGround"
            ],
            "deploymentDetails": {
                "description": "Buried in ducts"
            },
            "darkFibre": true,
            "fibreType": "G.652",
            "fibreTypeDetails": {
                "fibreSubtype": "G.652.D",
                "description": "CABLED STANDARD ENHANCED SINGLEMODE FIBRE ESMF, LOW WATER PEAK SINGLE MODE FIBRE G.652.D, OS2"
            },
            "fibreCount": 24,
            "fibreLength": 276000,
            "technologies": [
                "sdh"
            ],
            "capacity": 4.976,
            "capacityDetails": {
                "description": "2xSTM-16"
            },
            "countries": [
                "GH"
            ]
        }
    ],
    "phases": [
        {
            "id": "1",
            "name": "NextGen Phase 1",
            "description": "The first phase of the Ghana Fibre Network NextGen roll-out.",
            "funders": [
                {
                    "id": "1",
                    "name": "Ghana Infrastructure Investment Fund"
                }
            ]
        }
    ],
    "organisations": [
        {
            "id": "GH-RGD-CS111111111",
            "name": "FibreCo",
            "identifier": {
                "id": "CS111111111",
                "scheme": "GH-RGD",
                "legalName": "FibreCo Ghana Limited",
                "uri": "https://example.com/companies/CS111111111"
            },
            "country": "GH",
            "roles": [
                "physicalInfrastructureProvider"
            ],
            "roleDetails": "FibreCo owns and maintains the physical infrastructure.",
            "website": "https://www.example.com",
            "logo": "https://www.example.com/logo.png"
        },
        {
            "id": "GH-RGD-CS222222222",
            "name": "FastWeb",
            "identifier": {
                "id": "CS222222222",
                "scheme": "GH-RGD",
                "legalName": "FastWeb Ghana Limited",
                "uri": "https://example.com/companies/CS222222222"
            },
            "country": "GH",
            "roles": [
                "networkProvider"
            ],
            "roleDetails": "FastWeb provides wholesale services to service providers and dark fibre access to other network providers.",
            "website": "https://www.example.com",
            "logo": "https://www.example.com/logo.png"
        },
        {
            "id": "GH-RGD-CS333333333",
            "name": "ProjectCo",
            "identifier": {
                "id": "CS333333333",
                "scheme": "GH-RGD",
                "legalName": "ProjectCo Ghana Limited",
                "uri": "https://example.com/companies/CS333333333"
            },
            "country": "GH",
            "roles": [
                "supplier"
            ],
            "roleDetails": "ProjectCo was responsible for the construction of phase 1 of the NextGen network.",
            "website": "https://www.example.com",
            "logo": "https://www.example.com/logo.png"
        },
        {
            "id": "1",
            "name": "Ghana Infrastructure Investment Fund",
            "country": "GH",
            "roles": [
                "funder"
            ],
            "roleDetails": "The Ghana Infrastructure Investment Fund provided equity financing for the construction of phase 1 of the NextGen network.",
            "website": "https://www.example.com",
            "logo": "https://www.example.com/logo.png"
        }
    ],
    "contracts": [
        {
            "id": "1",
            "title": "NextGen Phase 1 Construction Contract",
            "description": "Contract for the construction of phase 1 of the NextGen network.",
            "type": "private",
            "value": {
                "amount": 1537459500.0,
                "currency": "GHS"
            },
            "dateSigned": "2016-06-30",
            "documents": [
                {
                    "title": "NextGen Phase 1 Construction Contract",
                    "description": "Signed contract.",
                    "url": "http://example.com/contracts/nextgen-phase1-construction.pdf",
                    "format": "application/pdf"
                }
            ],
            "relatedPhases": [
                {
                    "id": "1",
                    "name": "NextGen Phase 1"
                }
            ]
        }
    ],
    "website": "http://example.com/network",
    "publisher": {
        "name": "FibreCo",
        "identifier": {
            "id": "CS111111111",
            "scheme": "GH-RGD",
            "legalName": "FibreCo Ghana Limited",
            "uri": "https://example.com/companies/CS111111111"
        }
    },
    "publicationDate": "2022-01-01",
    "collectionDate": "2021-09-30",
    "crs": {
        "name": "urn:ogc:def:crs:OGC::CRS84",
        "uri": "http://www.opengis.net/def/crs/OGC/1.3/CRS84"
    },
    "accuracy": 250,
    "accuracyDetails": "Measured according to the US National Standard for Spatial Data Accuracy.",
    "language": "en",
    "links": [
        {
            "rel": "describedby",
            "href": "https://raw.githubusercontent.com/Open-Telecoms-Data/open-fibre-data-standard/0__3__0/schema/network-schema.json"
        }
    ]
}

The following example shows a network with references to separate files for nodes and spans:

{
    "id": "a096d627-72e1-4f9b-b129-951b1737bff4",
    "name": "Ghana Fibre Network",
    "links": [
        {
            "rel": "describedby",
            "href": "https://raw.githubusercontent.com/Open-Telecoms-Data/open-fibre-data-standard/0__3__0/schema/network-schema.json"
        },
        {
            "href": "http://example.com/networks/a096d627-72e1-4f9b-b129-951b1737bff4/nodes.jsonl",
            "rel": "tag:opentelecomdata.net,2022:nodesFile"
        },
        {
            "href": "http://example.com/networks/a096d627-72e1-4f9b-b129-951b1737bff4/spans.jsonl",
            "rel": "tag:opentelecomdata.net,2022:spansFile"
        }
    ]
}

The following example shows a nodes file in JSON Lines format.

{}
{}

The following example shows a spans file in JSON Lines format.

{}
{}

How to provide access to data

Where resources allow, it is best practice to provide multiple access methods for your data so that both humans and machines can access it easily.

With respect to your OFDS publication, which best practices are most important will depend on your priority use cases, but you are encouraged to consider providing bulk downloads and API access.

Bulk downloads

If you are publishing only one network, or a small number of networks, you ought to use the approach described in the small file option for each publication format.

If you are publishing a large number of networks, you ought to use the approach to streaming multiple networks described in the streaming option for each publication format.

If you are publishing a network that is very large, you ought to use the approach to streaming nodes and spans described in how to publish large networks.

Compression

OFDS data can be compressed in order to save on disk space and bandwidth.

When compressing packages, use ZIP or GZIP, as these are commonly available, often without additional software. Avoid RAR, which requires additional software.

Serving files

The web server providing access to bulk files ought to report the HTTP Last-Modified header correctly, so that consuming applications only need to download updated files.

Also, publishers ought to ensure that the data export is completed successfully, i.e. that no files were truncated.

API access

If you are publishing data via an API, you need to consider pagination. If you are publishing multiple networks, you ought to use the pagination method described in the API response option for each publication format.

If you are publishing a network that is very large, you ought to use the approach to paginating nodes and spans described in how to publish large networks.

API design is a deep topic. As such, the following guidance is not intended to be comprehensive or prescriptive. Wherever possible, you ought to carry out your own user research.

Discoverability

Ensure that the API endpoints and documentation are discoverable. For example, add a link to the footer of your website, and list the API endpoints in your government’s open data portal.

Documentation

Provide API documentation, with at least the lists of endpoints, methods and parameters. Many open data publishers use Swagger to document their APIs.

Access control and rate limiting

Avoid adding access controls (like user registration or API keys), in order to maximise the ease of access to the publication.

If access controls are necessary, do not use access tokens that need to be refreshed regularly. For example, every two hours is too frequent.

If the API implements rate limits (throttling):

Completeness

Ensure that all OFDS data can be accessed via the API.

Response format

  • Put the network package or GeoJSON feature collection at the top-level of the JSON data. For example, do not embed it under a results array.

  • Use a JSON library instead of implementing JSON serialisation yourself. This also guarantees that the encoding is UTF-8.

  • Remove NULL characters (\u0000) from the JSON response. These characters cannot be imported by users into some SQL databases.

  • If results cannot be returned, use an appropriate HTTP error code (400-599); do not return a JSON object with an error message and a 200 HTTP status code. That said, if a search request returns no results, it is appropriate to use a 200 HTTP status code, with an empty result set.

Monitoring

Set up error monitoring, so that if a request causes an HTTP 500 Internal Server Error, you can investigate.

How to transform coordinates to the correct coordinate reference system

To publish OFDS data, you need to specify coordinates in the urn:ogc:def:crs:OGC::CRS84 coordinate reference system (CRS). If the coordinates in your data sources are specified in a different CRS, before publishing your data in OFDS format, you first need to transform the coordinates to the correct CRS.

If your data pipeline includes a Geographic Information System such as ArcGIS or QGIS, these tools can transform coordinates from one CRS to another. If you are writing your own software, or if you prefer to use the command line, several libraries and tools are available, for example:

  • PROJ and its associated Python interface (PYPROJ) and JavaScript implementation (PROJ4JS are generic coordinate transformation tools that transform geospatial coordinates from one coordinate reference system (CRS) to another. They include command-line applications and an application programming interface.

  • GDAL is a translator library for raster and vector geospatial data formats. It also comes with a variety of useful command line utilities for data translation and processing.

  • Apache SIS is a free software, Java language library for developing geospatial applications. SIS provides data structures for geographic features and associated metadata along with methods to manipulate those data structures.

If you prefer to use a graphical user interface, several web-based tools are available, for example:

The urn:ogc:def:crs:OGC::CRS84 CRS is equivalent to EPSG:4326 with reversed axes so, if it is not supported by your chosen transformation tool, you can instead transform your coordinates to EPSG:4326 and manually order your coordinates in longitude, latitude order.

How to generate universally unique identifiers

If you are writing your own software or if you prefer to use the command line, several libraries and tools are available to generate universally unique identifiers (UUIDS), for example:

If you prefer to use a graphical user interface, several web-based tools are available, for example Online UUID Generator.

How to write a data user guide

Publishing OFDS data involves making choices about what data to include and exclude, and how to map existing data elements to the fields in OFDS.

In order for users to interpret data correctly and make effective use of it, it’s important to describe your decisions and to provide guidance to data users. Your data user guide ought to include:

Your data user guide ought to be made available as a public web page. You ought to link to the web page wherever you publish links to your data.

How to license your data

Publishing your data under an open license is important because it prevents restrictions on re-use, which could limit the usefulness of the data.

You are encouraged to use either a public domain dedication/certification or an attribution-only license:

The Open Knowledge Foundation maintains a list of licenses that conform to the open definition. If you use a custom license, you ought to check that it conforms to the open definition.

You need to ensure that a clear license statement is provided wherever publish links to your data.