PROPOSED Entity Reconciliation Working Group Charter
The mission of the Entity Reconciliation Working Group is to develop a protocol for entity reconciliation on the Web. This process, also known as entity linking or entity resolution, involves identifying and linking entities across different datasets.
This proposed charter is available on GitHub. Feel free to raise issues.
| Charter Status | See the group status page and detailed change history. |
|---|---|
| Start date | [dd monthname yyyy] (date of the "Call for Participation", when the charter is approved) |
| End date | [dd monthname yyyy] (Start date + 2 years) |
| Chairs | Antonin Delpeuch (Institut Mines-Télécom) Adrian Pohl (Hochschulbibliothekszentrum des Landes NRW) |
| Team Contacts | [team contact name] (0.1 FTE) |
| Meeting Schedule |
Teleconferences: teleconferences will focus on discussion of specifications and implementations, and will be conducted on a monthly basis.
Face-to-face: no face-to-face meetings are currently planned, but the group may choose to have face-to-face meetings if needed and if circumstances allow. |
Motivation
Many applications, both commercial and open-source, provide a way to link external data to textual entities like places, persons, etc. For instance, Microsoft Excel has linked entity cell values or Grist has reference columns. These links disambiguate the different names we use for entities and provide additional information, e.g. to show places on a map, or images and contact details for persons. To achieve that, applications must provide a way to connect to an external data provider and select the appropriate entities to link.
Because the Web is decentralized and any service can publish a new URI for an entity that already exists elsewhere, entities have multiple identifiers and varying attributes or descriptions across the Web. To integrate these for a specific entity, it is necessary to determine which URIs refer to the same entity across different datasets by comparing attributes (e.g. the name, country and mayor of a city). Without a standardized protocol, applications that want to link these entities require a dedicated importer or add-in to connect to each data provider (like Wikidata, ORCID, GeoNames, etc). At the same time, every application interested in integrating a specific data provider has to implement their own importer or connector.
With a standardized protocol, 1.) a single client implementation can integrate any conforming data service and 2.) a single data service implementation can be used in any conforming client application. So instead of implementing separate Excel add-ins or Grist importers for every data source, we can have a single reconciliation client in Excel or Grist that can talk to any data provider, and instead of implementing separate Wikidata extensions for every application, we can have a single Wikidata reconciliation service that can be used in any client application.
Providing an API to support this reconciliation process, which includes matching, previewing, suggesting, and extending, is the primary motivation behind this working group. It aims to develop a protocol that 1.) a tool provider can implement, enabling its users to efficiently integrate external data sources, and 2.) a data provider can implement, enabling its consumers to efficiently match their own data to the entities represented by the provider.
Background
The reconciliation API was originally designed in 2008 by Metaweb as a protocol for a use case in Freebase Gridworks (later Google Refine, since 2012 OpenRefine), where entities from a local dataset are matched to Freebase records during import. It was later generalized so any database could implement it, allowing data providers such as Crossref, Nomisma, Getty, Wikidata, and VIAF to be added as a Standard Service in OpenRefine. The API, originally documented on OpenRefine's wiki, also supports additional optional features like preview for entities, auto-suggest for various inputs, and, since 2018, data extension for pulling data from reconciled records in the target database. In 2019 the W3C Entity Reconciliation Community Group formed to promote and refine the specification beyond OpenRefine. It published two versions of the specification as CG reports.
Scope
The group aims to write specifications for a Web-based protocol for record linkage. This protocol is largely based on the one popularized by the OpenRefine application, which enables reconciliation of locally-held data to the identifiers curated by an authority database, exposed by a web server.
The Working Group therefore welcomes work in the following forms:
- Designing the specifications of the reconciliation API;
- Documenting how other existing protocols can be used for reconciliation on the web;
- Writing test suites, validators, libraries and other tools for such protocols;
- Surveying the existing workflows and available implementations;
- Promoting any suitable protocol around reconciliation to data providers.
Out of Scope
The following features are out of scope, and will not be addressed by this Working group.
- Implementation of the protocol in specific clients, such as OpenRefine, although clients should of course be kept in mind when designing any protocol;
- Implementation of the protocol in specific data providers;
- Creation of datasets for benchmarks of heuristics (this is done by OAEI already);
- Hosting of reconciliation services.
Deliverables
Updated document status is available on the group publication status page.
Draft state indicates the state of the deliverable at the time of the charter approval. Expected completion indicates when the deliverable is projected to become a Recommendation, or otherwise reach a stable state.
Normative Specifications
The Working Group will deliver the following W3C normative specifications:
- Reconciliation protocol
-
This specification describes mechanisms for web-based record linkage that can be used to reconcile locally-held data to the identifiers curated by an authority database.
Draft state: Draft Community Group Report
Expected completion: At the end of the Working Group's lifespan
Initial Draft: Reconciliation Service API (Draft Community Report, latest editor's draft)
Other Deliverables
Other non-normative documents may be created such as:
- Up-to-date test bench for all normative deliverables
- An up-to-date census of the reconciliation ecosystem
Timeline
- Month 1: publish the current draft of the specifications as a First Public Working Draft (FPWD)
- Month 2: hold public online meeting with stakeholders requesting implementer's feedback on the draft
- Month 6: publish the current draft of the specifications as a Candidate Recommendation
- Month 12: hold public online meeting to review implementer's feedback on the existing draft
- Month 18: request advancement to W3C Recommendation
Success Criteria
In order to advance to Proposed Recommendation, each normative specification is expected to have at least two independent implementations of every feature defined in the specification, where interoperability can be verified in an open testing environment.
Each specification should contain sections detailing all known security, privacy, and accessibility implications for implementers and end users, including ways specification features can be used to address them, and recommendations for implementers.
This group is expected to be guided by the following documents:
Coordination
For all specifications, this Working Group will seek horizontal review for accessibility, internationalization, performance, privacy, and security with the relevant Working and Interest Groups, and with the TAG. Invitation for review must be issued during each major standards-track document transition, including FPWD. The Working Group is encouraged to engage collaboratively with the horizontal review groups throughout development of each specification. The Working Group is advised to seek a review at least 3 months before first entering CR and is encouraged to proactively notify the horizontal review groups when major changes occur in a specification following a review.
Additional technical coordination with the following Groups will be made, per the W3C Process Document.
In addition to the above catch-all reference to horizontal review which includes accessibility review, please check with chairs and staff contacts of the Accessible Platform Architectures Working Group to determine if an additional liaison statement with more specific information about concrete review issues is needed in the list below.
W3C Groups
- None
External Organizations
- OpenRefine
- as software popularizing the original version of the protocol
- Wikimedia Deutschland
- as organization interested in hosting reconciliation services for Wikibase.Cloud
- e-editiones
- as maintainer of TEI publisher, a software that includes a reconciliation client and, perspectively, a server
Participation
To be successful, this Working Group is expected to have 6 or more active participants for its duration, including representatives from the key implementors of this specification, and active Editors and Test Leads for each specification. The Chairs, specification Editors, and Test Leads are expected to contribute half of a working day per week towards the Working Group. There is no minimum requirement for other Participants. Mention: The expected time commitment and level of involvement by the Team, see https://www.w3.org/2021/Process-20211102/#WGCharter
The group encourages questions, comments and issues on its public mailing lists and document repositories, as described in Communication.
The group also welcomes non-Members to contribute technical submissions for consideration upon their agreement to the terms of the W3C Patent Policy.
Participants in the group are required (by the W3C Process) to follow the W3C Code of Ethics and Professional Conduct.
Communication
Technical discussions for this Working Group are conducted in public: the meeting minutes from teleconference and face-to-face meetings will be archived for public review, and technical discussions and issue tracking will be conducted in a manner that can be both read and written to by the general public. Working Drafts and Editor's Drafts of specifications will be developed in public repositories and may permit direct public contribution requests. The meetings themselves are not open to public participation, however.
Information about the group (including details about deliverables, issues, actions, status, participants, and meetings) will be available from the Entity Reconciliation Working Group home page.
Most Entity Reconciliation Working Group teleconferences will focus on discussion of specifications and implementations, and will be conducted on a monthly basis.
This group primarily conducts its technical work on GitHub issues. The public is invited to review, discuss and contribute to this work.
The group may use a Member-confidential mailing list for administrative purposes and, at the discretion of the Chairs and members of the group, for member-only discussions in special cases when a participant requests such a discussion.
Decision Policy
This group will seek to make decisions through consensus and due process, per the W3C Process Document (section 5.2.1, Consensus). Typically, an editor or other participant makes an initial proposal, which is then refined in discussion with members of the group and other reviewers, and consensus emerges with little formal voting being required.
However, if a decision is necessary for timely progress and consensus is not achieved after careful consideration of the range of views presented, the Chairs may call for a group vote and record a decision along with any objections.
To afford asynchronous decisions and organizational deliberation, any resolution (including publication decisions) taken in a face-to-face meeting or teleconference will be considered provisional. A call for consensus (CfC) will be issued for all resolutions (for example, via email, GitHub issue or web-based survey), with a response period from one week to 10 working days, depending on the chair's evaluation of the group consensus on the issue. If no objections are raised by the end of the response period, the resolution will be considered to have consensus as a resolution of the Working Group.
All decisions made by the group should be considered resolved unless and until new information becomes available or unless reopened at the discretion of the Chairs or the Director.
This charter is written in accordance with the W3C Process Document (Section 5.2.3, Deciding by Vote) and includes no voting procedures beyond what the Process Document requires.
Patent Policy
This Working Group operates under the W3C Patent Policy (Version of 15 September 2020). To promote the widest adoption of Web standards, W3C seeks to issue Web specifications that can be implemented, according to this policy, on a Royalty-Free basis. For more information about disclosure obligations for this group, please see the licensing information.
Licensing
This Working Group will use the W3C Software and Document license for all its deliverables.
About this Charter
This charter has been created according to section 3.4 of the Process Document. In the event of a conflict between this document or the provisions of any charter and the W3C Process, the W3C Process shall take precedence.
Charter History
Note:Display this table and update it when appropriate. Requirements for charter extension history are documented in the Charter Guidebook (section 4).
The following table lists details of all changes from the initial charter, per the W3C Process Document (section 4.3, Advisory Committee Review of a Charter):
| Charter Period | Start Date | End Date | Changes |
|---|---|---|---|
| Initial Charter | [dd monthname yyyy] | [dd monthname yyyy] | none |
| Charter Extension | [dd monthname yyyy] | [dd monthname yyyy] | none |
| Rechartered | [dd monthname yyyy] | [dd monthname yyyy] |
[description of change to charter, with link to new deliverable item in charter] Note: use the class |
Change log
Changes to this document are documented in this section.