Submission Rules

This page describes the official submission rules for KnowledgeGraphEval 2026. Participants must follow the required output format for each subtask and submit their final predictions through the official Codabench competition pages.

Teams may participate in one or more subtasks. Final rankings are based only on valid official submissions made before the shared task deadline.

Note: Participation in the shared task implies consent for organizers to access and inspect submitted systems and outputs for evaluation and verification purposes.
General Rules Subtask 1 Output Subtask 2 Output Verification Checklist Submission Support

General Rules

Official platform

Submit final outputs through the official Codabench pages for the corresponding subtask.

Preserve IDs

Use the released sentence IDs, token IDs, entity IDs, and pair IDs exactly as they appear in the blind test files.

No hidden-label access

Do not manually inspect, infer, reconstruct, or use hidden gold labels during the evaluation phase.

Valid submissions

A submission is valid only if it follows the required file format, encoding, schema, and label inventory for the selected subtask.

All submitted files must use UTF-8 encoding. Participants should run the released validation script before uploading their outputs.

1

Cross-Domain Named Entity Recognition

AdaptNER output rules

For the NER subtask, systems must produce one named entity tag for each token in the blind test file. Token order and sentence boundaries must be preserved exactly.

Required output

  • Use the official CoNLL sequence-labeling format.
  • Provide one predicted BIO tag for each token.
  • Use only the official KnowledgeGraphEval entity label inventory.
  • Do not remove, reorder, normalize, or retokenize the test tokens.
  • Keep blank lines between sentences, following the released template.

Evaluation

Systems are evaluated using entity-level micro F1-score. A prediction is counted as correct only when the entity span and entity type exactly match the gold annotation.

2

Relation Extraction

Relation classification with predefined entities

For the RE subtask, predefined entity mentions are provided. Systems must predict the semantic relation label for each valid entity pair.

Required output

  • Submit predictions using the official TXT template released with the test data.
  • Preserve the sentence ID for every test instance.
  • Predict one relation label for each pair.
  • Use no-relation when no valid semantic relation exists.
  • Use only the official relation label inventory.

Required fields

sentence_id, predicted_relation

Evaluation

Systems are evaluated using micro F1-score over predicted relation labels. Official scores are computed using the released evaluation script.

Verification Checklist

  1. Run the released validation script before uploading predictions.
  2. Confirm that the file is encoded in UTF-8.
  3. Confirm that the output format is valid CoNLL or TXT as required.
  4. Check that every test example has exactly the expected number of predictions.
  5. Check that all predicted labels belong to the official label inventory.
  6. Keep a copy of the submitted outputs, model configuration, prompts, and post-processing scripts.
  7. Email the organizers before the deadline if Codabench rejects a valid file.

Submission Support

For submission issues, validation errors, or Codabench problems, contact the organizers before the official deadline.

Email: KnowledgeGraphEval@gmail.com

Back to Submission Section