Making Sense of Microposts (#MSM2013)

Big things come in small packages

Frequently Asked Questions — Concept Extraction Challenge

Theme: Making Sense of Microposts: Big things come in small packages

Award Sponsor: eBay


  • Is the challenge dataset licensed? What are the restrictions to (re-)using it?

    Creative Commons License

    Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License

  • I believe I've spotted an error in the annotations provided in the training dataset. How will such errors affect the evaluation of my solution?

    Let us know – we will verify this, and if we agree that it is indeed an error, we will update the training set, and also post a list of errors on the challenge pages. If you wish to receive immediate updates please join the #MSM mailing list.

  • There are some funny characters in the dataset. What encoding should I use to read it properly?

    These are due to issues with the encoding when the data was first retrieved. Try UTF-8 or ISO-8859-1; however this should not affect the parsing, you may simply filter any unusual characters out.

  • Are hashtags containing entities (concept hashtags) taken into account in the evaluation?

    No, we ignore hashtags, e.g., the following only annotates the first instance of 'Beirut' (without a hashtag) as a location, and also ignores '#Lebanon':
    LOC/Beirut; _Mention_ : The beautiful empty streets of Beirut \* sigh \* -LRB- = = panoramic view ; -RRB- ; #Beirut #Lebanon

    Updates to the (training) dataset (from v1.4 — 15 Mar 2013) also encode all hashtags as _HASHTAG_
    LOC/Beirut; _Mention_ : The beautiful empty streets of Beirut \* sigh \* '' -LRB- = = panoramic view ; -RRB- _HASHTAG_ _HASHTAG_

  • Are the microposts in the data sets in English only?

    Yes, we attempted to remove multi-lingual posts. If you discover any, please notifiy us and we will update the training set and post the updates to the challenge pages and via the #MSM mailing list. Any such posts will be discounted in the evaluation process.

  • What was the basis for selecting the types under the MISC category? Why does it include TV-shows, say, but not songs?

    MISC is broken down based on mappings from CONLL and OpenCalais; these are:

  • What qualifies as a location? Some of the annotated examples included what would be considered as an adjective — is this correct?

    Yes, this is because some institutions, for example, have a name that includes an adjective, e.g., OpenCalais provides examples of such instances for the concept Facility:
    Welsh High School hall is a school with a name that contains the adjective Welsh,
    Pratica di Mare military base is Europe's second largest military base, located 30 kilometers south of Rome, also contains a descriptive name.
    An example in the training set is the annotation:
    LOC/Thai Buddhist temple in the post:
    1 "2,000 fetuses found hidden at Thai Buddhist temple _URL_ via _Mention_" - the instance of the LOC in this case is the specific temple referred to.
    Note however that the evaluation is independent of OpenCalais — other examples may be found in CONLL and other relevant external sources.

  • I am testing my approach to carrying out the IE task. Can I get a copy of the evaluation script in order to fine tune my system and verify my results?

    The evaluation script will not be shared with contestants — this will be used by the challenge committee to carry out an independent and fair evaluation of each submission. Participants should calculate precision, recall and f-measure as per the submission instructions. The proceedings will contain extended abstracts of accepted submissions, a summary of the evaluation procedure and highlight also interesting and/or unusual solutions.

  • I want to hedge my bets, how many runs can I submit to the challenge?

    Up to 3 runs may be included in each submission. You may therefore modify the parameters for your approach for 3 separate attempts at extracting concepts from the test set. You must submit each run in a TSV file and describe the settings for each clearly in your extended abstract. The challenge committee will evaluate each run independently and judge the submission based on that with the best performance (based on the F1 score). Submission detail is available on the challenge page.

  • Can I trust the results of the evaluation of my solution?

    The challenge chairs will evaluate all results using a pre-prepared quantitative approach as described in the call. An independent supervisory committee will verify and validite the results, and also review the written descriptions submitted.

  • Can I use external sources (e.g. Linked Open Data, AlchemyAPI) in building my solution?

    Yes. However you must list these, and for data sources specify what they are and whether they are open or closed sources. Other APIs and existing tools and services may be reused and/or extended; your must however detail the novelty in your solution and what new contribution it makes over existing work.

  • Your concepts/categories are based on OpenCalais and CONLL definitions — does the evaluation therefore rely on tool-specific concept schema? Would this not bias the evaluation?

    The annotation was carried out manually, without any prior information on how instances would be annotated by third parties. The evaluation is therefore independent of OpenCalais, CONLL or any other relevant resource (e.g., Zemanta).

  • Detailed challenge call


  • What if I've more questions?

    Contact the #MSM team via e-mail, tweet or post to facebook or the W3C community group,
    and/or join the #MSM2013 mailing list

back to top