{"id":13306,"date":"2019-07-30T07:31:47","date_gmt":"2019-07-30T07:31:47","guid":{"rendered":"https:\/\/re20.org\/\/?page_id=13306"},"modified":"2020-05-08T14:51:19","modified_gmt":"2020-05-08T12:51:19","slug":"artifacts","status":"publish","type":"page","link":"https:\/\/re20.org\/index.php\/artifacts\/","title":{"rendered":"Artifacts"},"content":{"rendered":"<style type=\"text\/css\"> \n#cmsmasters_row_69eb68c8251b08_28525302 .cmsmasters_row_outer_parent { \n\tpadding-top: 0px; \n} \n\n#cmsmasters_row_69eb68c8251b08_28525302 .cmsmasters_row_outer_parent { \n\tpadding-bottom: 50px; \n} \n\n \n#cmsmasters_row_69eb68c82532d6_31297369 .cmsmasters_row_outer_parent { \n\tpadding-top: 0px; \n} \n\n#cmsmasters_row_69eb68c82532d6_31297369 .cmsmasters_row_outer_parent { \n\tpadding-bottom: 50px; \n} \n\n \n#cmsmasters_row_69eb68c826b6b1_17390184 .cmsmasters_row_outer_parent { \n\tpadding-top: 0px; \n} \n\n#cmsmasters_row_69eb68c826b6b1_17390184 .cmsmasters_row_outer_parent { \n\tpadding-bottom: 50px; \n} \n\n \n#cmsmasters_row_69eb68c826dbd0_84632738 .cmsmasters_row_outer_parent { \n\tpadding-top: 0px; \n} \n\n#cmsmasters_row_69eb68c826dbd0_84632738 .cmsmasters_row_outer_parent { \n\tpadding-bottom: 50px; \n} \n\n \n#cmsmasters_heading_69eb68c827191 { \n\ttext-align:left; \n\tmargin-top:0px; \n\tmargin-bottom:20px; \n} \n\n#cmsmasters_heading_69eb68c827191 .cmsmasters_heading { \n\ttext-align:left; \n} \n\n#cmsmasters_heading_69eb68c827191 .cmsmasters_heading, #cmsmasters_heading_69eb68c827191 .cmsmasters_heading a { \n\tfont-weight:normal; \n\tfont-style:normal; \n} \n\n#cmsmasters_heading_69eb68c827191 .cmsmasters_heading_divider { \n} \n\n \n#cmsmasters_heading_69eb68c827286 { \n\ttext-align:left; \n\tmargin-top:0px; \n\tmargin-bottom:20px; \n} \n\n#cmsmasters_heading_69eb68c827286 .cmsmasters_heading { \n\ttext-align:left; \n} \n\n#cmsmasters_heading_69eb68c827286 .cmsmasters_heading, #cmsmasters_heading_69eb68c827286 .cmsmasters_heading a { \n\tfont-weight:normal; \n\tfont-style:normal; \n} \n\n#cmsmasters_heading_69eb68c827286 .cmsmasters_heading_divider { \n} \n\n \n#cmsmasters_heading_69eb68c827374 { \n\ttext-align:left; \n\tmargin-top:0px; \n\tmargin-bottom:20px; \n} \n\n#cmsmasters_heading_69eb68c827374 .cmsmasters_heading { \n\ttext-align:left; \n} \n\n#cmsmasters_heading_69eb68c827374 .cmsmasters_heading, #cmsmasters_heading_69eb68c827374 .cmsmasters_heading a { \n\tfont-weight:normal; \n\tfont-style:normal; \n} \n\n#cmsmasters_heading_69eb68c827374 .cmsmasters_heading_divider { \n} \n\n \n#cmsmasters_heading_69eb68c827554 { \n\ttext-align:left; \n\tmargin-top:0px; \n\tmargin-bottom:20px; \n} \n\n#cmsmasters_heading_69eb68c827554 .cmsmasters_heading { \n\ttext-align:left; \n} \n\n#cmsmasters_heading_69eb68c827554 .cmsmasters_heading, #cmsmasters_heading_69eb68c827554 .cmsmasters_heading a { \n\tfont-weight:normal; \n\tfont-style:normal; \n} \n\n#cmsmasters_heading_69eb68c827554 .cmsmasters_heading_divider { \n} \n\n \n#cmsmasters_heading_69eb68c827764 { \n\ttext-align:left; \n\tmargin-top:0px; \n\tmargin-bottom:20px; \n} \n\n#cmsmasters_heading_69eb68c827764 .cmsmasters_heading { \n\ttext-align:left; \n} \n\n#cmsmasters_heading_69eb68c827764 .cmsmasters_heading, #cmsmasters_heading_69eb68c827764 .cmsmasters_heading a { \n\tfont-weight:normal; \n\tfont-style:normal; \n} \n\n#cmsmasters_heading_69eb68c827764 .cmsmasters_heading_divider { \n} \n\n<\/style><div id=\"cmsmasters_row_69eb68c8251b08_28525302\" class=\"cmsmasters_row cmsmasters_color_scheme_default cmsmasters_row_top_default cmsmasters_row_bot_default cmsmasters_row_boxed\">\n<div class=\"cmsmasters_row_outer_parent\">\n<div class=\"cmsmasters_row_outer\">\n<div class=\"cmsmasters_row_inner\">\n<div class=\"cmsmasters_row_margin\">\n<div class=\"cmsmasters_column one_first\">\n<div class=\"cmsmasters_text\">\n<p>Authors of accepted papers in the research, RE@Next! and industry tracks of RE\u201920 are invited to submit an artifact to the Artifact Track. Research papers with artifacts receive a \u201cbadge\u201d on the front page of their paper in the proceedings. These badges certify that the paper has associated artifacts of the following form:<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<div id=\"cmsmasters_row_69eb68c82532d6_31297369\" class=\"cmsmasters_row cmsmasters_color_scheme_default cmsmasters_row_top_default cmsmasters_row_bot_default cmsmasters_row_boxed\">\n<div class=\"cmsmasters_row_outer_parent\">\n<div class=\"cmsmasters_row_outer\">\n<div class=\"cmsmasters_row_inner\">\n<div class=\"cmsmasters_row_margin\">\n<div class=\"cmsmasters_column one_fourth\">\n<div class=\"cmsmasters_text\">\n<p style=\"text-align: center;\"><strong>Artifacts Reusable<\/strong><\/p>\n<\/div>\n<div class=\"aligncenter\">\n<div class=\"cmsmasters_img  cmsmasters_image_c\">\n<img src=\"https:\/\/re20.org\/wp-content\/uploads\/2020\/03\/reusable-300x300.png\" \/>\n<\/div>\n<\/div>\n<div class=\"cmsmasters_text\">\n<p>The artifacts associated with the research are found to be documented, consistent, complete, exercisable, and include appropriate evidence of verification and in addition, they are very carefully documented and well-structured to the extent that reuse and repurposing is facilitated. In particular, norms and standards of the research community for artifacts of this type are strictly adhered to.<\/p>\n<\/div>\n<\/div>\n<div class=\"cmsmasters_column one_fourth\">\n<div class=\"cmsmasters_text\">\n<p style=\"text-align: center;\"><strong>Artifacts Available<\/strong><\/p>\n<\/div>\n<div class=\"aligncenter\">\n<div class=\"cmsmasters_img  cmsmasters_image_c\">\n<img src=\"https:\/\/re20.org\/wp-content\/uploads\/2020\/03\/available-300x300.png\" \/>\n<\/div>\n<\/div>\n<div class=\"cmsmasters_text\">\n<p>This badge is applied to papers in which associated artifacts have been made permanently available for retrieval. Author- created artifacts relevant to this paper have been placed on a publically accessible archival repository. A DOI or link to this repository along with a unique identifier for the object is provided.<\/p>\n<\/div>\n<\/div>\n<div class=\"cmsmasters_column one_fourth\">\n<div class=\"cmsmasters_text\">\n<p style=\"text-align: center;\"><strong>Results Replicated<\/strong><\/p>\n<\/div>\n<div class=\"aligncenter\">\n<div class=\"cmsmasters_img  cmsmasters_image_c\">\n<img src=\"https:\/\/re20.org\/wp-content\/uploads\/2020\/03\/replicated-300x300.png\" \/>\n<\/div>\n<\/div>\n<div class=\"cmsmasters_text\">\n<p>The main results of the paper have been obtained in a subsequent study by a person or team other than the authors, using, in part, artifacts provided by the author.<\/p>\n<\/div>\n<\/div>\n<div class=\"cmsmasters_column one_fourth\">\n<div class=\"cmsmasters_text\">\n<p style=\"text-align: center;\"><strong>Results Reproduced<\/strong><\/p>\n<\/div>\n<div class=\"aligncenter\">\n<div class=\"cmsmasters_img  cmsmasters_image_c\">\n<img src=\"https:\/\/re20.org\/wp-content\/uploads\/2020\/03\/reproduced-300x300.png\" \/>\n<\/div>\n<\/div>\n<div class=\"cmsmasters_text\">\n<p>The main results of the paper have been independently obtained in a subsequent study by a person or team other than the authors, without the use of author-supplied artifacts.<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<div id=\"cmsmasters_row_69eb68c826b6b1_17390184\" class=\"cmsmasters_row cmsmasters_color_scheme_default cmsmasters_row_top_default cmsmasters_row_bot_default cmsmasters_row_boxed\">\n<div class=\"cmsmasters_row_outer_parent\">\n<div class=\"cmsmasters_row_outer\">\n<div class=\"cmsmasters_row_inner\">\n<div class=\"cmsmasters_row_margin\">\n<div class=\"cmsmasters_column one_first\">\n<div class=\"cmsmasters_text\">\n<p><u>Disclaimer:<\/u>\u00a0Badges are loosely adopted from ACM badges as of December of 2019. For more details regarding the badges please visit https:\/\/www.acm.org\/publications\/policies\/artifact-review-badging.<br \/>\nPapers with such badges contain reusable products that other researchers can use to bootstrap their own research. Such papers are likely to earn increased citations and greater prestige in the research community. Artifacts of interest include the following. If your proposed artifact is not on this list, you are welcome to email the chairs before submitting.<\/p>\n<ul type=\"disc\">\n<li>Software, which are implementations of systems or algorithms potentially useful in other studies. For instance, tools to search for optimally consistent requirements specifications.<\/li>\n<li>Machine learning scripts, which can be understood as parts of a software tool, e.g., as Jupyter notebooks.<\/li>\n<li>Machine readable requirements models, e.g., in IstarML or UML interchange formats.<\/li>\n<li>Traceability relations between artifacts, such as requirements to source code.<\/li>\n<li>Data repositories, which are data (e.g., requirements models, requirements text, survey raw data) that can be used for multiple software engineering approaches.<\/li>\n<li>Requirements in natural language form, e.g., textual requirements in a spreadsheet, DOORS, JIRA export.<\/li>\n<li>User reviews, such as app reviews, product changelogs, and release notes.<\/li>\n<li>Frameworks, which are tools and services illustrating new approaches to requirements engineering that could be used by other researchers in different contexts. For example, a service to highlight inconsistencies in natural language.<\/li>\n<\/ul>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<div id=\"cmsmasters_row_69eb68c826dbd0_84632738\" class=\"cmsmasters_row cmsmasters_color_scheme_default cmsmasters_row_top_default cmsmasters_row_bot_default cmsmasters_row_boxed\">\n<div class=\"cmsmasters_row_outer_parent\">\n<div class=\"cmsmasters_row_outer\">\n<div class=\"cmsmasters_row_inner\">\n<div class=\"cmsmasters_row_margin\">\n<div class=\"cmsmasters_column one_first\">\n\n<div id=\"cmsmasters_heading_69eb68c827191\" class=\"cmsmasters_heading_wrap cmsmasters_heading_align_left\">\n\t<h1 class=\"cmsmasters_heading\">Selection Criteria<\/h1>\n<\/div><div class=\"cmsmasters_text\">\n<p>The RE\u201920 artifacts will be evaluated using the criteria in the last row of the above table. The goal of this track is to encourage reusable research products that can accelerate science.<\/p>\n<\/div>\n\n<div id=\"cmsmasters_heading_69eb68c827286\" class=\"cmsmasters_heading_wrap cmsmasters_heading_align_left\">\n\t<h1 class=\"cmsmasters_heading\">Review Process<\/h1>\n<\/div><div class=\"cmsmasters_text\">\n<ul>\n<li>Artifacts will be reviewed via an open review GitHub-based process. All reviews will be Github issues.<\/li>\n<li>Artifacts will be checked for their \u201cbadge worthiness\u201d by one reviewer unless there is an author\/reviewer dispute in which case a second check will be conducted by a second reviewer or one of the chairs.<\/li>\n<li>Both authors and reviewers will interact using their real GitHub ID (non-anonymously).<\/li>\n<li>Reviewers will have 7 days to assess the artifacts.<\/li>\n<li>Reviewers and authors will then have 3 days to interact such that (e.g.) if there is one line missing in a config file, then that bug can be fixed and the artifact declared \u201creusable\u201d.<\/li>\n<li>Track chairs will then meet to write a \u201cdecisions.md\u201d list written to the RE\u201920 GitHub (https:\/\/github.com\/researchart\/rose7re20) that contains a table of submissions and their associated badge (if any). These will be stored as labels in the Github repo.<\/li>\n<\/ul>\n<\/div>\n\n<div id=\"cmsmasters_heading_69eb68c827374\" class=\"cmsmasters_heading_wrap cmsmasters_heading_align_left\">\n\t<h1 class=\"cmsmasters_heading\">Submission Process<\/h1>\n<\/div><div class=\"cmsmasters_text\">\n<p>Step 1: Prepare all the supporting materials needed to evaluate an artifact:<\/p>\n<ul>\n<li>a README.md main markdown file describing what the artifact does;<\/li>\n<li>A STATUS.md markdown file stating what kind of badge you are applying for (one of reusable, available, replicated, reproduced) as well as the reasons why you think your artifact deserves that badge.<\/li>\n<li>a LICENSE.md markdown file describing the distribution rights (note that the license needs to be some form of open source license).<\/li>\n<li>an INSTALL.md markdown file with installation instructions. These instructions should include notes on what output to expect, which confirms the code is installed and working.<\/li>\n<li>For artifacts where the code \/ virtual machine \/ data sets \/ log files are larger than 1GB, do not place those artifacts within the RE\u201920 Artifacts GitHub repository, but make sure the INSTALL.md should include the appropriate download instructions.<\/li>\n<li>Enough associated code and data such that some CS person with a reasonable knowledge of scripting, build tools, etc. could install, build, run your code.<\/li>\n<\/ul>\n<p>Step 2: Contact the RE\u201920 artifacts chairs to get access to the submission repository, by communicating your GitHub ID.<\/p>\n<p>https:\/\/docs.google.com\/forms\/d\/e\/1FAIpQLScRpjcQsOHjjsTQqgEFE02z3xgW1fa5-bFypAorD69ib7PLEQ\/viewform<\/p>\n<p>Step 3: Submit all of your materials [artifact, *.md files] to a folder within the RE\u201920 repository. Use the &lt;paper ID&gt;-&lt;contact author last name&gt; template; for example, 23-smith or 34-jones. For large artifacts (&gt;1 GB), please store only the .md files in the folder, and make sure the instal.md file includes appropriate download instructions.<\/p>\n<\/div>\n<div id=\"cmsmasters_notice_69eb68c82749c2_88727645\" class=\"cmsmasters_notice cmsmasters_notice_warning cmsmasters-icon-alert\">\n<div class=\"notice_content\">\n<p>This call is only open for authors of a paper accepted in the research, RE@Next!, or industry innovation track.<\/p>\n<\/div>\n<\/div>\n<div class=\"cmsmasters_text\">\n<p><\/p>\n<\/div>\n\n<div id=\"cmsmasters_heading_69eb68c827554\" class=\"cmsmasters_heading_wrap cmsmasters_heading_align_left\">\n\t<h1 class=\"cmsmasters_heading\">Key Dates<\/h1>\n<\/div><div class=\"cmsmasters_wrap_table\"><table class=\"cmsmasters_table normal_table\"><caption><\/caption><tr><td class=\"cmsmasters_table_cell_alignleft\">Artifact Submission<\/td><td class=\"cmsmasters_table_cell_alignleft\">May 22, 2020<\/td><\/tr><tr><td class=\"cmsmasters_table_cell_alignleft\">Artifact Notification<\/td><td class=\"cmsmasters_table_cell_alignleft\">June 12, 2020<\/td><\/tr><tr><td class=\"cmsmasters_table_cell_alignleft\">Camera Ready Due<\/td><td class=\"cmsmasters_table_cell_alignleft\">June 22, 2020<\/td><\/tr><\/table><\/div><div class=\"cmsmasters_text\">\n<p><i>All deadlines are 23:59 <\/i><a href=\"http:\/\/www.timeanddate.com\/time\/zones\/aoe\" class=\"broken_link\"><i>Anywhere on Earth (Standard Time)<\/i><\/a><i>.<\/i><\/p>\n<\/div>\n\n<div id=\"cmsmasters_heading_69eb68c827764\" class=\"cmsmasters_heading_wrap cmsmasters_heading_align_left\">\n\t<h1 class=\"cmsmasters_heading\">Contact<\/h1>\n<\/div>\n<div id=\"69eb68c8277a01_77963744\" class=\"cmsmasters_profile vertical\">\n\r\n<!-- Start Vertical Profile -->\r\n\r\n<article id=\"post-13781\" class=\"cmsmasters_profile_vertical post-13781 profile type-profile status-publish has-post-thumbnail hentry pl-categs-artefacts pl-categs-organizing-committee\">\r\n\t<div class=\"profile_outer\">\r\n\t<figure class=\"cmsmasters_img_wrap\"><a href=\"https:\/\/re20.org\/index.php\/profile\/fabiano-dalpiaz\/\" title=\"Fabiano Dalpiaz\" class=\"cmsmasters_img_link\"><img width=\"300\" height=\"300\" src=\"https:\/\/re20.org\/wp-content\/uploads\/2019\/08\/profile_fabiano_dalpiaz-300x300.png\" class=\" wp-post-image\" alt=\"Fabiano Dalpiaz\" loading=\"lazy\" title=\"Fabiano Dalpiaz\" srcset=\"https:\/\/re20.org\/wp-content\/uploads\/2019\/08\/profile_fabiano_dalpiaz-300x300.png 300w, https:\/\/re20.org\/wp-content\/uploads\/2019\/08\/profile_fabiano_dalpiaz-150x150.png 150w, https:\/\/re20.org\/wp-content\/uploads\/2019\/08\/profile_fabiano_dalpiaz-90x90.png 90w\" sizes=\"(max-width: 300px) 100vw, 300px\" \/><\/a><\/figure><div class=\"profile_inner\"><div class=\"cmsmasters_profile_header_wrap\"><header class=\"cmsmasters_profile_header entry-header\"><h6 class=\"cmsmasters_profile_subtitle\">Artifact Evaluation<\/h6><h3 class=\"cmsmasters_profile_title entry-title\"><a href=\"https:\/\/re20.org\/index.php\/profile\/fabiano-dalpiaz\/\">Fabiano Dalpiaz<\/a><\/h3><\/header><div class=\"profile_social_icons\"><ul class=\"profile_social_icons_list\"><li><a href=\"mailto:f.dalpiaz@uu.nl\" class=\"cmsmasters_social_icon cmsmasters_social_icon_69eb68c82a168 cmsmasters-icon-mail-1\" title=\"\"><\/a><\/li><li><a href=\"http:\/\/www.staff.science.uu.nl\/~dalpi001\/\" class=\"cmsmasters_social_icon cmsmasters_social_icon_69eb68c82a1cc cmsmasters-icon-home-1\" title=\"\" target=\"_blank\"><\/a><\/li><\/ul><\/div><\/div><\/div>\t<\/div>\r\n<\/article>\r\n<!-- Finish Vertical Profile -->\r\n\r\n\r\n<!-- Start Vertical Profile -->\r\n\r\n<article id=\"post-13776\" class=\"cmsmasters_profile_vertical post-13776 profile type-profile status-publish has-post-thumbnail hentry pl-categs-artefacts pl-categs-organizing-committee\">\r\n\t<div class=\"profile_outer\">\r\n\t<figure class=\"cmsmasters_img_wrap\"><a href=\"https:\/\/re20.org\/index.php\/profile\/mona-rahimi\/\" title=\"Mona Rahimi\" class=\"cmsmasters_img_link\"><img width=\"200\" height=\"200\" src=\"https:\/\/re20.org\/wp-content\/uploads\/2019\/08\/profile_mona_rahimi_colored.jpg\" class=\" wp-post-image\" alt=\"Mona Rahimi\" loading=\"lazy\" title=\"Mona Rahimi\" srcset=\"https:\/\/re20.org\/wp-content\/uploads\/2019\/08\/profile_mona_rahimi_colored.jpg 200w, https:\/\/re20.org\/wp-content\/uploads\/2019\/08\/profile_mona_rahimi_colored-150x150.jpg 150w, https:\/\/re20.org\/wp-content\/uploads\/2019\/08\/profile_mona_rahimi_colored-90x90.jpg 90w\" sizes=\"(max-width: 200px) 100vw, 200px\" \/><\/a><\/figure><div class=\"profile_inner\"><div class=\"cmsmasters_profile_header_wrap\"><header class=\"cmsmasters_profile_header entry-header\"><h6 class=\"cmsmasters_profile_subtitle\">Artifact Evaluation<\/h6><h3 class=\"cmsmasters_profile_title entry-title\"><a href=\"https:\/\/re20.org\/index.php\/profile\/mona-rahimi\/\">Mona Rahimi<\/a><\/h3><\/header><div class=\"profile_social_icons\"><ul class=\"profile_social_icons_list\"><li><a href=\"mailto:%20mrahimi@cs.toronto.edu\" class=\"cmsmasters_social_icon cmsmasters_social_icon_69eb68c82b760 cmsmasters-icon-mail-1\" title=\"\"><\/a><\/li><li><a href=\"https:\/\/www.linkedin.com\/in\/mona-rahimi-89217b39\/\" class=\"cmsmasters_social_icon cmsmasters_social_icon_69eb68c82b7c1 cmsmasters-icon-home-1\" title=\"\" target=\"_blank\"><\/a><\/li><\/ul><\/div><\/div><\/div>\t<\/div>\r\n<\/article>\r\n<!-- Finish Vertical Profile -->\r\n\r\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n\n","protected":false},"excerpt":{"rendered":"","protected":false},"author":1,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":[],"_links":{"self":[{"href":"https:\/\/re20.org\/index.php\/wp-json\/wp\/v2\/pages\/13306"}],"collection":[{"href":"https:\/\/re20.org\/index.php\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/re20.org\/index.php\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/re20.org\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/re20.org\/index.php\/wp-json\/wp\/v2\/comments?post=13306"}],"version-history":[{"count":14,"href":"https:\/\/re20.org\/index.php\/wp-json\/wp\/v2\/pages\/13306\/revisions"}],"predecessor-version":[{"id":15199,"href":"https:\/\/re20.org\/index.php\/wp-json\/wp\/v2\/pages\/13306\/revisions\/15199"}],"wp:attachment":[{"href":"https:\/\/re20.org\/index.php\/wp-json\/wp\/v2\/media?parent=13306"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}