Viability Review Process for ACM Publications

Updated April 2018

 

An important role of the ACM Publications Board is to monitor the performance of ACM journals and the Editor-in-Chief (EiC) and magazines to ensure that they are contributing to the goals of the communities they serve and that their operations are consistent with ACM standards. As part of this monitoring, the Board has instituted a periodic assessment of journals and EiCs to be carried out by the Assessment and Search Committee of the ACM Publications Board.

The overarching goals for ACM EiCs are to produce high quality content consistent with the editorial charter of the publication, and to do so in a manner consistent with Roles and Responsibilities in ACM Publishing. Expectations for an EiC are that he/she (1) faithfully executes the editorial charter of the publication, (2) recruits a strong editorial board and high quality reviewing talent, (3) ensures that the journal is well run, in the sense that reviews are timely and issues are published on-time, and (4) ensures the authors and referees have a satisfactory experience in their interactions with the publication.

The expectations of a journal are that the journal: (1) serves an intellectually vibrant and viable community; (2) is a leading journal and preferably *the* leading journal in its field; and (3) incorporates innovations are a continuing part of updating and improving the journal. Please see the Publication Board policies titled Evaluation Criteria for Editors-in-Chief and Editor-in-Chief Appointment Policy for more details.

The Committee undertakes two types of assessments: 

1. An in-depth assessment of a journal and its viability, to be carried out prior to appointing a new EiC (normally once every six years);

2. A summary assessment of the EiC and journal prior to possible reappointment of an EiC to a second term (normally three years after initial appointment) which is more weighted to assessing the EiC than the journal itself).

The A & S Committee will handle both type of assessment above. Both assessments are holistic: the A & S committee will use the statistical data collected, the descriptive reports, the feedback, as well as any other information they deem necessary, to evaluate the journal (Assessment A) or journal and EiC (Assessment B). The A & S committee will formulate a recommendation to the Publications Board for a final decision. The Publications Board may wish to include any assessment outcome in the annual written report to the ACM Council.

1. In-depth Journal Assessment Procedure and Criteria

2. Assessment A is triggered by the need to find a new EiC (most often towards the completion of a second three-year term). This assessment and subsequent search must be initiated approximately one year in advance of when the EiC's term is over. At this time, the A & S Committee will assess the journal based on the following procedure and criteria:

   1. Evaluate statistical and other data provided by ACM HQ to determine the overall health of a 
      journal (since its last assessment), such as:

      1. its financial condition (based on subscriber numbers, expense and revenue);

      2. its impact (as measured by downloads and citations in the DL) compared to other ACM journals;

      3. its management as measured by:

         • acceptance ratio;

         • average time from submission to first AE assignment;

         • average time from submission to final decision;

         • extreme outlier times and frequency in the peer review process;

         • an overview of any competing publications in the field, supplemented by ISI or other comparative rankings;

         • pages allocated vs. pages used; and

         • backlog.

      4. strength of the pipeline (number of submissions year-to-year); and

      5. topical coverage as given by Computing Classification System index terms and author-supplied keywords for the papers published during the tenure of the out-going EIC.

   2. Solicit and evaluate a report (template A.2) from the outgoing EiC which includes:

      1. his/her assessment of the overall health of the journal,

      2. a comparison of actual coverage during EIC's tenure with the journal's editorial charter

      3. a detailed description of the specific review process used by the journal, and

      4. the status of any new initiatives that were pioneered by the EiC.

   3. Solicit and evaluate feedback (A.3 template) from the Editorial Board of the journal, all SIGs, and all journal EiCs regarding:

      1. their perception of the journal and the EiC Report,

      2. the strength of the editorial board, and

      3. the perceived ranking of the journal for publishing and as a venue for reading about the topics covered by the journal

   4. Possible In-depth Journal Assessment Outcomes
      The assessment could produce any of the following recommendations:

      1. The journal is viable and should continue as it has been.

      2. The journal has specific problems that must be addressed but the journal should continue

      3. The journal should be placed under probation with explicit actions to be taken.

      4. The scope or charter of the publication should be changed.

      5. The journal does not meet ACM standards and should be terminated.

      6. The journal should be merged with another.

3. Summary Journal and EiC Assessment Procedure
   A summary review and EiC Assessment occurs when an EiC expresses interest in a second-term reappointment. It should begin one year in advance of EiC term expiration with the A & S committee asking the EiC if there is interest in continuing into a second term. In the case that an EiC may not be interested in seeking a second term, the A & S Committee must undertake the Assessment A: In-depth Journal Assessment and Procedure, described above.

   1. Evaluate a summary overview provided by ACM HQ of the same statistical data used in the In-depth Journal Assessment Procedure above, and a report by the staff journals manager on the performance of the EIC, Editorial Assistant, and Information Director. The main purpose of this report is to flag significant problems which may affect the outcome of the assessment (See C below).

   2. Solicit and evaluate a status report (template B.2) from the EiC seeking a second term, which includes:

      1. strengths and weaknesses of the journal, and plans for addressing the weaknesses,

      2. innovations in generating content, such reaching out to new constituencies via special issues, and in improving operations,

      3. list of new Associate Editors named during first term and their research areas

      4. comparison of actual coverage during EIC's tenure with the journal's editorial charter,

      5. an overview of any competing publications in the field, and

      6. a detailed description of the specific review process used by the journal, including the number of reviews received before decisions are made.

   3. Solicit and evaluate feedback (template B.3) from the editorial board of the journal, all SIG chairs and other EICs regarding:

      1. their perception of the journal,

      2. the strength of the Editorial Board,

      3. the perceived ranking of the journal for publishing and as a venue for reading about the topics covered by the journal, and

      4. the EiC reappointment.

4. If the EiC would like to continue for a second term, approximately six months prior to EiC first-term expiration, the A&S Committee must begin a summary assessment of the journal. The A & S Committee will assess the journal and EiC based on the following procedure and criteria:

5. Possible Summary Journal and EiC Assessment Outcomes

   1. The journal is viable, and the EiC should be reappointed to a second term.

   2. The journal has specific problems that must be addressed, but the EiC has performed well and should be reappointed to a second term with a plan to fix the problems.

   3. The journal is viable but there is a problem with the EIC, and reappointment should be denied and a new EIC should be found.

   4. The journal has specific problems that must be addressed, but the EiC is not the appropriate person to address them and a new EiC should be found.

   5. The journal is not meeting ACM standards and should be merged or cancelled.

Template: A.2. Assessment of Journal (to be filled out by the Outgoing EiC)

Name of Journal: 
Name of Outgoing EiC:
We thank you for all of your years of service as EiC. Before we appoint a new EiC, we are seeking your help for assessing the journal. Your help is much appreciated.

1. What is your assessment of the overall health of the journal?
2. How would you compare the actual coverage of topics during your term with the journal's editorial charter?
3. What is the specific review process that you used (please be specific)?
4. What is the status of new initiatives that you pioneered for the journal?
5. Is there anything else that you can say to help us in the assessment of the journal?

Template: A.3. Assessment of Journal (to be filled out by Editorial Board of Journal, Sig Chairs and EiCs of other ACM journals)

Name of Journal:
Name of Respondent:
Are you:

1. A member of editorial board of this journal?

2. A SIG chair?
         1. If yes, what SIG?

3. An EiC of another journal?
         2. If yes, what journal?

The term of EiC of <name of journal> is terminating and we need to appoint a new EiC. The outgoing EiC has provided a status report, which is attached. We also attach the names of the members of the Journal's Editorial Board. Before we appoint a new EiC, we are assessing the viability of the journal, which is the standard practice of the Assessment and Search Committee of the ACM Publications board. We are asking for your help in assessing the journal by providing input to the following questions. We, as well as the ACM Community, greatly appreciate your help.

1. What are your perception of the journal and the content of the EiC's report?
2. What do you think about the strength of the editorial board?
3. What is your perceived ranking of the journal for publishing and as a venue for reading about the topics covered by the journal?
4. Is there anything else you would like to say to help us with the assessment?

Template B.2. A summary assessment of the EiC and journal prior to possible reappointment of an EiC to be filled out by the current EiC.

Name of Journal:
Name of EiC:
First, we want to thank you for all of your efforts as EiC of <name of Journal> and are pleased that you would like to continue. It is the standard practice of the ACM Publications Board to carry out an assessment of the journal and EiC before reappointment. Please help us with the assessment by answering the following questions.

1. What are the strengths and weaknesses of the journal, and plans for addressing the weaknesses?
2. What are the innovations you have pioneered in generating content, such reaching out to new constituencies via special issues, and in improving operations?
3. Who are the new Associate Editors you solicited during your first term and their research areas
4. What is the comparison of actual coverage during your tenure with the journal's editorial charter?
5. Who are the competing publications in the area of the journal?
6. What review process did you use? Please provide details.

Template B3: Assessment of Journal and EiC (to be filled out by Editorial Board of Journal, SIG Chairs and EiCs of other ACM journals)

Name of Journal:
Name of Respondent:
Are you:

1. A member of editorial board of this journal?

2. A SIG chair?
   If yes, what Sig?

3. An EiC of another journal?
   If yes, what journal?

The EiC of <name of journal> is has expressed a desire to continue for another term as EiC. Before a continuing EiC is reappointed, it is standard procedure of the ACM Publications Board to carry out an assessment of the journal and EiC's performance. The EiC has provided a status report, which is attached. We also attach the names of the members of the Journal's Editorial Board. We are asking for your help in assessing the journal by providing input to the following questions. We, as well as the ACM Community, greatly appreciate your help.

1. What are your perception of the journal and the content of the EiC's report?
2. What do you think about the strength of the editorial board?
3. What is your perceived ranking of the journal for publishing and as a venue for reading about the topics covered by the journal?
4. Do you have any opinion on whether the EiC should be reappointed?
5. Is there anything else you would like to say to help us with the assessment?