• Giovanni Ciatto
    Giovanni Ciatto, 31/03/2021 15:17

     REVIEW 1 -
    SUBMISSION: 4
    TITLE: Shallow2Deep: Restraining Neural Networks Opacity through Neural Architecture Search
    AUTHORS: Andrea Agiollo, Giovanni Ciatto and Andrea Omicini


     Overall evaluation -
    SCORE: 2 (accept)


     TEXT:
    the paper presents a method for evolutionary neural architecture search, aiming at restraining opacity of the resulting network.

    The authors should better argue why lower complexity necessarily implies lower opacity. Having 50 or 100 M nodes doesn't necessarily make the network less opaque. The results presented in the paper are also mostly on complexity -  and not opacity as such. If the connection between opacity and complexity cannot be made clear, it is doubtful if the topic and title of the paper is correct. 

    IF the paper is mainly about reduction of complexity, I believe there is considerable relevant earlier work in network pruning. The relation to this kind of work should be discussed in the background section. 

    Table 1: The M value probably represents some normalised count of parameters in the network. Please explain how it is computed.

    Also explain what the st.dev refers to. Multiple runs with the same data?  The small st.dev values make me believe that it is not related to  cross-validation (it would be very interesting to see how the accuracy/st.dev looks like if you cores-validate data).


     REVIEW 2 -
    SUBMISSION: 4
    TITLE: Shallow2Deep: Restraining Neural Networks Opacity through Neural Architecture Search
    AUTHORS: Andrea Agiollo, Giovanni Ciatto and Andrea Omicini


     Overall evaluation -
    SCORE: 2 (accept)


     TEXT:
    This paper provides a very large number of references and the paragraphs cover all the references, that the authors used to compare their approach with the others NAS approaches. The paper proposed a first approach to design an automatic tool to produce efficient NN architectures that differs from other NAS approaches in a number of ways. The new approach called Shallow2Deep aimes at keeping NN structural complexity under control by controlling NN opacity, it shows that Shallow2Deep approach can effectively achieve NN complexity reduction, while reaching performances comparable to the state-of-the-art. The authors presented an experimental results that show how  variability over local structures is actually desirable fearure to obtain well performing models.
    This paper is well written and easy to read, the paragraphs are well chosen as well as the consistency between the paragraphs, the demonstration steps are clear to highlight the new concepts adopted for the new approach

    However, here some comments and points to be improved:

     How experts’ experience and intuition in manual engineering process performed by data scientists could be represented in automatic tool such the Neural Architecture Search approach ?
     The authors mention that several approaches are being explored, modelling the NAS as a search problem in the space of all possible NN architectures and their work focuses on controlling network structural complexity, how that could be argued ?
     The authors said that the State-of-the-art NAS approaches mostly differ in which and how many blocks and cells are exploited, how these can be connected with each other’s, now, how you they could justify that their new approach based on selecting the best internal structure could reduce the complexity rather than the others approaches ?
     Shallow2Deep – aimed at keeping NN structural complexity under control,it is not clear and justified
     NAS is it an algorithm or an approach ? Sometimes, the authors mention it as approach and sometimes as algorithm
     DAG has to be extended, what is it ?
     The authors confirm several times that their work represents the first attempt to produce a NAS strategy capable of identifying both a good cell structure and a global architecture, avoiding constraints on architecture design. Compared to others NAS approaches, are the others approaches of the background section based on CNN model or all types of NN since your approach has limited to CNN ?
     You argue that the best elementary structure for shallow and deep layers of NN are not architecturally equal, it would be useful to give more explanations about this reasoning
     to tackle local search you used evolutionary algorithm which is a local search algorithm employed to automate the selection of the actual cells structures, it would be more clear if the authors explain the evolutionary algorithm and its concepts in order to complete the understanding of the new approach


     REVIEW 3 -
    SUBMISSION: 4
    TITLE: Shallow2Deep: Restraining Neural Networks Opacity through Neural Architecture Search
    AUTHORS: Andrea Agiollo, Giovanni Ciatto and Andrea Omicini


     Overall evaluation -
    SCORE: 1 (weak accept)


     TEXT:
    The paper presents a preliminary study on Neural Architecture Search (NAS). The authors propose Shallow2Deep as an evolutionary NAS algorithm exploiting local variability to restrain DL-systems' opacity through NN architectures simplification.

    #FORM:
    - The paper is well written. However, several sentences are quite long and sometimes intertwined. The authors should simplify the presentation for the sake of readability and limit the parenthetical element  (  xxx ) only when really necessary.
    - In Section 1 and Section 2, there is a lack of references. Please check carefully  
    - Again, very long sentences...
    - IMHO several digressions can be avoided, prizing focus and readability.

    #CONTENT:
    - Opacity is in the title, yet it is not addressed.
    - The most concerning point is the lack of connection with any interpretability or explainability. It is necessary that the author clarify, reinforce and extend the sole statement "Shallow2Deep effectively reduces NN complexity – therefore their opacity". As is, it is not enough.
    - What do the authors mean by "a good cell structure" (page 4, ... ) please clarify this statement/concept
    - I understand the mechanism proposed by the authors. Nevertheless, to the best of my knowledge, I cannot grasp the underlying reasons supporting their choices (please add them).
    - the (meta)parameter $v^th$ should be better introduced
    - it would be interesting if the authors can briefly elaborate on the possible connections between opacity and complexity.

    Overall, besides the study's embryotic stage, the paper undertakes an interesting direction that can play a relevant role in simplifying NAS to facilitate interpretations. Yet, such a concept needs to be better (or elaborated at all).

Partita IVA: 01131710376 - Copyright © 2008-2021 APICe@DISI Research Group - PRIVACY