In April 2018, Berlin held the 4th Workshop on Challenges in Performance Methods for Software Development (WOSP-C) co-located with the 9th ACM/SPEC International Conference on Performance Engineering (ICPE 2018).

This fourth edition of WOSP-C, explored the performance implications of the evolution in architecture and development of cloud-based applications, and their impact on the inclusion and development of performance. As implied by the title, the workshop focused on methods usable anywhere across the life cycle, from requirements to design, testing and evolution of products; and an important number of sessions were devoted to discussing on the future of the field

One example of such a session was the Challenges in automating performance tool support panel, moderated by Connie U. Smith, and where the MegaM@Rt2 project had an important representation since 2 out of the 5 panelists are researches of this project. In this session we discussed about how research and development (R&D) of new tools for performance analysis faces many challenges from immaturity and lack of documentation of supporting tools and infrastructure, incompatibility of tools, lack of access to realistic case studies and performance parameters for them, validation of results, time required versus benefit of results, subsequent maintenance, and many, many others. Yet tool development is an essential part of practical R&D. Vittorio Cortellessa (MegaM@Rt2 member from Università degli Studi dell’Aquila), Abel Gómez (MegaM@Rt2 member from Universitat Oberta de Catalunya), Samuel Kounev, Catalina M. Lladó, and Murray Woodside shared their experiences in developing tools, discussed what needs improvement, and identified opportunities in developing R&D tools. As a summary,  find below a few quick morsels from the panel discussion.

We identified the following challenges in performance analysis:

  • There’s a high interest in performance but it’s all short-term (Murray Woodside).
  • Tools we develop are typically not too usable (Murray Woodside).
  • There should be a tighter connection of modeling to code and testing (Murray Woodside).
  • Academia usually relies on student projects to create tools, but often don’t match well with industrial work (Murray Woodside, Catalina Lladó).
  • Education is typically weak in subjects such as mathematics/probability/performance (Vittorio Cortellesa).
  • There’s a continuous turnaround of students (Vittorio Cortellesa).
  • Languages/notations and of software systems evolve continuously (Vittorio Cortellesa).
  • Standardisation and maturity is really important for industry in order to avoid vendor lock-in (Industrial audience).
  • DSLs are easy to learn and manage, but require expertise to be developed and are not interoperable per se (Abel Gómez).
  • Current general purpose modeling languages and standards (UML/MARTE) are too extensive, thus organisations create their own “idioms” (Abel Gómez).
  • Standards are not specially designed to be machine-understandable (ambiguities, poor tool support) (Abel Gómez).
  • Modeling frameworks are problematic when the layout has semantics (e.g., UML Sequence Diagrams) (Abel Gómez).

These challenges, however, offer us the following opportunities:

  • There’s a lot of room for improvement in the presentation of results in order to improve usability (Murray Woodside).
  • Advanced MDE techniques and tools are needed (megamodelling, metamodel evolution) (Vittorio Cortellesa).
  • MDE techniques can help in the development and evolution of Domain Specific Languages using incremental prototyping (Abel Gómez).
  • MDE enables the use of different analysis formalisms (Abel Gómez).
  • Eclipse and its MDE ecosystem are an excellent platform for model/tool sharing (Vittorio Cortellesa, Abel Gómez).
  • Opportunity to use/create sharing/pivot languages and formats (e.g., PMIF) (Vittorio Cortellesa).
  • Declarative performance engineering (Samuel Kounev).
  • Automation of model extraction / learning (Model extraction-as-a-service) (Samuel Kounev).
  • Tool standardisation (SPEC RG) (Samuel Kounev).

Indeed, some of theses challenges and opportunities have been already identified as points to be addressed by the MegaM@Rt2 project, so stay tuned! 😉


Share This