Development and Testing at Scale

Research questions:

• Are tests "just tests" or are they SW products themselves?
• Should test systems be architected, designed and developed or … just created?
• How to organize reviews, how to document and maintain long term test projects?
• What has the biggest impact on the quality of large scale test systems?
• How do test systems grow?
• How are test systems scaling?
• How to scale test project life-cycle (e.g. agile?)
• How "good" are standard test suits?
• How to select test cases when various target parameters are given (coverage, running time, dependencies, etc.)?
• What are the technical debts of several standardized test suits?
• What are the effects on the quality of test systems during their evolution?

Results:

• We researched and developed automated code quality checks for standardized and industrial test systems (written in TTCN-3).
• We released solutions as part of our partner’s TTCN-3 toolset called Titan.
• We researched automated “whole-project” code quality improving refactoring solutions. We look at tests as software products that have their own quality needs, which need to be built and maintained.
• We analyzed the evolution of test systems in order to understand what has the biggest impact on their quality and derive "best practices".
• We analyzed the growth and scaling of test systems in order to understand and remove limitations of scalability.

• We are working together with our industry partner to provide participating students with up-to-date and relevant programming challenges and knowledge.
  Our developed code is part of the Titan toolset, used in several standardization bodies of contemporary large scale systems and in the industry supporting the newest technologies.
• As Titan is being a modern IDE, participating students can get an understanding of how to develop environments internally (useful in all areas of their career afterwards).
• As Titan is open source, our work is available in open source.
• Students entering the project usually start with a small and easy task, which may grow to a Diploma thesis, scientific articles or even a PhD thesis.

• Description

Bibliography:

• I. Farkas, K. Szabados, A. Kovács, Improving productivity in large scale testing at the compiler level by changing the intermediate language from C++ to Java, submitted. An example containing a "Hello World", some simple types in TTCN-3, and the compiled C/C++ and Java codes, Measurement data with configuration 1, Measurement data with configuration 2, Regression test data, Short film
• K. Szabados, Quality Aspects of TTCN-3 Based Test Systems, PhD thesis, ELTE, PDF
• K. Szabados, Quality Aspects of TTCN-3 Based Test Systems, tézisfüzet (in hungarian), ELTE, PDF
• Bianka Flóra Békefi, Quality Analysis of Test System Source Code, MSc thesis, ELTE IK, PDF
• A. Zsiga, Productivity trends and patterns in software development projects, MSc thesis, ELTE IK, (in hungarian) PDF with data PDF
• A. Kovács, K. Szabados, Internal quality evolution of a large test system - an empirical study, Acta Univ. Sapientiae, Informatica , 8/2, 216-240, 2016. PDF
• K. Szabados, A.Kovács, G. Jenei, G Góbor, Titanium: Visualization of TTCN-3 system architecture, IEEE International Conference on Automation, Quality and Testing, Robotics (AQTR), Cluj-Napoca, Romania, DOI: 10.1109/AQTR.2016.7501275  PDF
• N. Bartha: Scalability on IT projects, MSc thesis, 2016. PDF
• A. Kovács, K. Szabados, Technical Debt of Standardized Test Software, IEEE 7th International Workshop on Managing Technical Debt (MTD), Bremen, Germany, 57-60, 2015, ISBN: 978-1-4673-7378-4/15, PDF
• P. Oláh: Improving the semantic analysis in TTCN-3 environment, (in Hungarian), MSc thesis, 2016. PDF
• A. Bánsághi, A. Kovács, Trend Analysis Based On Semantic Graphs - A Case Study, Annales Univ. Sci. Budapest, Sect. Comp., 42, 55-68, 2014, PDF
• A. Kovács, K. Szabados, Advanced TTCN-3 Test Suite validation with Titan, Proceedings of the 9th International Conference on Applied Informatics, Vol. 2. 273-281, 2014, PDF, doi: 10.14794/ICAI.9.2014.2.273, Data for TestSuite Validation
• A. Kovács, K. Szabados, Test software quality issues and connections to international standards, Acta Univ. Sapientiae , 5/1, 77-102, 2013,  PDF
• A. Bánsághi, B.G. Ézsiás, A. Kovács, A. Tátrai, Source Code Scanners in Software Quality Management and Connections to International Standards, Annales Univ. Sci. Budapest, Sect. Comp., 37, 81-92, 2012, PDF
•  R. Gecse, A. Kovács, Consistency of stochastic context free grammars, Mathematical and Computer Modelling, , 52/3-4, 490-500, 2010,  PDF
• K. Szabados, Structural Analysis of Large TTCN-3 Projects, Proc. TESTCOM '09/FATES '09 Proceedings of the 21st IFIP WG 6.1 International Conference on Testing of Software and Communication Systems and 9th International FATES Workshop, 241-246, ISBN: 978-3-642-05030-5 doi: 10.1007/978-3-642-05031-2_19