This paper discusses the advantages of formalisation and automation of software production and the challenges yet to master with current operation platforms.

Communication interfaces are particularly challenging to test using automatically generated test data. The test data sent through the interface must be ”valid enough” to overcome initial sanity checks of the interface and reach functions deep inside the integrated software. Machine-readable information about what data forms ”valid enough” messages is rarely available to test data generation tools. So instead, we evolve the messages with an evolutionary algorithm. This enables efficient fuzz testing for the communication interface between a satellite and its ground station. In this pa- per, using an algorithm implementation in our fuzzing tool DCRTT, we investigate the impact of algorithm parameter selection on the performance and the possi- bility of efficient general default parameter values. The preliminary results promise significant improvements to automated testing with respect to software security testing and quality assurance.

Manual requirements-based testing is time- consuming: Input data must cover the requirements and observed output data must be checked for their compatibility with the requirements. Testcases can also be automatically generated from test models. However, these models first have to be established manually. In contrast, the approach to be presented here uses simpler ways of formalizing requirements to automatically map test data generated for automatic robustness testing using massive stimulation to requirements and to check the results for correctness.

Manual requirements-based testing is time-consuming: Input data must cover the requirements and observed output data must be checked for their compatibility with the requirements. Testcases can also be automatically generated from test models. However, these models first have to be established manually. In contrast, the approach to be presented here uses simpler ways of formalizing requirements to automatically map test data generated for automatic robustness testing using massive stimulation to requirements and to check the results for correctness.

Generating useful test data is one of the big challenges in automatic software testing. While random test data generation is the easiest method, the test inputs generated by it may fail to exercise the software under test properly if the internal structure of the data is unknown to the generator and at the same time relevant for the decisions taken in the code. Handling of telecommands in space onboard software is one example where this is the case. We investigate a method of generating test data for these cases using genetic algorithms.

Within the small satellite mission TechnoSat of Technische Universität Berlin, a verification strategy based on Dynamic Analysis has been applied to the C++-operating system RODOS using automated massive stimulation of the software- under-test. This approach is aiming at evaluating the robustness of the software and to derive feedback on the implemented messaging scheme of the on-board process chain. For fault detection and recording of message exchange the code is automatically instrumented with application-independent indicators which shall flag anomalies. Manual fault analysis is limited to the reported issues highlighting fault potential in contrast to usual reviews on the full code. The suggested reviews were extended to similar code, an approach which turned out as being effective. For the verification of the messaging scheme observed functional and performance properties were evaluated. The verification strategy targets the reduction of costs of verification and risks. Within this paper, the different verification steps are described and examples for reported issues are given.

Automated software verification tools support devel- opers in detecting faults that may lead to runtime errors. A fault in critical software that slips into the field, e.g., into a spacecraft, may have fatal consequences. However, there is an enormous variety of free and commercial tools available. Suppliers and customers of software need to have a clear understanding what tools suit the needs and expectations in their domain. We selected six tools (Polyspace, QA C, Klocwork, and others) and applied them to real-world spacecraft software. We collected reports from all the tools and manually verified whether they were justified. In particular, we clocked the time needed to confirm or disprove each report. The result is a profile of true and false positive and negative reports for each tool. We investigate questions regarding effectiveness and efficiency of different tools and their combinations, what the best tool is, if it makes sense at all to apply automated software verification to well-tested software, and whether tools with many or few reports are preferable.

Untyped data such as the byte streams used in communications between spacecraft and ground stations present a specifically challenging field for au- tomatic test data generation. We investigate variations of genetic algorithms to improve test data generation, and present measurements and preliminary results ob- tained using our prototype. The future goal is to extend our white-box random testing tool DCRTT with these methods and thus apply the approach to industry-grade software.

In a previous study six software verification tools have been applied to a representative space software package. The findings reported by each tool have been compared in order to derive footprints regarding fault identification. In a continuation three more tools were applied to the previously selected application software and to another application together with two tools previously used in order to broaden the base of evaluation. More aspects were considered regarding the evaluation of results: an additional evaluation criterion was added and a comparison of reported defects with the outcome of unit tests was performed. Due to a higher degree of formalization and automation the manual evaluation effort could be decreased while extending the number of considered reports and the number of tools. The encountered evaluation and verification issues are discussed in detail. All results together shall provide a detailed view on the defect identification capabilities of the considered tools w.r.t. current software base. Altogether, the high quality of reports as obtained in the previous study was not obtained again: in context of a different set of tools and another (object-oriented) language a lot of trivial reports were observed.

Testing as a method of software verification is limited in that it can only prove the presence of defects, not their absence. To be useful, a large number of test cases may be needed, a strategy that is often in conflict with project constraints such as available time and funds. Test automation may be considered as an interesting approach to alleviating this conflict. However, test automation requires accurate and computer-accessible information about the system to be tested, both in terms of the interfaces by which the system is to be stimulated as well as the desired properties of these interfaces. Within the FASTII activity (FAST=Flow-optimised Automated Source-code based Testing) the possibility of deriving this information from available requirements and design documents is being investigated. Preliminary results of this investigation as well as suggestions for future changes in the process are presented in this paper.

We present results from the ESVW activity (Evaluierung von Software-Verifikationsmethoden und -Werkzeugen, Evaluation of Software Verification Method and Tools) BSSE and its subcontractor etamax space GmbH have performed on behalf of the German Space Center (DLR). In the course of this activity, six different verification tools were evaluated for their fault detection capabilities. The presentation provides an insight into the activity, the methods used, the results as well as general and specific lessons learned relevant for the use of software verification tools.

Six software verification tools have been applied to space flight software and the findings reported by each tool have been compared in order to derive footprints of the tools regarding capabilities of fault identification. Currently available results are provided in this paper: sensitivity and precision of individual tools and combinations of pairs of tools out of the set. A reader should bear in mind that the results as presented here depend on the spectrum of fault types as present in the reference software and on the configuration of tools towards real defects and fault types which are of interest for embedded systems and space flight software.

Mastering the continuously increasing amount of software requires identification of more efficient strategies for software verification. Currently, fault coverage is only indirectly addressed, e.g. by code coverage. The idea as presented in this paper is to get a better understanding of fault coverage by a systematic classification of software fault types, derivation of footprints of verification tools regarding coverage of such fault types, and recording of required effort. A number of issues regarding fault identification and classification are discussed in this context.

We present an example for application of Constraint Handling Rules to automated test data generation and model checking in verification of mission critical software for satellite control.

With the advent of languages such as C++ and Java in mission- and safety-critical space on-board software, new challenges for testing and specifically automated testing arise. In this paper we discuss some of these challenges, consequences and solutions based on an experiment in automated source- code-based testing for C++.

We present results of a case study on a test generation approach called Flow-optimized Automated Source-code-based unit Testing (FAST) which generates test stimuli from information available in the source code, in particular taken from the detailed software interfaces. This allows automation of a significant part of testing, ranging from the test stimuli generation to the generation of the test report. A huge number of stimuli can be generated exploring the behaviour of the software under test under nominal and non- nominal conditions. Symptoms like timeouts, unexpected termination, run-time exceptions, out-of-range conditions and missing coverage are applied for defect detection. The goal of this study was to evaluate the FAST process in context of a real spacecraft flight software application and to get a feedback on its scalability regarding larger applications, its sensitivity on detecting defects in the code, the achievable test coverage, its compliance with software standards and potential limitations. We also consider the impact of coding style on suitability for automated testing. The results confirm that the approach (1) provides acceptable code coverage results without requiring manual intervention for test preparation and execution, (2) raises the probability of activation of exotic fault conditions, (3) may provide hints on locations in the code where robustness needs to be verified, and (4) identifies defects not found before by static analysis and intensive testing

We give an overview over the principles of constraint- based test data generation, discuss its limitations and potentials and touch some of the domains which may be interesting to combine with constraint-based testing techniques. Automated generation of test data is an example where this technique can be applied and significantly increase the degree of automation, but it is not limited to. This paper is intended to give interested readers a quick entry into the methods and applications to allow a deeper understanding and an informed verdict about the actual capabilities and potential future directions.

Standards are used to describe and ensure the quality of products, services and processes throughout almost all branches of industry, including the field of software engineering. Contractors and suppliers are obligated by their customers and certification authorities to follow a certain set of standards during development. For example, a customer can easier actively participate in and control the contractor’s process when enforcing a standard process.

However, as with any requirement, a standard may also impede the contractor or supplier in assuring actual quality of the product in the sense of fitness for the purpose intended by the customer.

This is the case when a standard defines specific quality assurance activities requiring a considerable amount of effort while other more efficient but equivalent or even superior approaches are blocked. Then improvement of the ratio between cost and quality exceeding miniscule advances is heavily impeded.

While in some parts being too specific in defining the mechanisms of the enforced process, standards are sometimes too weak in defining the principles or goals on control of product quality.

Therefore this paper addresses the following issues: (1) Which conclusions can be drawn on the quality and efficiency of a standard? (2) If and how is it possible to improve or evolve a standard? (3) How well does a standard guide a user towards high quality of the end product?

One conclusion is that the analyzed standards do interfere with technological innovation, though the standards leave a lot of freedom for concretization and are understood as technology-independent.

Another conclusion is that standards are not only a matter of quality but also a matter of competitiveness of the industry depending on resulting costs and time-to- market. When the costs induced by a standard are not adequate to the achievable quality, industry encounters a significant disadvantage.

This paper describes an improved process for development of software for critical systems fully bridging the gap between a specification and an executable target version by an automaton. The process covers the broad class of distributed and/or fault-tolerant and/or real-time systems, and meets the needs of critical systems. It has its roots in the space domain.

We present a new method for automatic test data generation (ATDG) applying to semantically annotated control-flow graphs (CFGs), covering both ATDG based on source code and assembly or virtual machine code. The method supports a generic set of test coverage criteria, including all structural coverage criteria currently in use in industrial software test for safety critical software. Several known and new strategies are supported for avoiding infeasible paths, that is paths in the CFG for which no input exists leading to their execution. We describe the implementation of the method in CHRv and discuss difficulties and advantages of CHR in this context.

The importance of the quality of requirements for successful execution and completion of a project from a technical and contractual point of view is being recognized more and more. Many methods are targeted to improve the support for collecting requirements while still focusing on natural language. However, the ambiguities in the semantics of natural language are the biggest obstacles towards success. The approach presented in this paper focuses on the elements of a domain while keeping the expressiveness of natural names and terms and introducing clear semantics. This brings the advantage that immediate verification of the human-provided inputs is possible, immediate contributions to validation are available and inconsistencies can be detected by a tool immediately. This leads to guidance of an engineer by a tool towards consistent, complete and correct requirements - requirements of high quality - and eases maintenance for the same reasons. As most of the complexity is handled by the tool due to its good knowledge on the domain, the approach is scalable towards large specifications. Several examples of application domains are described which illustrate the universality and feasibility of the approach across domain boundaries.

Presentation on behalf of the Working Group of Space-SME (AKRK) of the position of German SME regarding the Open Source Software Strategy of the European Space Agency.

Presentation of questions, risks and advantages of the Open Source Software Strategy of the European Space Agency.

Generalised framework for constraintbased test-data generation using program flow graphs.

Various strategies for fault identification exist - e.g. based on formal analysis of code or on testing - of which each focuses on certain identification aspects and fault types. This paper characterises the strengths and weaknesses of methods – in theory and practice - focusing on application-independent identification strategies, and it suggests strategies to maximise the number of detected faults while minimising the related effort. Fault activation conditions are discussed in detail, resulting in an extended scope on stimulation needs. In particular, the contribution of automation in raising the activation probabilities is investigated. Various examples of fault activation mechanisms and statistics on fault types vs. identification methods are provided as observed in practice. An interesting result is the identification of application-dependent test cases by application-independent test strategies.

Certification is based on compliance of the code of the code generator with given standards. Such compliance never can guarantee correctness of the whole chain through transformation down to the environment for execution, though the belief is that certification implies well-formed code at a reduced fault rate.

The approach presented here goes a direction different from manual certification.. It is guided by the idea of automated proof: each time code is generated from a model the properties of the code when being executed in its environment are compared with the properties specified in the model. This allows to conclude on the correctness of the whole chain for every application and related generated code.

A new approach for constructing path constraints in constraintbased test-data generation using results from analysis of control-flow graphs.

We present an approach of fully-automated software test, using application-independent criteria for fault identification.

The intention of this paper is to highlight the benefits of model exchange between different tools, methods and notations on one side, and to identify issues of proper modelling on the other side which have been detected during model transformation and code generation from models.

As the test effort takes a significant part of the software development lifecycle, efficient test strategies are a precondition for reduction of development costs and time. In this respect two main issues exist: firstly, the tuning of the test track from test case identification to evaluation, secondly, the reduction of number of test cases to be processed and evaluated. Both aspects were considered in the work presented in this paper.

This paper was presented at the DASIA 2006 conference in Berlin, Germany. It lays out principal approaches for tackling the "small target" and "oracle" problems in statistical automatic testing by applying deterministic methods for assistance.

This paper was presented at the DASIA 2005 conference in Edinburgh, Scotland. It shows the result of an Automated ISVV (AISVV) activity executed on the Flight Application Software (FAS) of the Autonomous Transfer Vehicle (ATV).

This paper was presented at the DASIA 2004 conference in Nice, France. The answer to this hypothetic question is "yes", of course. The paper will approach the problem in two steps: firstly, we will discuss if and which measures exist to identify an overrun early enough, secondly, we will analyse the sources of overruns and which means may be applied not to exceed the planned budget.

This is the presentation as given during the Colloquium: Technology Exchange between Space and Automotive Industry in Darmstadt, Germany (Nov 6th, 2002).

This is the more practice-oriented presentation as given in Munich (Oct 15, 2002).

If you would like to get an example file package as shown during the presentation please contact us.

This is the strategy-oriented presentation as given during the MicroConsult Praxis Forum "Embedded Quality - Neue Dimension der QualitÀtssicherung" in Zurich (Oct 10th, 2002).

This paper was presented at the DASIA 2002 conference in Dublin, Ireland. It shows results of benchmarks on projects carried out using ISG/ASaP.

The ISG/ASaP approach allows automated system generation from engineering information and to immediately build an executable system from engineering level.

This approach has already been successfully applied to provide the infrastructure and major parts of the application software for the Material Science Laboratory aboard the International Space Station(ISS).

This paper describes the potential of ISG/ASaP for cost-savings and quality improvements.

This paper describes the benefits of ISG/ASaP in the context of embedded systems.

The impact on the improvement of software quality by ISG/ASaP is discussed by this paper.

This paper describes the challenges of maintaining software on different platforms.

The presentation covered application of automation in the area of software engineering and included a demo of generation of a distributed realtime system with 16 processes within ca. 40 minutes.

This paper discusses the potential conflict between performance and robustness issues of software development.

A comparison of the conventional and the ISG life-cycle and evaluation of cost savings by ISG.

This presentation summarizes the results as obtained by ISG/ASaP in the course of the ESPRIT project CRISYS (EP 25514). The goal was to evaluate the robustness in presence of time-jitter and fault-injection for a critical distributed control system.

In this paper BSSE's motivation behind ISG/ASaP is presented and its relevance to the Capability Maturity Model(CMM) defined by Carnegie-Mellon Software Engineering Institute. It is shown that by use of Automated Software Production the upper levels of CMM are reached immediately.

Verification and validation of distributed systems as defined by the ISG-approach is described in this paper.

This paper describes how ISG was used to built MSL software.

This paper was presented during the DASIA 2000 conference in Montreal, Canada by Kayser-Threde.

It describes the organisation of the database from which MSL software was automatically generated by ISG/ASaP.

This is the final report of the ESA study on behavioural validation of MSL.

In the presentation application of incremental development and validation in the area of complex distributed systems was explained.

Description of an approach for incremental software development and validation applying automated generation of an executable system from a minimum of system engineering information.

This paper gives advice how to master state explosion.

This paper was presented during the ESA'98 4th conference on simulators, November 3th-5th 1998 in Noordwijk. It shows BSSE's experience with tool integration.

This paper was presented during the DASIA'98 conference, May 25th-28th 1998 in Athens. It shows BSSE's experience with the use of (Commercial-)Off-The-Shelf-Software based on a project dealing with the integration of a number of (C)OTS packages and software developed from scratch.

This paper was presented during the workshop on Performance and Time in SDL and MSC, at the University of Erlangen, February 17th - 19th 1998.

This paper was presented during the workshop on Performance and Time in SDL and MSC, at the University of Erlangen, February 17th - 19th 1998.

This paper was published by ESANEWS, December 1997.

This paper was published by Elsevier Science Publishers B.V., ISBN: 0-444-82816-8

The presentation included an online demonstration on a Sparc Workstation and a PC laptop.

The presentation included a tool demonstration.

Describes a generic architecture and decomposition scheme.

This paper was presented during the DASIA'96 conference, May 20th - 23th 1996 in Rome.

This paper was presented during the Systems Engineering Workshop, November 28th - 30th 1995 at ESTEC, Noordwijk.

This paper was presented during the International Symposium on On-Board Real-Time Software, November 13th - 15th 1995 at ESTEC, Noordwijk.

This paper was presented during the Ada in Europe'95 conference October 02nd - 06th 1995 in Frankfurt/Main.

This paper was presented during the "CASE Anwendertag", September 19th 1995 at the DLR in Göttlingen.

This paper was presented during the 3rd ESTEC Workshop on Simulators for European Space Programmes, November 15th - 17th 1994 at ESTEC, Noordwijk.

The presentation included a demo on the PC platform.

The presentation included a demo on the PC platform.

Data base concept for real-time access of databases in continuos operation under fail-operational conditions.

Time consumption of typical mathematical functions and operations on 6502

Fast algorithms for stable, recursive computation of sin, cos, exp, sinh, cosh for 8bit-processors, CNC application)