This article is about Automation and Artificial Intelligence in Software Engineering: Experiences, Challenges, and Opportunities. It represents a paper published at a highly ranked HAWAII INTERNATIONAL CONFERENCE ON SYSTEM SCIENCES (HICSS).

This paper was written with great help and supervision of prof. Viktoria Pammer-Schindler.

Given article and paper might be relevant for you as well.

  • Firstly, you might be the owner of a software development company thinking about improving your business processes.
  • Secondly, you might a researcher engaging in these topics and searching for relevant literature and references.
  • Maybe you are an investor thinking about engaging in this disruptive domain.
  • Finally, you might be a software developer curious about how your peers are doing development, what challenges, issues, and opportunities they have.

If any of these cases apply to you, you should read this article together with the paper which is presented here.

Table of Contents

Quick overview of Automation and Artificial Intelligence in Software Engineering

If you don’t have time to read through the whole article and paper, I made an up-to-point video presentation that gives you all the needed insights. Afterward, if you want more details you can read the official paper here.

Transformative influence of the Automation and Artificial Intelligence

What Is our research about?

Automation and artificial intelligence have a transformative influence on many sectors.

For example:

  • Firstly, increased automation and deployment of data analytics and Artificial Intelligence has been coined Industry 4.0 or smart manufacturing
  • Secondly, in medicine – machine-learning computer-aided detection and automated diagnosis in radiology
  • Finally, auto-encoder neural networks can help financial auditors identify unusual journal entries in an audit context

We understand automation as a set of computational tools executing domain activities, based initially on conditional logic.

For Artificial Intelligence, we understand a broad range of technologies and technological capabilities based on processing natural language, audio, images, and video; formal representations of knowledge and automated reasoning; and machine learning.

What are the important questions?

So, from previous examples, we can clearly recognize that Automation and Artificial Intelligence have a transformative influence on many sectors.

We recognize Software engineers as the actors who engineer this transformation.

However, there is little knowledge of how automation and Artificial Intelligence impact them, meaning software engineering practice. We approached this problem with two research questions, presented on this slide.

First one, aiming to understand the current practice. So, we are asking:

What are approaches for automation and Artificial Intelligence enabled tools and practices adopted by interviewed software engineers?

The second one, aiming to understand the attitudes, hopes, and concerns of software engineers.
Here, our question is:

What are approaches for automation and Artificial Intelligence enabled tools and practices adopted by interviewed software engineers? The second one, aiming to understand the attitudes, hopes, and concerns of software engineers.

What kind of research we did?

In the early stage of our research, we were thinking about which approach to take. For finding answers to our questions we decided to conduct qualitative research, based on semi-structured interviews.

We wanted an opportunity to listen and take notes first and then discuss.

We did this by means of screen share, typing quasi-verbatim notes and sharing them with participants, so we can discuss, validate and agree on those. 

Our questions started from common questions to get to know our participants better:

What is their job? What their company does?

Then, we wanted to know more about their environment. What did they automate in software engineering and in their work?

We were curious if they have automated something them selves. 

We also wanted to know who initiated automation in their company.

Finally, we asked if and how is Artificial Intelligence fitting in their activities.

Once we gathered data, next step was analysis.

Here, we did inductive and reflexive thematic analysis.

  • Inductive means: Reading through notes. Identifying a priori unknown patterns in the described experiences of our interviewees
  • Reflexive means: Describing and summarizing the experiences of interviewees. We developed themes during the data analysis. 

These themes, compared with the literature overview provided answers to our questions, but also triggered additional discussion topics.

Who we interviewed?

List of software practitioners we interviewed - Automation and Artificial Intelligence in Software Engineering

Related to interview participants’ profiles. We interviewed experienced software practitioners across frontend, backend development, DevOps, R&D, and integration.

Diversity in gender and age,  sector (industry or academia), industry domain (Software as a Service – SaaS, logistics, networking, software security, semiconductors, automotive, industrial automation, or computer hardware), professional position, and technology stack (ongoing experience with various technologies). 

We recruited participants online through the Linked-In network and in-person within corporate environments and conferences. 

Some participants recommended others for interview as well.

What were our results? What did we find out about Automation and Artificial Intelligence in Software Engineering?

commonly automated activities, tools, and challenges in software engineering

One of the first things we recognized, even during interviews, was enthusiasm toward automation.

Software practitioners have common activities they need to automate:

  • Environment configuration – for example, new developer setting up a machine (one script vs days of configuring and installation tutorials)
  • Static code checking – recognizing is code syntax is good, dead code, obvious bugs, bad coding practices, also known as a code smell. These things are closely related to code review best practices.
  • Especially, testing – running tests automatically when a new code is sent to the server.
  • Literature also states that Artificial Intelligence is helpful for writing technical specifications and estimates.

Developers use different tools and approaches to accomplish same automation goals.

Process bridging with micro-automation

Automation and Artificial Intelligence in Software Engineering in software development might appear in different processes.

Each process is set of steps that initially has to be done manually.

When developers have large number of manual steps, mistakes can happen and some steps can be forgotten.

Also, if steps are happening between different environments and systems, this can be time consuming.

This is why developers are writing micro automations to make their work easier.

Micro automation is usually implemented in form of custom scripts, automatic configurations, development helpers, testing optimizations, and so on.

In the given example developers want to make a backup on a specific server, and as a result, he needs backup hash.

He triggers script on server, which has automated backup procedure.

This procedure is connected with team chat where backup hash is delivered.

There are multiple benefits from this automation.

  • The developer does not have to do the backup manually.
  • Hash is saved in team chat where everyone can see and use it
  • Everyone is aware that backup activity is happening and there is a clear history of backup actions.

Automation of software engineering as a bottom-up driven side product of operative activity

One of our most interesting findings is that the automation of software engineering is bottom-up driven.

Meaning, our participants reported that top-level management was not the one who initiated automation.

Also, automation is usually side affect or bonus activity, but rarely the goal itself.

One of our candidates compared automation with defensive player in football.

Meaning, everyone knows that automation is important but there is no immediate cache returns out of it.

Most of our participants reported automation as quick solutions tailored to a specific context and need.

Automation and effects on work efficiency

Our participants also reported different effect of automation on their work efficiency.

What we need to understand here is that automation is usually a black box for person who uses it.

Automation does something for software developers but they want to be informed about it.

Still, not all automated tools are good at this.

We had reports of automated tools which were sending large number of notifications.

This required software developers to constantly get distracted, check or double-check some things they already know.

Sometimes, automation would send lots of data that is not useful or, at least, not structured in a useful way.

Our biggest takeaway from this is this.

Automation should communicate with software developers, but by providing good data with good context, if needed and when needed.

Unclear impact of Artificial Intelligence on the future of SW engineering practice

When it comes to Artificial Intelligence and impact on software engineering, results are unclear.

Our interview participants have not reported AI-enabled tools that are substantially changing their work.

When we asked them how they feel about AI as a helping tool or as a teammate, their responses were mixed.

Most of them were enthusiastic about having AI as a helping tool.

However, they did not see AI as an independent, final decision maker.

Let us make an example for this.

Having a smart coding assistant which recognizes issues and offers solution and advice is AI-helping tool.

Our participants would like to have tool like this.

AI-tool can also implement solution and deploy it to server, without consulting developer.

This is something our participants were not so enthusiastic about.

What do we make of these findings?

Bottom-up and Micro-Automation

Our research provided several interesting results. In the paper, we provided our interpretation of these results, also taking into consideration the existing literature review. This resulted in different discussion topics. 

For example: How are bottom-up automation and micro-automation linked? 

Is it possible that bottom up nature of automation leads to micro-automation.

This would make sense because is it not likely that a bottom-up perspective can result in a scalable solution that is implemented across a complete organization.

It simply might not have complete perception, motivation or required resources to do so.

On the other hand, we ask our selves what is causing automation to have this bottom-up nature?

Is the because of the nature of software engineering? Maybe because software engineers know how to automate they simply do it, but only within their area of work, without scaling it.

Finally, how to evaluate the socio-technical effect of automation? How to calculate the cost-benefit of automation?

We find these very interesting discussions with both theoretical and practical implications.

Future of AI-driven applications

Our interviewees were enthusiastic about AI-enhanced tools but skeptical about “AI being in a driver’s seat” and making decisions. Also, in literature research, we recognized different AI-driven applications.

For example, code error prediction, error localization to help developers write code, and assistance in effort estimation, to help project managers define efforts based on different algorithms.

We asked our participants what they think about these applications.

Answers were different, but the constant was enthusiasm toward AI-enhanced tools but skepticism toward AI-driven tools. This is something that requires careful interpretation.

We must take into considerations that our participants were NOT AI experts.

Still, maybe software engineers do not seriously engage in thinking about how is their profession affected by Artificial Intelligence?

On the other hand, maybe software engineering is moving toward the “skill diversity phase”. This would divide software developers between users and architects. Users would do programming supported by AI, while architects would create AI-enhanced and AI-driven tools.

These are all very interesting topics, where additional research should be conducted.