My team at Raet is a bit different then other teams. We use a different programming language, we’re on a different operating system. We do our own maintenance, deployments, releases and monitoring. This brings extra work, but also has many benefits. We rarely have issues and are able to roll out new releases and patches in no time. We are ‘in control’. We are only a small step away from continuous delivery and have automated many parts of the process, that other teams are still struggling with manually on a daily basis. I’m proud of what my team accomplished so far, and I think in general people are happy with their work. We get asked often how we do this, so, inspired by the Spotify Engineering Culture movies (part 1 and part 2), I thought I’d try to write down my thoughts about the culture in our team.
Like the Spotify team, we haven’t figured it all out. We are definitely not perfect and we have a lot to learn, but we’re trying to move forward all the time. One of the most important factors for running smoothly, is the fact that everybody feels responsible. Responsible if something goes wrong, and responsible for successes. This does not mean that if something breaks, 7 people will jump at it to fix it, but that if someone notices something that’s wrong, he or she will never leave it thinking ‘not my job’.
In order to take responsibility, you need control over the process and environment and that’s what we have. It means, when there’s an issue, we don’t need to run to the manager to get permission before we can do anything. Also we don’t need to go to an Operations department to beg for some of their busy busy highly valued time, and please get access to the server to be able to analyze the issue. Yes, currently we still need to ask our manager for permission before doing an actual patch, to discuss the risk of the fix and decide about extra communication to the customer, but fortunately this rarely slows us down.
But it’s not just about the patches. We are able to do any change to our system. When we see opportunities to improve the process or the product, we discuss this within the team and then we just do it. Especially when there’s no dependencies with other teams, we can act very quickly. Now you might think: isn’t it a big risk if the team can access everything on it’s own without ‘the all-knowing mighty manager who never makes mistakes and always knows what’s best to do’? In theory yes, but we have one rule: “Making mistakes is allowed, but not learning from them is not”. Also, not everyone in the team has the same permissions and not everyone can do everything. This has been enough to create a very stable environment for the past few years.
Having fun is another important aspect of our culture. This can be achieved by doing fun things, like having lunch together, but also by doing fun projects. There should be a balance between doing responsible (and sometimes boring) stuff, and fun stuff. We don’t have a 20% rule at Raet (yet), but that does not mean that there’s no room for good initiatives. In my team I try to allow people to work on innovative ideas when they come up with something. Fun projects can definitely be useful. As long as the ultimate goal is to improve the customer intimacy. Fun is required for innovation. Unhappy people are not expected to come up with great ideas. As long as people keep coming up with great ideas to improve, I know we’re still taking that extra step to stay ahead of the competition. That way we keep the end user happy, and also ourselves.
To make sure everybody is happy and stays happy, we do frequent retrospectives. We use the MSGL (mad/sad/glad/learned) board to discuss the past few weeks. When during the sprint someone is mad, sad or glad about something and wants to discuss it during the next retrospective, he puts it on the board. Also when someone learned something that he wants to share, he adds it to the board. This way you get it out of your head and you can continue with what you were doing. Also, when during the sprint we see people have been working too hard, we suggest they go home early today, …because they deserve it. Commitment to the team is great, but staying healthy is even more important. And if you see that other people in the team care about you, that likely increases the commitment again.
One of the nice things in our team is that everybody has their own specialties and preferences. We have a guy who likes frontend gui stuff, a girl who prefers the backend work, someone for mobile, a tester and a scrum master who likes agile coaching and keeping everybody happy. I am the architect, and I like to work on continuous delivery and security. As said before, interests don’t mean we don’t work on anything else. However, it helps to decide who can work on what. Pair programming and code review make sure that your knowledge is not just limited to your own interest. Also, at the end of the sprint, everybody just works on what needs finishing, whatever the preferences.
As a team, it is important to try to stay as independent as possible. Not just regarding procedures and controlling our own server, but also regarding our software. We don’t want dependencies with other teams or products. When we need to build an interface for some other product, we build it in such a way that it doesn’t break if the other product is down, or if the other product has not released some update yet. We’re always backwards compatible. That way we can move forward independently and go as quick as we can. Other teams just need to try to keep up.
Training is also important. There are many ways we try to keep our knowledge up to date. One way is by going to conferences. That’s where most of the ideas come from, but it can also be expensive, so unfortunately we can’t do that every week. Besides, after a conference, you need some time to actually use the newly gained knowledge and ideas and start working on it. In the meantime, we do knowledge sharing, within the team, and when possible also between teams. For example, we have workshops where we discuss some book. Anyone who is interested can join. Everybody reads the same chapters, and we discuss the interesting parts. This helps us understand better how to use the theory in practice and it’s a fun way of teaching others. Also, several team members are active in guilds where they share knowledge and where they find new inspiration for our own team. E.g. We have a scrum master guild, a security guild, a testing guild and an UX guild.
We still have plenty of challenges and struggles. One challenge lies in continuous delivery. We’ve already come quite far, but the last bits are always the hardest. Continuous Delivery and continuous deployment would be a huge improvement for the quality and the efficiency of our work. For now we have continuous integration. On each code commit, unit tests and integration tests are run. If successful we automatically deploy to our Test environment. Every night we do an additional nightly build that also includes a vulnerability scan. Separately we have static code analysis, which unfortunately is not yet linked to the build. Features are released through feature toggles. Often first to a pilot group and later to the rest of our customers. The last bits are in automating the SQL scripts, automatically deploying to Acceptance and Production, and most importantly, gaining enough trust in the quality by increasing the amount and relevance of automated tests.
Another challenge is that we have a team that is getting bigger and more distributed. Communication is vital and issues arise quickly if we forget to update our team members. Different people from different countries have different expectations, communication styles and backgrounds. It takes time to adjust and let the team grow.
Perhaps the biggest challenge we have is autonomy. We’re the only team at Raet working as devops, so keeping our autonomy is really difficult. Often new restrictive rules get applied to all teams, including ours, even though it doesn’t make sense for us. Fortunately, in the long run, everybody moves forward, so we’re getting there sooner or later…