DevOps Introduction
### Article Based on Video Transcription: "Introduction to DevOps Culture, Automation, and Practices"
---
#### Introduction
Klaus, a software developer with over 10 years of experience and currently specializing in GoLang, Pearl, and TypeScript, delivered an insightful talk about DevOps at Free Code Camp. Known for his expertise in causing and fixing technical challenges, Klaus shared his journey from being a system administrator to becoming a software developer and how he integrates DevOps principles into his work. His talk focused on the cultural, automated, and lean aspects of DevOps, emphasizing the importance of collaboration across teams, automation, and continuous improvement.
---
#### The Evolution of DevOps
Klaus began by explaining how DevOps emerged from the silos that often existed between developers and system administrators. In the past, these two groups operated in separate worlds: developers focused on writing code and experimenting with new tools, while system administrators were responsible for ensuring stability and security. This divide led to inefficiencies and delays in delivering products.
At Klaus's previous university job, he managed 20,000 users and realized that manual interventions were impractical. Instead, the team began scripting infrastructure and automating repetitive tasks. This marked the early seeds of DevOps. Tools like cfengine (before Puppet) were used to manage configurations, and the team started moving away from manual processes.
---
#### Key Principles of DevOps
1. **Culture of Collaboration**
Klaus highlighted that DevOps is not just about tools or processes but primarily about culture. Traditional hierarchical structures often lead to blame-based environments, where mistakes are met with criticism rather than learning opportunities. Instead, DevOps encourages a blame-free culture where everyone can experiment and learn from failures.
- **Shared Responsibility**: Teams should work together across silos, including marketing, design, and sales. Every team member is responsible for the product's success.
- **Autonomous Teams**: By fostering collaboration, teams become more self-sufficient and less reliant on hierarchical decision-making.
2. **Automation**
One of the core principles of DevOps is automation. Klaus emphasized that any repetitive task should be automated to save time and reduce errors. He referenced the "Site Reliability Engineering (SRE)" book by Google, which introduces the concept of "toil" — manual, repetitive work that should be eliminated through automation.
- **Infrastructure as Code**: Use tools like Ansible or Terraform to manage infrastructure programmatically rather than manually configuring servers.
- **Continuous Integration and Delivery (CI/CD)**: Implement pipelines for automated testing, building, and deploying code. Tools like Git, Jenkins, and Kubernetes can help streamline this process.
3. **Lean Practices**
Lean practices focus on eliminating waste, amplifying learning, deciding as late as possible, and delivering value quickly. Klaus shared his experience with lean practices in DevOps:
- Avoid over-engineering solutions. Instead, start simple and iterate based on feedback.
- Use monitoring and instrumentation to identify bottlenecks and areas for improvement.
4. **Measurement and Instrumentation**
Klaus stressed the importance of measuring everything in a system. Without data, it's impossible to analyze performance or improve processes.
- **Monitoring Tools**: Use tools like Prometheus, Grafana, or ELK (Elasticsearch, Logstash, Kibana) for monitoring and logging.
- **A/B Testing**: Experiment with features and use metrics to determine their impact on users.
---
#### Challenges and Common Pitfalls
Klaus also discussed common mistakes teams make when adopting DevOps:
1. **Half-Commitment**: Partially implementing DevOps practices leads to inefficiencies and frustration. Either fully embrace DevOps or don't start at all.
2. **Over-reliance on Pet Servers**: Avoid managing servers manually or favoring one-off configurations. Instead, abstract infrastructure as code.
3. **Outsourcing Core Functions**: Outsourcing critical tasks like DNS hosting can lead to dependency issues and reduced control over the product.
---
#### Continuous Learning and Improvement
Klaus emphasized the importance of continuous learning in DevOps. Every team member should strive to improve their skills and stay updated with industry trends. He also highlighted the role of communication within teams, especially in small environments where collaboration is easier.
Finally, Klaus shared a cautionary tale about the risks of manual interventions. For example, manually restarting a web server to fix memory leaks may seem quick but can lead to long-term issues. Instead, automate such processes to reduce human error and workload.
---
#### Conclusion
Klaus's talk on DevOps was both practical and inspiring. He demonstrated how integrating cultural changes, automation, and lean practices can transform software development and operations. By fostering collaboration, embracing automation, and prioritizing continuous learning, teams can build resilient systems and deliver value more efficiently. As he concluded, "DevOps is not just a buzzword; it's about creating sustainable processes that benefit both teams and customers."
"WEBVTTKind: captionsLanguage: enso my introduction i haven't given you my introduction today i'm a first timer actually at free code camp and um already giving a talk um how good is that and i'm gonna talk about devops it's not really coding i won't show much code but i want to show you something or talk about the culture of devops a little bit and i see many developers in every day and i don't see many companies that get devops right and it's very a very important topic and i think everybody should know something about it first thank you thanks we accelerate i think it's really great because um we are starting to do more and more meetups here in this uh place and i think it's um good to have we accelerate here and giving us the opportunity to to enjoy this venue um just a few links i'll probably put the slides online and i like to do things and references first because otherwise nobody will see them anymore in in afterwards uh introduction who am i i'm a software developer for 10 years i'm not really a starter and currently my favorite language is golang i do a lot of pearl and i do typescript mostly because i have to i'm not too good at it and i started off as a cis admin some what 15 years ago i thought about it yesterday when i wrote my slides and was cis admin at the university of economics in austria and so i had some 20 000 users we had to administer and you guessed uh probably there wasn't much manual intervention possible doing that so we had to do a lot of coding um so i switched to becoming a software developer when i actually quit the university and i'm quite an expert at causing and fixing disasters um causing is easy but it doesn't happen as much anymore and fixing is okay it takes some time and experience but i'm pretty relaxed with the situations i get dumped into my name is klaus ita forgot to say that usually i have a nick somewhat combined with cokie cookie is very quickly taken in all the public websites so look for derivatives of that that's my twitter handle it's easy to remember and um sorry for the layout because my slides got messed up when i arrived here um they were all written in markdown and i should have some plugin that does them to html converts them to html but something happened so i had to rewrite some macro in my editor to just quickly make html from my slides so for that they are pretty beautiful and i'm a total dope at css so please just get used to those slides and they won't get prettier than that some buzzwords not important just some get the feeling if you have any questions just ask them okay i don't like having all the questions afterwards because maybe you don't remember and you might have it's good to have everyone all the questions at the end because we don't have a microphone for the okay okay sorry no questions hate you no questions uh devops um the initial setting um how how did we find that devops or how was it incubated or whatever uh there was this admins and there were developers and and there were business requirements and sales people and each of those silos actually hated each other um or i think they still do more or less but um something um is happening so um the two sides that were the most opposed sides were the scissor admins and the developers and and i'm both so it's easy for me to to tell you this um it gave me hard times uh quite often sorry for the small uh funds but that's my css skills okay um so then there's a there's a conflict in says admin and development as a developer you always check for the latest library and you always want to have the latest and greatest tools and try this and try try that some ai or whatever and um as a says admin you usually are being paid for sitting in the back room and just keeping everything stable and as soon as something moves a little bit you just bash it down so it doesn't move anymore and so those two groups are somewhat uh not in sync and someone someday just said well you know um we could work together and um at the time i was in at that university i told you about um there was i think that there was the seed of devops coming up that there were configuration engines um starting to get bigger and bigger and um it was the very early i think puppet wasn't no puppet what there was no puppet at that time but there was cf engine at the time and and people started scripting infrastructure and and people stopped actually doing things manually um the bad thing is i still see a lot of people manually intervening with their computers and the what are the goals of devops um we just want to keep a constant flow through the technology um value stream so so what does that mean it's um in the old days you always had infrastructure and you had development and those two didn't work well together and especially in bigger companies there was software development and software development cycles where months or years long and then there was an integration phase and that handed over the software project to the ops people which are system administrators and they tried to figure out how to actually get a project working and running and stably running and that pushed the launch date of every product further and further away and i still see that in many startup companies now also i mean people are developing and developing and then there's release day and they find out oh we never thought about infrastructure where we actually want to run the project but it works on my machine i see that quite a lot and it's it's a lot of about culture to actually put together all the team and make everybody participate from from the team so it's not only developers or this admins but it's also the marketing people it's the icon designers it's the i don't know css people are those developers yeah let's say and it's the sales people that mustn't sell things that that can't be developed or everybody needs to be in a team and this is a bit what devops is about um also the the thing i explained before day zero day one day two day end the product has to run all the time and yeah um basically work together in all um all phases of its life devops is about culture automation lean measurement and sharing these are the key points of um of devops let's go through them culture is um something i see a lot in in technology is there's that boss and and the boss shouts at everybody and um this is this won't i mean this this leads to results in short term but people leave and people join companies people leave managers and it doesn't really lead to a good team and and to a good work experience and what we really want to impose is get blame-free learning experiment and experimentation uh framework so we all make mistakes we make a lot of mistakes probably but the only difference be between a well established developer or devops guy or whatever and and the beginner is the number of mistakes they have made that that's the only difference and yeah this means we really um we do have volatile in environment so anything we do changes all the time and because we do experiment everybody is allowed to experiment everybody's allowed to bring in new technologies we change everything all the time but still we managed to reduce risk i'll explain that later how we managed to work that out we tried to have no more silos and which is actually fun because we start actually talking to people there were some people who came here to to talk to others this is true for every company the developers can start talking to marketing people and this admins can start talking to sales people and actually this is an interesting thought that should be should be a baseline in every company anyway we all share the responsibility for our finished product we all earned the money that the company earns and we all have to bring in money and we're all important for the company so of course we share responsibility and it's important that everybody knows that everybody is responsible for having the product out and autonomous teams is just because we share responsibility we are able to actually decide things on our own we don't need to ask the boss all the time automation um there's a i think on the first slide i had those thank you uh things um there's a book it's um it's a long book and it gets a bit boring after a while but it's the sre book of by google i don't know if you know site reliability engineering and um but they have amazing chapters that are really good and one of them is about uh automation and toil especially and i like the term it's it's um it was created in google i think and it means that anything you do and anything you do more than once that can be automated must be automated and this is one of the really basic thoughts in devops so whatever can be triggered by whatever metrics needs to be automatically automatically solved because otherwise what you might see you have many many busy people that do the same thing over and over and again and this is the worst that can happen okay so anything that's manual repetitive and automatable tactically important for our company has no value at all that's very important you know the the web server that needs to be restarted because whatever we have bad code and and we have mem leaks or whatever and the sysadmin goes to and look logs into that web server ssh is in there and and just types in nginx restart that's a waste of time you can automate that yeah you can just watch memory and just kill that thingy and and it'll have the same effect it's not a solution it's not a long-term solution but it it will uh take away work from the cis admin who can then develop a better solution and probably reduce man leaks okay and doing this removing toil all the time is a huge resource saver and um imagine i mean at google the sre teams they are not allowed they are not even allowed to do more than 50 toil in their their work time which is i mean think of your scissor admins what they do all the time um least of them are actually living that um well there is the term lean eliminate waste amplify learning decide as late as possible deliver as fast as possible that's all the the uh um i mean i hope you you live by those standards now and empower the team build integrity and see the whole see that all is i think the only thing i really have to explain more um even as a developer or says admin or whatever person or in whatever topic you are in just see what the goal is and and always know what is my company doing and what are we actually trying to get out to our customer it's very important to to see that all the time is what i am doing currently important for my customers does it actually give value to the customer um instrumentation anything and this gets forgotten by most i don't know how many startups are here and i just saw some everybody who who's making new products and many people have started new products and they forget about measurements and instrumentation and this should be one of the core built-ins in every product that you create um if you don't instrument everything you do and if you don't have graphs for every whatever information you you generate in your software you will fail at devops because if you can't measure things and if you can't analyze things you can't react and of course a human being can react and i have a very good feeling about who i don't know i think the system is in a bad state and i have to do something but if you don't have numbers to prove and if you don't have numbers to analyze you can't automate these things here you see a graph from this is something actually a marketing guy created yeah these graphs um we we are generating all the the information and the marketing guy just clicks together it's some health whatever tool and and they can just see okay how do the numbers perform did we i don't know we had some release and numbers went down and then other releases and we tried to to get the release uh see the numbers up again and just just to show you that anything we do and whatever whatever this is a registration verify verification profile setup so so people that try to register um verify their prof profile and then or change data in the their profile which is important in this project we just measure what they do so we have the i i call it the devops pretzel and if you go to devops meetups nowadays uh you see this pretzel all the time and i just wanted to show it to you um it's it's a never-ending cycle of okay you release a product you configure it you monitor monitoring okay you plan for the next release you build you you create you check the numbers again from monitoring you package it release it and so on and so on and on and that cycle goes forever and now the testimony i'm a developer okay devops is something that should actually um impact the whole company and i don't know much about how marketing people feel about it i mean i know that they work in my team together with me and and we have a good relationship but i don't know what they feel about the whole devops movement um so what i can only talk about is how software development goes software operations and how we execute it okay i don't really care about the finance department so much okay how do we live it um we use git or whatever revision controls system you you're using do whatever you want but i mean i think everybody uses git um if you're not and i've i've seen that recently in the last two years actually companies not using it start using it use some integration service everybody knows what an integration service is it's an automated tool that takes your product and puts it into production it must be automated don't have someone say okay should we release today maybe you talk about releasing today but it must be automated the whole integration and releasing process must be such a non-op that you do it all the time you did all all the time and use build tools so however you try to make your final product i don't know if you do web design you probably have some grunt or webpack or whatever running automate that you can have it running but if it needs any parameters script that at least um we have build tools built into the git chain so if i get pushed then the product gets built and released um i would recommend i mean this is not an official statement i'd recommend um you use a cloud it doesn't matter if you have your own cloud if you have i don't know a service center and 2000 servers and just run your cloud find that that's fine but the concept of cloud um must be implemented so so probably if you're in smaller companies use i don't know aws google something like that use provisioning tools for everything you don't start naming servers or anything like that the a i'll come to that later and and see your hardware as software okay so don't don't name your hardware don't don't know your hardware anymore abstract it make it software okay continuous integration there's a great article by martin fowler about it martin fowler you all know martin fowler he's i i don't know an and i can in i.t i'd say um he writes articles all of them are like uh 15 pages long and and quite quite a read but um always well well thought of and and well designed and and he's look him up martin fowler and read all his articles um it'll take you a month or so but it's worth it and he says about continuous integration what's the base point check out mainline main line is the master branch yeah develop a quick feature and i mean we even have tools for um if we have an issue or ticket that we want to work on that actually creates a branch named like that ticket so everybody who works on that same issue creates the same branch name so we can share and and interact with that build test and merge fully automated very often and always know your code is running and failing or broken builds don't make it to mainline so we have all that automated except for the coding part we need to do that unfortunately and we push into the repo everything gets built um i don't need to type to build and merge and whatever um it all gets done by tooling and after a minute i get an email if the build fails other or pop up on my screen actually and otherwise my changes get merged into mainline and are in the staging staging environment continuous delivery deliver all the time so what i've done before i coded i merged it into mainline and now it's about delivery and every successful merge you see gets into that staging environment so that we have a web page that has a total clone of everything we do we can have multiple clones also but but staging is the most important preview it runs against a real clone of the actual data and we have buttons for for creative review promote to production or rollback so there's no manual intervention with anything we do actually so whatev and these buttons can be pressed by a marketing guy and this is very important that it's not always the developers that decide okay we can go online no if if all the tests pass and if the product looks fine to the marketing guys they are the ones that wanted to feature of course they press the button to to actually release that feature if they don't press it i press it yeah anybody can and so this is standard deployments each of those what's that vertical lines are a deployment and that's standard for us that we actually deploy all the time and then you watch i don't know something happens or it doesn't happen and and so we always mark everything that's the instrumentation part i talked about you need to know okay we had this code running and it had this impact on our customers so maybe we need to change something or maybe that was a good change and let's party um by deploying so often we keep things unstable but we reduce risk if we deploy only once a month or only once a year or a half year then the deployment gets really tough and now we just deploy and everything is fine unless you do disaster deployment like i did last week where i blacked out for half an hour i think the whole product but that was very special it didn't happen for years actually because usually we have that but this is also the experience nobody shouted at me when that happened we just found out okay what was the problem and how do we fix it and how do we prevent it from happening the next time it was my mistake i did something wrong but how can we prevent it in the future and so what we want is that we actually have the queue and it's running the pipeline is running and it says okay sorry i can't continue something something happened i don't know what happened here but whatever and we are safe we have way too many features to test them all um we've been working on this product i didn't i think for nine years or something and it's just too many features and sometimes we have long workflows so we gather information and we export it to different clients and and and we improve the data we recycle it and there's so many workflows they take a while and and they need to take a while and and we have different performances you just can't test everything um and but we do have alerts for every core business case so whatever happens we know so we see we act we know okay and and if we if we see we have a problem and we don't have measurements yet we act so we introduce measurements instrumentation for that problem and in the future we have something automated for it and we have many alerts that shouldn't trigger okay and um we always know exactly when which version of code was online so we have our stats and we can always say okay at that point in time we had this revision running very important i see that a lot that people don't know they just go to service to get pull so that they think they're advanced because they use git they do get pull server restart have no idea what happens and um yeah we have much better things to do than testing our code manually so i don't want to browse our website all the time and and then we have products that i don't even want to touch too much because i don't know i just don't like them and it's good to have that automated and we need to detect errors as fast as possible and stop them from passing along the value stream so even when i blacked out last week i knew it within seconds and that's very important and that you know so you can act quickly and and fix things this is still marked on syntax um be very disciplined so if you want to do that you and the whole team must be very disciplined we we do we create strict 12-factor apps we don't ever do quick fixes even if i know okay i could just log in and change that line in nginx and and fix some i don't know proxy path or something and it would work we don't do it um we create the fix we patch we commit we release and that always has to be this way and if a broken commit exists in in your tool chain that's the most important bug to fix because as soon as you let your tools down and you start to work around the tools and manually install things the whole system breaks yeah always use tools and if you can't do it you can script it that that's my philosophy whatever you do it is scriptable it might be hard it might be hard the first time it gets easier and easier okay and very important is learn all the time i read books all the time i read articles all the time there's not a single day i don't learn something and i could say that for my whole team nobody nobody ever lays down and says okay i know enough just doesn't happen and communicate a lot i mean it's easy for us we're with less than a two pizza team i eat one pizza and no way more than that but it's easy for us to communicate otherwise we are also not in danger to have the silos because we are very close together we are still in one office which is in one floor so it's not um not a big i don't know like unico whatever over there we have huge buildings and and it's hard to communicate so for us really communication is easy um i want to talk about failures things that did not work i've worked at multiple companies with multiple companies i've seen things fail i failed i failed with my team i saw teams fail after investing hundreds of thousands of euro into devops what doesn't work being half committed doesn't work okay either you do it or you don't do it halfway costs a lot of money ansible or script deployments on pet servers everybody knows what the pet server is it's one way you know the host name it's a server you like and never have a server you like okay um wiki playbooks for deployments and upgrades i see that a lot people think they're so smart because they have everything written down in in wiki and they copy and paste every line and then create a new project oh we are got a new customer and creating new customer takes a quarter of an hour i don't know three hours i don't know copy paste all the lines from wiki i mean just script that or make make a microservice that creates a new customer strongly wired services doesn't know it doesn't never work i mean we've fully migrated to microservice environments and initially the first migration started with um service a knows exactly about service b and we can talk to each other no it's like pet servers it's just pet software um what didn't work using kvm then or vm where in in our deployments so for the software we were trying to deploy it those virtualization technologies work very well to abstract hardware but they don't work well to abstract software for infrastructure yes but not for the deployed product itself delegating core services um sorry to say whoever outsourcing companies here um the i don't know dns hosting all these core things that are needed for your product you must do all that in-house outsourcing software development didn't work for us nobody can outsource a philosophy and it's the main thing you do if you if you do software and every company does software nowadays and every company mainly does i.t nowadays different aspects of it but everybody needs it you can't outsource it outsourcing scissor admins very bad idea taking responsibility away from people we tried that with people who didn't want to devop and who didn't want to work in that strategy so we said okay you just fix that little issue and then you commit it and we cover for the rest um doesn't work you unfortunately have to get rid of those people multiple git repositories didn't work for us we have a lot of i don't know that that company has seven seven hundred makes of of source code and we have it all in in one repo and it's fine with git nobody cares and if you have multiple repositories you have to work with git sub modules and stuff doesn't work so well just use one monorepo and we do that everything you do must always be code and all truth lies in the repo so if i die today doesn't really matter because i mean i wouldn't care and and otherwise anything i've done even my talk which i just rewrote isn't git anything not committed has no value at all because it can just be gone or broken or what i don't know you can't find it anymore um especially in case you run into a bus and or you get hacked because we've got that hacked server i want to see or you want to change your infrastructure which is the most important point i mean that happens frequently or frequently every few years you want to switch from one provider to another one if you don't have domain or anything actually everything script scripted you won't be able to make that move if you have to manually set up infrastructure won't work and thanks very much questionsso my introduction i haven't given you my introduction today i'm a first timer actually at free code camp and um already giving a talk um how good is that and i'm gonna talk about devops it's not really coding i won't show much code but i want to show you something or talk about the culture of devops a little bit and i see many developers in every day and i don't see many companies that get devops right and it's very a very important topic and i think everybody should know something about it first thank you thanks we accelerate i think it's really great because um we are starting to do more and more meetups here in this uh place and i think it's um good to have we accelerate here and giving us the opportunity to to enjoy this venue um just a few links i'll probably put the slides online and i like to do things and references first because otherwise nobody will see them anymore in in afterwards uh introduction who am i i'm a software developer for 10 years i'm not really a starter and currently my favorite language is golang i do a lot of pearl and i do typescript mostly because i have to i'm not too good at it and i started off as a cis admin some what 15 years ago i thought about it yesterday when i wrote my slides and was cis admin at the university of economics in austria and so i had some 20 000 users we had to administer and you guessed uh probably there wasn't much manual intervention possible doing that so we had to do a lot of coding um so i switched to becoming a software developer when i actually quit the university and i'm quite an expert at causing and fixing disasters um causing is easy but it doesn't happen as much anymore and fixing is okay it takes some time and experience but i'm pretty relaxed with the situations i get dumped into my name is klaus ita forgot to say that usually i have a nick somewhat combined with cokie cookie is very quickly taken in all the public websites so look for derivatives of that that's my twitter handle it's easy to remember and um sorry for the layout because my slides got messed up when i arrived here um they were all written in markdown and i should have some plugin that does them to html converts them to html but something happened so i had to rewrite some macro in my editor to just quickly make html from my slides so for that they are pretty beautiful and i'm a total dope at css so please just get used to those slides and they won't get prettier than that some buzzwords not important just some get the feeling if you have any questions just ask them okay i don't like having all the questions afterwards because maybe you don't remember and you might have it's good to have everyone all the questions at the end because we don't have a microphone for the okay okay sorry no questions hate you no questions uh devops um the initial setting um how how did we find that devops or how was it incubated or whatever uh there was this admins and there were developers and and there were business requirements and sales people and each of those silos actually hated each other um or i think they still do more or less but um something um is happening so um the two sides that were the most opposed sides were the scissor admins and the developers and and i'm both so it's easy for me to to tell you this um it gave me hard times uh quite often sorry for the small uh funds but that's my css skills okay um so then there's a there's a conflict in says admin and development as a developer you always check for the latest library and you always want to have the latest and greatest tools and try this and try try that some ai or whatever and um as a says admin you usually are being paid for sitting in the back room and just keeping everything stable and as soon as something moves a little bit you just bash it down so it doesn't move anymore and so those two groups are somewhat uh not in sync and someone someday just said well you know um we could work together and um at the time i was in at that university i told you about um there was i think that there was the seed of devops coming up that there were configuration engines um starting to get bigger and bigger and um it was the very early i think puppet wasn't no puppet what there was no puppet at that time but there was cf engine at the time and and people started scripting infrastructure and and people stopped actually doing things manually um the bad thing is i still see a lot of people manually intervening with their computers and the what are the goals of devops um we just want to keep a constant flow through the technology um value stream so so what does that mean it's um in the old days you always had infrastructure and you had development and those two didn't work well together and especially in bigger companies there was software development and software development cycles where months or years long and then there was an integration phase and that handed over the software project to the ops people which are system administrators and they tried to figure out how to actually get a project working and running and stably running and that pushed the launch date of every product further and further away and i still see that in many startup companies now also i mean people are developing and developing and then there's release day and they find out oh we never thought about infrastructure where we actually want to run the project but it works on my machine i see that quite a lot and it's it's a lot of about culture to actually put together all the team and make everybody participate from from the team so it's not only developers or this admins but it's also the marketing people it's the icon designers it's the i don't know css people are those developers yeah let's say and it's the sales people that mustn't sell things that that can't be developed or everybody needs to be in a team and this is a bit what devops is about um also the the thing i explained before day zero day one day two day end the product has to run all the time and yeah um basically work together in all um all phases of its life devops is about culture automation lean measurement and sharing these are the key points of um of devops let's go through them culture is um something i see a lot in in technology is there's that boss and and the boss shouts at everybody and um this is this won't i mean this this leads to results in short term but people leave and people join companies people leave managers and it doesn't really lead to a good team and and to a good work experience and what we really want to impose is get blame-free learning experiment and experimentation uh framework so we all make mistakes we make a lot of mistakes probably but the only difference be between a well established developer or devops guy or whatever and and the beginner is the number of mistakes they have made that that's the only difference and yeah this means we really um we do have volatile in environment so anything we do changes all the time and because we do experiment everybody is allowed to experiment everybody's allowed to bring in new technologies we change everything all the time but still we managed to reduce risk i'll explain that later how we managed to work that out we tried to have no more silos and which is actually fun because we start actually talking to people there were some people who came here to to talk to others this is true for every company the developers can start talking to marketing people and this admins can start talking to sales people and actually this is an interesting thought that should be should be a baseline in every company anyway we all share the responsibility for our finished product we all earned the money that the company earns and we all have to bring in money and we're all important for the company so of course we share responsibility and it's important that everybody knows that everybody is responsible for having the product out and autonomous teams is just because we share responsibility we are able to actually decide things on our own we don't need to ask the boss all the time automation um there's a i think on the first slide i had those thank you uh things um there's a book it's um it's a long book and it gets a bit boring after a while but it's the sre book of by google i don't know if you know site reliability engineering and um but they have amazing chapters that are really good and one of them is about uh automation and toil especially and i like the term it's it's um it was created in google i think and it means that anything you do and anything you do more than once that can be automated must be automated and this is one of the really basic thoughts in devops so whatever can be triggered by whatever metrics needs to be automatically automatically solved because otherwise what you might see you have many many busy people that do the same thing over and over and again and this is the worst that can happen okay so anything that's manual repetitive and automatable tactically important for our company has no value at all that's very important you know the the web server that needs to be restarted because whatever we have bad code and and we have mem leaks or whatever and the sysadmin goes to and look logs into that web server ssh is in there and and just types in nginx restart that's a waste of time you can automate that yeah you can just watch memory and just kill that thingy and and it'll have the same effect it's not a solution it's not a long-term solution but it it will uh take away work from the cis admin who can then develop a better solution and probably reduce man leaks okay and doing this removing toil all the time is a huge resource saver and um imagine i mean at google the sre teams they are not allowed they are not even allowed to do more than 50 toil in their their work time which is i mean think of your scissor admins what they do all the time um least of them are actually living that um well there is the term lean eliminate waste amplify learning decide as late as possible deliver as fast as possible that's all the the uh um i mean i hope you you live by those standards now and empower the team build integrity and see the whole see that all is i think the only thing i really have to explain more um even as a developer or says admin or whatever person or in whatever topic you are in just see what the goal is and and always know what is my company doing and what are we actually trying to get out to our customer it's very important to to see that all the time is what i am doing currently important for my customers does it actually give value to the customer um instrumentation anything and this gets forgotten by most i don't know how many startups are here and i just saw some everybody who who's making new products and many people have started new products and they forget about measurements and instrumentation and this should be one of the core built-ins in every product that you create um if you don't instrument everything you do and if you don't have graphs for every whatever information you you generate in your software you will fail at devops because if you can't measure things and if you can't analyze things you can't react and of course a human being can react and i have a very good feeling about who i don't know i think the system is in a bad state and i have to do something but if you don't have numbers to prove and if you don't have numbers to analyze you can't automate these things here you see a graph from this is something actually a marketing guy created yeah these graphs um we we are generating all the the information and the marketing guy just clicks together it's some health whatever tool and and they can just see okay how do the numbers perform did we i don't know we had some release and numbers went down and then other releases and we tried to to get the release uh see the numbers up again and just just to show you that anything we do and whatever whatever this is a registration verify verification profile setup so so people that try to register um verify their prof profile and then or change data in the their profile which is important in this project we just measure what they do so we have the i i call it the devops pretzel and if you go to devops meetups nowadays uh you see this pretzel all the time and i just wanted to show it to you um it's it's a never-ending cycle of okay you release a product you configure it you monitor monitoring okay you plan for the next release you build you you create you check the numbers again from monitoring you package it release it and so on and so on and on and that cycle goes forever and now the testimony i'm a developer okay devops is something that should actually um impact the whole company and i don't know much about how marketing people feel about it i mean i know that they work in my team together with me and and we have a good relationship but i don't know what they feel about the whole devops movement um so what i can only talk about is how software development goes software operations and how we execute it okay i don't really care about the finance department so much okay how do we live it um we use git or whatever revision controls system you you're using do whatever you want but i mean i think everybody uses git um if you're not and i've i've seen that recently in the last two years actually companies not using it start using it use some integration service everybody knows what an integration service is it's an automated tool that takes your product and puts it into production it must be automated don't have someone say okay should we release today maybe you talk about releasing today but it must be automated the whole integration and releasing process must be such a non-op that you do it all the time you did all all the time and use build tools so however you try to make your final product i don't know if you do web design you probably have some grunt or webpack or whatever running automate that you can have it running but if it needs any parameters script that at least um we have build tools built into the git chain so if i get pushed then the product gets built and released um i would recommend i mean this is not an official statement i'd recommend um you use a cloud it doesn't matter if you have your own cloud if you have i don't know a service center and 2000 servers and just run your cloud find that that's fine but the concept of cloud um must be implemented so so probably if you're in smaller companies use i don't know aws google something like that use provisioning tools for everything you don't start naming servers or anything like that the a i'll come to that later and and see your hardware as software okay so don't don't name your hardware don't don't know your hardware anymore abstract it make it software okay continuous integration there's a great article by martin fowler about it martin fowler you all know martin fowler he's i i don't know an and i can in i.t i'd say um he writes articles all of them are like uh 15 pages long and and quite quite a read but um always well well thought of and and well designed and and he's look him up martin fowler and read all his articles um it'll take you a month or so but it's worth it and he says about continuous integration what's the base point check out mainline main line is the master branch yeah develop a quick feature and i mean we even have tools for um if we have an issue or ticket that we want to work on that actually creates a branch named like that ticket so everybody who works on that same issue creates the same branch name so we can share and and interact with that build test and merge fully automated very often and always know your code is running and failing or broken builds don't make it to mainline so we have all that automated except for the coding part we need to do that unfortunately and we push into the repo everything gets built um i don't need to type to build and merge and whatever um it all gets done by tooling and after a minute i get an email if the build fails other or pop up on my screen actually and otherwise my changes get merged into mainline and are in the staging staging environment continuous delivery deliver all the time so what i've done before i coded i merged it into mainline and now it's about delivery and every successful merge you see gets into that staging environment so that we have a web page that has a total clone of everything we do we can have multiple clones also but but staging is the most important preview it runs against a real clone of the actual data and we have buttons for for creative review promote to production or rollback so there's no manual intervention with anything we do actually so whatev and these buttons can be pressed by a marketing guy and this is very important that it's not always the developers that decide okay we can go online no if if all the tests pass and if the product looks fine to the marketing guys they are the ones that wanted to feature of course they press the button to to actually release that feature if they don't press it i press it yeah anybody can and so this is standard deployments each of those what's that vertical lines are a deployment and that's standard for us that we actually deploy all the time and then you watch i don't know something happens or it doesn't happen and and so we always mark everything that's the instrumentation part i talked about you need to know okay we had this code running and it had this impact on our customers so maybe we need to change something or maybe that was a good change and let's party um by deploying so often we keep things unstable but we reduce risk if we deploy only once a month or only once a year or a half year then the deployment gets really tough and now we just deploy and everything is fine unless you do disaster deployment like i did last week where i blacked out for half an hour i think the whole product but that was very special it didn't happen for years actually because usually we have that but this is also the experience nobody shouted at me when that happened we just found out okay what was the problem and how do we fix it and how do we prevent it from happening the next time it was my mistake i did something wrong but how can we prevent it in the future and so what we want is that we actually have the queue and it's running the pipeline is running and it says okay sorry i can't continue something something happened i don't know what happened here but whatever and we are safe we have way too many features to test them all um we've been working on this product i didn't i think for nine years or something and it's just too many features and sometimes we have long workflows so we gather information and we export it to different clients and and and we improve the data we recycle it and there's so many workflows they take a while and and they need to take a while and and we have different performances you just can't test everything um and but we do have alerts for every core business case so whatever happens we know so we see we act we know okay and and if we if we see we have a problem and we don't have measurements yet we act so we introduce measurements instrumentation for that problem and in the future we have something automated for it and we have many alerts that shouldn't trigger okay and um we always know exactly when which version of code was online so we have our stats and we can always say okay at that point in time we had this revision running very important i see that a lot that people don't know they just go to service to get pull so that they think they're advanced because they use git they do get pull server restart have no idea what happens and um yeah we have much better things to do than testing our code manually so i don't want to browse our website all the time and and then we have products that i don't even want to touch too much because i don't know i just don't like them and it's good to have that automated and we need to detect errors as fast as possible and stop them from passing along the value stream so even when i blacked out last week i knew it within seconds and that's very important and that you know so you can act quickly and and fix things this is still marked on syntax um be very disciplined so if you want to do that you and the whole team must be very disciplined we we do we create strict 12-factor apps we don't ever do quick fixes even if i know okay i could just log in and change that line in nginx and and fix some i don't know proxy path or something and it would work we don't do it um we create the fix we patch we commit we release and that always has to be this way and if a broken commit exists in in your tool chain that's the most important bug to fix because as soon as you let your tools down and you start to work around the tools and manually install things the whole system breaks yeah always use tools and if you can't do it you can script it that that's my philosophy whatever you do it is scriptable it might be hard it might be hard the first time it gets easier and easier okay and very important is learn all the time i read books all the time i read articles all the time there's not a single day i don't learn something and i could say that for my whole team nobody nobody ever lays down and says okay i know enough just doesn't happen and communicate a lot i mean it's easy for us we're with less than a two pizza team i eat one pizza and no way more than that but it's easy for us to communicate otherwise we are also not in danger to have the silos because we are very close together we are still in one office which is in one floor so it's not um not a big i don't know like unico whatever over there we have huge buildings and and it's hard to communicate so for us really communication is easy um i want to talk about failures things that did not work i've worked at multiple companies with multiple companies i've seen things fail i failed i failed with my team i saw teams fail after investing hundreds of thousands of euro into devops what doesn't work being half committed doesn't work okay either you do it or you don't do it halfway costs a lot of money ansible or script deployments on pet servers everybody knows what the pet server is it's one way you know the host name it's a server you like and never have a server you like okay um wiki playbooks for deployments and upgrades i see that a lot people think they're so smart because they have everything written down in in wiki and they copy and paste every line and then create a new project oh we are got a new customer and creating new customer takes a quarter of an hour i don't know three hours i don't know copy paste all the lines from wiki i mean just script that or make make a microservice that creates a new customer strongly wired services doesn't know it doesn't never work i mean we've fully migrated to microservice environments and initially the first migration started with um service a knows exactly about service b and we can talk to each other no it's like pet servers it's just pet software um what didn't work using kvm then or vm where in in our deployments so for the software we were trying to deploy it those virtualization technologies work very well to abstract hardware but they don't work well to abstract software for infrastructure yes but not for the deployed product itself delegating core services um sorry to say whoever outsourcing companies here um the i don't know dns hosting all these core things that are needed for your product you must do all that in-house outsourcing software development didn't work for us nobody can outsource a philosophy and it's the main thing you do if you if you do software and every company does software nowadays and every company mainly does i.t nowadays different aspects of it but everybody needs it you can't outsource it outsourcing scissor admins very bad idea taking responsibility away from people we tried that with people who didn't want to devop and who didn't want to work in that strategy so we said okay you just fix that little issue and then you commit it and we cover for the rest um doesn't work you unfortunately have to get rid of those people multiple git repositories didn't work for us we have a lot of i don't know that that company has seven seven hundred makes of of source code and we have it all in in one repo and it's fine with git nobody cares and if you have multiple repositories you have to work with git sub modules and stuff doesn't work so well just use one monorepo and we do that everything you do must always be code and all truth lies in the repo so if i die today doesn't really matter because i mean i wouldn't care and and otherwise anything i've done even my talk which i just rewrote isn't git anything not committed has no value at all because it can just be gone or broken or what i don't know you can't find it anymore um especially in case you run into a bus and or you get hacked because we've got that hacked server i want to see or you want to change your infrastructure which is the most important point i mean that happens frequently or frequently every few years you want to switch from one provider to another one if you don't have domain or anything actually everything script scripted you won't be able to make that move if you have to manually set up infrastructure won't work and thanks very much questions\n"