Automation Journey

When you are working on some large projects, it grows exponentially complicated to keep in mind when you don’t automate things. Think about an application that allows you to share your thoughts to your network. You’re building the app, testing it, and deploying it to multiple environments. The work is not over yet, you also want to know how people are using it, what are the runtime errors and if your services are running smoothly or not. And this “in writing” is few statements but there is a lot of work to be done in order to achieve final product operating under control. In terms of a web application and web services, there are many steps you can take to automate things.

Code formatting and linting
Database migrations
Writing unit tests
Code snippets
Automating CI/CD pipelines
Instrumentations and alerts
Handy scripts

Let’s discuss each point in detail.

1. Code formatting and linting

Do you manually format the code? As a developer, you’re more into writing code and more concerned about efficiently writing code, taking care of your code is performant, and covering a long list of requirements.

When it comes to code formatting, it’s an important step to ensure the code is clean and readable. Most of the time you or your team of developers will be looking at the code, again and again. if you don’t structure your code in a standard convention, you might make it harder for you as well as for contributors to understand the code. Most of the code editors have support for extensions that you can configure and install to perform code formatting automatically when you commit the code or save a file while editing. For example, you can install Prettier in VS Code and configure a few things-

Husky to run code formatter before every commit
Configure VS Code to format the code automatically when you save the file
Prettier config so that the same rule applies to everyone contributing to the project

By looking at the steps that you have taken above, you have solved the following problems-

You don’t need to manually format code again and again.
You and your team will be on the same page because the rules are common and the code conventions are taken care of automatically.

Similarly, code linting is another step you can take to check syntax, find problems, and enforce coding conventions. You can configure rules and the linter will take care of reporting syntax or code-convention-related errors upfront which is not possible just by formatting libraries like Prettier. Combining both, you get good control of how the code should be written, which is the first part of our goal - automation.

2. Database migrations

When it comes to backend services that talk with databases, most of the time you have some database and tables required for the services. Because of ever increasing requirements, you might be required to alter the existing table or perform data migration in a consistent way. So, when the solution is promoted to other environments like staging or production, it automatically takes care of creating tables for you if they do not exist, or run the data-migrations on them.

Writing migrations are the best way to deal with these operations where every time you promote your services, you don’t need to keep a long list of things to do related to database migrations. For example, if you are using any RDBMS, knex is out there which you can use to generate migration templates and be safe even if you have many nodes running for the services. Generally, these tools provide a locking mechanism under the hood and keep track of the migrations which are already used with this, it ensures that when you deploy your services in a cluster it doesn’t run migration multiple times.

3. Writing unit tests

This is one point where I stress more because this is critical and super important. The good coverage of your code ensures that anyone making the breaking changes will get to know what they will be breaking in the system with their changes. And this takes a lot of worries off your head when you’re shipping new features or fixing any bug in a very huge codebase. You focus more on the functional aspect of the feature rather than maintenance.

Even though your codebase is small, you should write unit tests from day one. Because with time when the codebase will grow, you will never be able to cover all cases, and even if you do a dedicated amount of work to cover cases you will not be sure if you have covered all the cases. If your team doesn’t have the setup yet, take initiative, and introduce unit testing in your projects from day 1.

4. Code snippets

Do you repeat a lot of stuff writing from scratch? For example, writing React components and every time writing imports and class definitions? Use emitters or custom code snippets.

This is not super critical but your snippet collection might improve so better over time that you will start seeing the development time improvements. Just to do the math, let’s assume you spend 3 minutes setting up the base file and writing instrumentations, and let’s assume you have approximately 1000 such files then, without automating this, you’ll consume 3000 minutes. This is just about one case, there might be so many patterns in your day-to-day activities. Adding it one by one in your snippet collection will be more helpful.

However, enforcing this practice to the team might not work better in the start, because you might create snippets for every small thing but other team members might keep it very specific to complex stuff. And when the list is large they might feel too much effort to type and get the snippet. Overdoing this will kill productivity. Remember, do not overdo anything, especially engineering.

5. Automating CI/CD pipelines

Now there are many tools that you can utilize to automate CI/CD pipelines, for example, you can use your remote repository service that you are using to automate the build and test pipelines. Gitlab, Github, and Bitbucket have support for it. If you are not using any of these, still you can write hooks and automate the pipelines and enable your setup to run the tests and see if everything succeeds then publish the docker images and trigger service deployments.

6. Instrumentations and alerts

Instrumentation refers to an ability to monitor or measure the level of a product’s performance, to diagnose errors and to write trace information.

Until you don’t instrument your code, you will not have enough metrics to decide what is going on operationally with your services. The more the merrier. AWS, GCP, or Digital ocean give you basic details about how your containers are doing but as an application developer, since you know the domain of your system very well, you are the right person to decide which metrics are more useful to you. And when you have enough metrics, you can start pushing alerts to your email, Slack, Teams, or any medium you’re using on-the-go, because you can’t stare at the Grafana dashboard the whole day and still miss any critical problems within the system.

For client side applications and understanding UX issues you can use tools like Google Analytics, Kissmetrics, or Mixpanel. Using these tools, you don’t have to get the user’s time for existing features. Most of the issues where users experience the problem, they themselves will not be able to explain. Because they might not be good at understanding what is feasible or not. But you can learn these things by exploring what is the dropping point in your applications, which features are hard to discover and so on. And when you try to understand the user’s side of the story, you will be able to relate their pain point and make decisions which will increase productivity of users.

7. Handy scripts

Quite often you undergo tight deadlines for shipping features and time doesn’t permit to automate some part of it. In these cases keep the scripts handy so that whenever needed you don’t have to run clueless. Start your machine and run the script!

Conclusion

Automation is a mindset, which you should adopt from begining otherwise the process and the whole system will become unmanageable. Do you have any thoughts, please post them in comment.

Thanks to Abhishek for reviewing the draft in the last minute :)