It’s not necessary to talk about development work, which is the same as writing code in general. This article is mainly about issues in my operation and maintenance process; I also recommend a few good tools for you to use.
The first half of this article is a compilation of some of the O&M issues I encountered at work, The second half is about Termux, a small tool.
My Operation and Maintenance Process
From my point of view, some of the tasks in operations are actually the ones that cannot be automated. If we had better infra, if our code were better and more automated, then the work of Ops could be reduced gradually.
Ops work can be divided into a variety of situations, and the following is a list I encountered at work.
By degree of urgency
Not urgent: these are mainly errors such as timed-script timeouts; usually we selectively ignore them. Some scripts need to manipulate the database, it may be slow. It’s alright as long as the two scripts do not run together. These are the alerts we receive most often.
General emergency: for example, some containers’ CPU or memory exceeds the quota because of a sudden traffic spike. A good system could deal with this problem automatically, but manual adjustment is also fine. At this time, we just adjust the threshold to silence the alarm.
Emergency, in some cases, the machine will appear to be dead. At this time, you usually have to contact the SA to reboot the machine. We can’t actually do anything about it, but try to reduce the number of crashes and improve stability. In my opinion, crashes are not particularly urgent, because Nginx will block the nodes that crash, and immediately switch to other nodes. Some services may be suspended for a while and then recovered.
Especially urgent: a problem with our platform itself, such as a sudden failure to deploy, or a problem with a deployment program on one of the machines. Although there is no impact on the existing service, it is really related to the stability of the service,
By processing method
Alarms that can be blocked directly can be done directly in the admin interface.
Temporary blocking, application redeployment or other actions can also be done from the admin interface.
You need to use ssh to connect to the machine, similar to checking after a crash and reboot, or fixing if you can’t deploy.
By whom to deal with
Platform administrators: the admin should know all the problems encountered above
The developer or the user of the PaaS system: if the application traffic is too heavy, it’s always necessary to notify the developer to add machines to handle it
SA: as mentioned above, crashes and similar problems that the administrator can’t deal with
DevOps pains
I’m both the developer and the maintainer of the system, which is a kind of simple DevOps. Actually, the development is fine, I just need to work on it during office hours. But maintenance may take some time out of my life, and, basically, it’s impossible to avoid.
Office hours
I was not assigned a specific time to finish tasks, because I also had the operation and maintenance work.
Suddenly a problem appears; it takes a day or half a day to deal with, so development progress is not so easy to control.
No office hours
There are no specific development tasks, but you still need to be on call — similar to the Chinese New Year, you take turns with your colleagues so there’s always someone to solve problems.
Termux
During non-office hours, I can’t always carry my laptop. Connecting to machines to fix problems is really quite troublesome, so I thought of two ways:
- Find a cyber cafe — you just need a simple SSH environment configuration
- Install ssh on my smartphone — small screen, but it just works.
Well, I think Termux is the best terminal solution on Android smartphones. I have also tried JuiceSSH, but I could not even copy a secret key into it.
Termux’s main function is not actually ssh; it is a subsystem-like environment, presumably using chroot —
ssh is just one of the tools you can install in it.
Install it
htop
ssh connection
OpenSSH supports the ssh_config file; you can refer to My OpenSSH Series to configure environment
PS
I’m looking for a part-time job in Ops and I’m very familiar with Linux systems and Kubernetes.