Action-based VS Process-based Solutions to Problems
Last updated:Examples here are usually related to engineering / data science, but should be applicable to other areas.
Whenever we deal with problems of any sort, we usually default to solving it now, especially when we are pressed for time, but solving problems consistently in a way that they are always dealt with is very different from solving it multiple times as a one-off solution.
These are two very different things:
Solving a problem once (hacky solutions are acceptable if they reach the objective, inefficiency and manual work are not a huge issue because this is a one-off solution)
Building a process to keep solving it or avoiding the problem in the future (needs a system, needs to address human factors, needs to be efficient and sustainable, must deal with feedback loops)
Examples
| Problem | Action-based (one-off) Solution | Process-based Solution |
|---|---|---|
| Hiring | Search for candidates for a single position, screen for skills, preselect a list of 5 people, interview those and make one offer for the best performing candidate. | Set up a system with a well-oiled funnel for several stages of the hiring pipeline, such that an adequate amount of people are available to execute those and a steady stream of new poeple are hired to replace people who leave a company. Set up a feedback system to detect problems in the pipeline. Find inefficiencies in the process and iterate. Think about training such that different interviewers have similar behaviour. |
| Data Labelling for Data Science | For a single dataset, you can do it all yourself. You can take the data, sample an adequate amount of examples as consistent with the time you want to dedicate to this. Then proceed onto adding labels for the data. |
To set up a system such that labels are added to new data as it becomes available, you need a different approach. You will probably need tools to label those (manually creating CSV datasets and sending them over the wire isn't scalable). You'll need to think about rotating labellers to avoid biasing your data with the views of a single person. You'll need to have quality-control systems in place so that you can afford to employ less technically savvy people to label the data, without losing quality. |
| Decoupling Services | In order to decouple a single pair of micro-services, you just need to add tests such that the current functionality is maintained on both sides and then proceed onto the code, seeing what files/modules you need to change, etc. | In order to decrease coupling between micro-services as a whole in a company (for current and future systems) you need a different approach (as you can't possibly go through every piece of code in all services, nor can you make yourself a bottleneck by requiring that every single code change in the company is approved personally by you) In order to solve this problem you would need to build automated tools (for instance, checking code during Pull requests), you would also need to think about training and educating engineers as to why they want to keep services loosely coupled and you will need to think about incentives. |
Emergent effects
Emergent effects are those that surface when you are dealing with a complex system.
Most problems at scale are complex systems because there are multiple interactions between parts of the system (most importantly, humans).
These interactions are very hard to understand and predict and they increase exponentially as the number of elements in the system.
Some examples of emergent effects:
Communication patterns/overhead: Communication becomes an increasingly important part of any system as it grows. Most nontrivial systems require coordination between people and people making mistakes usually affect other parts of the system.
Feedback loops: Feedback loops are cases where the system affects and changes the environment it's operating.
These can be positive or negative loops, depending on whether they make their own operation easier or harder.1
In addition, they can be reinforcing, meaning that the continued operation of this system increases its own impact or balancing, where the effects of the system make itself weaker.2 2: footnote-content-here
- Office politics. Every mid-to-large system will have people and people will sometimes have ego issues and/or irrational opinions and may act destructively, especially if there are hierachies. This needs to be kept in check to that the system can operate without problems.
Incentives
As was said before, whenever there are people involved, there will be incentives at play.
People will always have their own individual objectives (make more money, attain higher status, attain more power, etc) so whenever there is a conflict between the system objectives and people's own individual aims, the latter will always take precedence.
The key is to make it so that they align, i.e. think about ways to make those two coincide (performance-based pay, making people share the risks/rewards).
1: People sometimes refer to these as virtuous or vicious cycles, respectively.